A signal finder that runs on my own machine
Python program that reads Hacker News, Reddit, GitHub and RSS, drops the duplicates and scores what's worth reading. Runs entirely on my machine, no paid services.
Reading everything relevant published each day is impossible by hand. This pipeline turns it into a data problem: ingest, deduplicate, classify, and rank — reproducibly and privately.
The problem
Useful signals are scattered across HackerNews, dozens of subreddits, GitHub trending, Indie Hackers, Product Hunt, and high-value RSS feeds. The noise is enormous, and most tools ship your data off to a SaaS. I wanted the opposite: full control, near-zero marginal cost, and nothing leaving my machine.
The approach
A staged pipeline, each stage isolated and testable:
Async scrapers → Dedupe (hash + cosine) → LLM classification → 0–100 scoring → Markdown report- Ingestion is async and parallel with
asyncioacross 6+ sources. - Dedupe works at two levels: exact hashing for literal duplicates, and cosine similarity over
bge-m3embeddings for near-duplicates. - Classification runs on a local reasoning model via Ollama: category, problem, audience, and viability.
- Composite scoring 0–100 plus a Markdown report with an actionable brief.
Persistence is DuckDB — embedded analytics, no server. Ruff + pytest keep the code clean and the stages verified.
The result
A private, reproducible system that collapses hours of reading into a ranked report, running on a single consumer GPU. It's the same local↔cloud routing discipline I apply as a service: use the cheapest model that clears the task.