News

Setting Up a Reliable News Aggregation Pipeline

Most people think building a news feed is just slapping an RSS library into a script and calling it a day. It isn't. The first time I tried to monitor breaking coverage across about forty sources, my pipeline collapsed within three weeks. Too many feeds returning malformed XML, rate limits hitting without warning, and duplicate articles from syndication networks made the raw data nearly unusable. I rebuilt the whole thing from scratch with actual constraints in mind. The RSS standard is nearly dead in practice. Many publishers disabled their feeds or switched to walled gardens like Substack, Beehiiv, or plain JavaScript-rendered sites that serve zero machine-readable content. You end up scraping instead, which is a different problem entirely and introduces its own set of breakage patterns. Here is the practical breakdown of what I actually use now, after burning through about six months of failed attempts.

The Core Architecture

I run a Python-based pipeline using feedparser for known RSS and Atom sources, and a lightweight scraper fallback for the rest. The scraper uses httpx with randomized headers and a retry loop capped at three attempts with exponential backoff. Every source gets its own configuration file with a custom polling interval, because treating BBC and a small niche blog the same way wastes resources and misses updates. The deduplication layer is the part that actually matters. I hash each article's canonical URL combined with a slugified headline. If the hash already exists in a SQLite database, it gets skipped. This handles syndication duplicates across sites like Reuters wires picking up AP stories or Medium mirroring of newsletter content.

Handling Edge Cases

The hardest problem I ran into was timezone-aware date parsing. Feeds list pub dates in dozens of formats — some include timezone offsets, some don't, and a few just have raw timestamps with no context. When I was cross-referencing breaking News from European and Asian outlets, my pipeline would sort articles incorrectly by over an hour, making it look like sources were reacting to events before they happened. The fix was converting every parsed date to UTC immediately upon ingestion and storing that alongside the raw string for audit trails. Another edge case: YouTube video titles and podcast show notes often get pulled into text-only feeds, inflating the noise floor. I filter these out early using a heuristic that checks for common video platform markers in the description field and drops anything that matches.

Storage and Query Layer

All articles land in a single SQLite database with two tables — one for raw ingest data and one for the deduplicated article index. The raw table keeps everything; the index is what your queries hit. This separation means you can re-run deduplication logic when your strategy changes without losing the original data. Full-text search runs through SQLite's built-in FTS5 extension. For a personal monitoring setup, this is more than fast enough and requires zero additional infrastructure. If you scale past roughly ten thousand sources or need real-time alerting, you would want to move to something like Meilisearch or even Elasticsearch, but that is overkill for most users.

Polling Strategy That Actually Works

Rather than hitting every source on a fixed interval, I use a variable polling schedule. Sources that haven't published in over twelve hours get checked every thirty minutes. Sources that published within the last hour get checked every five minutes. This cuts total API and HTTP requests by roughly sixty to seventy percent compared to a dumb round-robin approach, while still catching breaking updates within a reasonable window. Rate limiting is handled per-domain with a token bucket. Each domain gets a configured request budget per minute. If a source starts responding slowly or returning 429s, the pipeline automatically reduces its poll frequency for that domain and logs a warning. I once had a medium-sized tech blog's CDN start rate-limiting my scraper aggressively. The automatic throttling caught it before I had to manually intervene.

What This Doesn't Solve

Paywalled content stays paywalled. This pipeline cannot fetch articles behind subscription walls, and no amount of RSS wrangling will change that. Platforms like The Athletic, Bloomberg, and several major publications have effectively killed their public feeds. Your options there are limited to archive services, paying for access, or waiting for the content to appear on free syndication channels after a delay of anywhere from a few hours to several days. Misinformation and bot-generated content are impossible to filter at the ingestion level. A well-crafted fake story will look identical to a legitimate one in the metadata. You need a separate layer for fact-checking or at minimum a trusted editorial whitelist if accuracy matters to you. And yes, Google News and similar aggregators exist. They handle the infrastructure problems for you. The reason to build your own is control — over what sources you include, how you deduplicate, what you store, and how long you keep it. If you just want to read News, those products will save you time. If you need to analyze it, programmatically access it, or run it through your own classification models, you are on your own.

Getting Started

The code I ended up with lives on GitHub under a project called news-pipeline. It is not polished. The documentation is sparse. It has been running in production for my own use for about fourteen months and handles roughly twenty thousand sources with minimal manual intervention. You can clone it, set your source list in the config directory, point it at a local SQLite database, and run the ingestion daemon. From there, you build whatever interface you need on top of the data. If you are starting from zero and do not want to maintain your own code, the alternative is running miniflux or Inoreader in self-hosted mode. Both handle most of the plumbing I described above. Miniflux is free and open-source. Inoreader has a paid tier that adds more aggressive deduplication and better feed detection. Neither gives you the same level of customization, but they are stable enough that most people should probably start there instead of writing their own scraper first.