Newsletter Ingestion: From Inbox to Feed
I’ve implemented the core newsletter ingestion pipeline, but it’s not quite ready for prime time. Here’s the current state and what remains to be done.
What’s Working
Email Processing: Cloudflare Email Workers receive incoming newsletters and forward them to the Dejour ingestion endpoint. The system correctly handles:
- Multiple newsletter formats (plain text, HTML)
- Various encoding schemes
- Attachments and inline images
Content Extraction: Claude AI processes the raw email content to extract:
- Individual story links and titles
- Author information when available
- Publication dates
- Topic tags and categories
Feed Population: Extracted stories are added to the main feed with proper attribution and metadata.
What’s Missing
Cross-Newsletter Matching: The biggest remaining piece is identifying when the same story appears in multiple newsletters. Currently:
- Stories from different sources appear as separate entries
- No deduplication or “mentioned in X newsletters” aggregation
- Manual review is needed to spot duplicates
Quality Filtering: Need to implement:
- Spam detection
- Paywall content handling
- Relevance scoring
Performance Optimization: The current pipeline works but could be more efficient:
- Batch processing for multiple newsletters
- Caching of extraction results
- Parallel processing of attachments
Next Steps
- Story Matching Algorithm: Implement fuzzy matching to identify the same story across different newsletters
- Aggregation UI: Show “mentioned in X sources” badges and combined metrics
- Weekly Digest: Automated summary of most-mentioned stories
- Performance Tuning: Optimize the extraction and matching pipeline
The ingestion pipeline is functional enough for testing, but I’m holding off on the public announcement until the cross-newsletter matching is implemented. This feature will make the feed truly valuable by surfacing what’s actually important across the tech newsletter ecosystem.
Stay tuned for updates as I work through the remaining pieces! Stay tuned for updates as I work through the remaining pieces!