I ran a similar personal project where weekly I scraped a country's news stories across curated sources using Brave News API and used an LLM to dedupe and cluster stories. This got expensive fast, so I'm wondering what your approach is to keep costs down across what I imagine is a lot of volume, i.e., one months' worth of stories across several sources.
replyThis sounds really interesting and something I'd love to try. Do you have more details on how someone else can utilise this?
replyNot yet, but I'll see if I can publish the basics for it this week.
replyThe what now. But not all sources have epubs, right? Is there any one that you're missing now?
replydeleted
replyCalibre has a feature that will fetch a rss feed or (IIRC) scrape a site and make an epub from it.
replyNope! Every source I care about either has an RSS feed, or I've been able to make a custom recipe [1] to handle things like auth for paid sites and whatnot.
reply[1] https://manual.calibre-ebook.com/glossary.html#term-recipe
epub is ZIP with HTML+CSS+images+some metadata.
reply