Agree. I think many people forget that not long ago, HTML markup on many sites was a lot richer than it is today. Making it trivial to produce a good trimmed down markdown version.
replyThe reason it may be more difficult today is because we've lost a lot of that. Some of it because of modern JS frameworks, but some also because publishers simply don't want to make it easy for the useful stuff to be scraped and extracted easily.
I'm not convinced that's changing because of AI agents (it's getting worse in many ways with anti-agent rules). Maybe improving for documentation pages intended for agents. But if it is changing, I think it'd be far easier to improve the HTML and let the agent take care of the rest.
> in many cases I’d love to skip the CSS/JS and read the markdown directly
replyI've been enjoying https://defuddle.md since learning about it. Works great. Not affiliated.
Exa also has an API for it that has worked well for me, returning markdown for a URL, which means you don't need to render js or anything yourself. It doesn't need an account for up to 1k requests/month, which is more than I've ever needed.
replyMost harnesses have a "zero config" plugin that works, eg: https://pi.dev/packages/pi-exa
(no affiliation to either)
I think you are operating under the misconception that software developers are in their right mind
replyWho in their right mind would feed any random LLM crawler their website's content?
reply