undefined

points

[-]

I fully agree. Some of these tautologies, RAGs or RAG adjacent ideas feel like there is real potential. I know in my own testing adding a knowledge graph provided cost saving benefits.

Ultimately, the problem is people aren't actually benchmarking any of these ideas or if they are its via an extremely misleading methodology. For example, headroom violates cache frequently, nullifying any potential savings it could have; but does their benchmarks show that? Nope. [1]

As an aside, I wonder how many days are we away from Codex or Claude taking all the "read" tool calls and re-assembling a full map of your codebase and then offering it as a feature.

[1] https://docs.headroomlabs.ai/docs/benchmarks

by everforward2 hours ago|

parent|

[-]

I think people don’t do genuine benchmarks because the market forces them to pretend their solution works for anything you can throw AI at. Companies whose valuation is based on them being the RAG/compression/routing/etc company. They can’t admit it only works well in a specific domain because then they’re immediately $300M in the hole.

I have more faith in companies with a more targeted approach. Eg gzip does fine, but video codecs beat compressing raw video by a ton.

> As an aside, I wonder how many days are we away from Codex or Claude

That sounds like SourceGraph but twice as expensive, although it does have “AI” so probably lol