it seems to me that would only be true if the amount of knowledge needed to complete a task far exceeds the context limit. Why would that be the case? I find most people are using horribly inefficient contexts, and a naive approach to dark factories/software eng swarms seems to only compound this
reply> significantly optimizes accuracy, latency, and token consumption.
replyOn what basis do you come to this conclusion with certainty?
Anyone who has used local LLMs will come to this conclusion.
replyToo much context confuses the LLM. Reduce unnecessary context and performance improves. Not to mention, too much context and you just run out of vram.
Such limitations may be mitigated in future architectures, but it certainly isn't something you can ignore currently.
> Too much context confuses the LLM. Reduce unnecessary context and performance improves. Not to mention, too much context and you just run out of vram.
replyI don't think you can reduce context enough on a local LLM to get performance comparable to a 500k context without those reductions...
[dead]
reply