The model already has its own quality benchmarks elsewhere. The article is just about running the model on X hardware, so the remaining question is then how fast it is. Or does the output quality somehow depend on the hardware too?
replyThe quality is obviously much worse, but still useful as a reference if you generally know what you are doing
replyIt solve the "I'm coding on the plane and need to look up this thing I've forgotten" problem, for me at least
Local model as such will give you "autocomplete on steroids" but it is not going to run away and implement cross project feature like frontier model in let's say Cursor.
replySo there is no value in testing quality of answers, but there is value in testing token speed.
You just have to have correct expectations.
Is autocomplete using LLMs really useful? Even with frontier models I found it to be about 50% right, I turned it of and prefer to use IntelliJ built-in, it is way more reliable.
replyFor me local models is all about quality, and how to achieve that - e.g. by providing guardrails that test the job done.
That's fair. There are even many dimensions to define 'quality' which include use case (coding? writing? multimedia?) and prompt. I suppose if you ask testers to provide benchmarks with their analysis, that might hamper their desire to share.
reply