undefined

points

[-]

I think it's niche now because getting the hardware to run it is expensive and the quantized models don't work as well. If those improve then it would be a no brainer to pay one off for the hardware instead of a fortune for API calls.

by dofm5 hours ago|

parent|

[-]

I am not really convinced that four bit quantisation is that bad; almost certainly six will be enough. But Google are making claims for their QAT tech in Gemma that they are surely using or testing in Gemini that it preserves nearly source model quality while reducing footprint.

The hardware for 50 tokens per second with a four bit quantisation of Gemma 4 26B or the sparse Qwen 3.6 is not really that expensive: it’s a secondhand M1 Max.

Beyond that, I agree. I think moving planning tasks to local is a now thing, not that it really has much impact on token spend. I also think many small coding tasks are fully within the grasp of the above two models.

The main issue right now is that the software landscape is rather confusing, but I reckon uncomplicated Gemma 4 26B QAT support with MTP is a few weeks away.

by 2 hours ago|

parent|

prev|

[-]

deleted

by jqpabc1235 hours ago|

parent|

prev|

[-]

AI vendors are attempting to offer the whole apple. And they are spending huge sums of money in the process.

But most businesses don't really care about most of the apple --- they only need their special bite out of it.

For example, doctors mainly care about medicine. Nvidia is attempting to provide the hardware needed for local, specialized models.

by dofm5 hours ago|

parent|

[-]

I think it is likely to appeal to video and photo editors who want to use AI tools (the press release has a quote from Blackmagic Design, as well as from Adobe, who I think have no stomach for their own cloud AI).

But I don’t know about specialised: this could run quite large models with MoE.

by dgellow5 hours ago|

prev|

[-]

Performances of local models are pretty bad compared to what AI vendors offer, token generation is just too slow to be that useful. And you need to allocate GBs of memories, something that will stay very expensive to buy for a long time.

Running local models will stay niche for a while, unless we see breakthroughs

by jqpabc1235 hours ago|

parent|

[-]

Dumb idea --- how about if we limit local models to specific domains --- medicine for example.

Most doctors don't care much about engineering or accounting or software development or 10000 other things that big vendor models address.

This area is yet to be really explored. Nvidia aims to provide the hardware to do so.

by CamperBob21 hours ago|

parent|

[-]

That's a fairly obvious idea, not dumb at all, but unfortunately it doesn't seem to pan out. Trying to specialize an LLM in one area harms its 'cognition' in all areas. For instance, if you train a coding model without all the Shakespeare and soap operas and Wikipedia and pirated Stephen King books and ancient Roman history and whatever, you end up with a worse coding model.

I'm not sure anyone really understands why.