undefined

points

[-]

The joke is that macs are famously slow at prompt prefill and you are not getting anything back in 3 seconds, or probably even 30. Once they get generating, it can be acceptable, but the TTFT is horrendous.

There's a ton of well-understood things Apple can and hopefully will do to massively accelerate every stage of this pipeline and hopefully they're hard at work implementing most of them for m7.

by EagnaIonat2 hours ago|

parent|

[-]

> The joke is that macs are famously slow at prompt prefill and you are not getting anything back in 3 seconds.

Your knowledge is out of date. In truth it depends on the Mac and the models used.

I asked this question on M5 Max 128GB, using Ollama model Quen3.8:27b-mlx, with thinking enabled.

Question: "Give me a python code snippet that opens a file and sorts the lines of text. "

In 2.4 seconds it gave me 4 examples that work with different sorting configurations and a summary of when to use each.

Compare that to an older model of gpt-oss:20b, took 5 seconds to finish thinking and 2 seconds to stream the answer. It gave me one python example snippet and two one liners that do the same thing.

by desterothx2 hours ago|

parent|

[-]

We are talking about models of the flash size, 100s of billions of parameters, don't listen to the media, size does matter

by EagnaIonat1 hours ago|

parent|

[-]

I was just pointing out your claim that you can't get a response in 3 seconds. If I had asked the model just for the code it was under a second.

Local models are good enough that it's not an issue.

But keep changing the goalposts if it makes you happy.

by mike_hearn1 hours ago|

prev|

[-]

I think it's been pretty much proven by now that there are no cases where local inferencing is better than remote inferencing, unless absolute privacy is a hard requirement. The efficiencies that come with datacenter scale and hw can't be beaten.