upvote

points

by slowin1 days ago |
comments
upvoteby nowittyusername1 days ago|
next
[-]
With the latest codex (weekly quota burn) fiasco I tried open weight alternatives for the first time. And tyeah... open weight models cant compete with likes of astra yet. But, my hope is that by the time I get my Mac studio at end of november an open weight models would have closed the gap (which i think is realistic at the speed of progress). Now its true a better gpt version will also be available then but it also seems the gap is shrinking with time so theres that.
reply
upvoteby Octoth0rpe1 days ago|
parent|
[-]
> And tyeah... open weight models cant compete with likes of astra yet

I think this is true, but also misses that a lot of us are just doing basic flask apps with a react front end. We don't need astra; Something sonnet 4.6 level locally is perfectly sufficient 95% of the time, and maybe 99% of the time.

reply
upvoteby brandon2721 days ago|
parent|
[-]
This. People have convinced themselves that the absolute frontier is what is needed, anything below it is an unacceptable compromise, and we seem to be speaking different languages when it comes to discussing model capability.

It's like watching a discussion about cars available to take on a 100km road trip. A new car gets released that is on par with a Toyota Corolla but it is dismissed as completely useless for a 100km trip because it doesn't have the seat massagers and air ride suspension that the new Escalades have.

The reality is that something like Sonnet 4.6 is still amazingly capable for so many programming tasks, especially if you already have some reasonable level of experience to steer it in the right direction.

And if you think Sonnet 4.6 is still worthwhile, then it seems undeniable that something like Qwen 3.8-27B is also worthwhile.

reply
upvoteby dash21 days ago|
parent|
[-]
The problem is that even if you're doing CRUD apps, Sonnet level will be good enough... 95% of the time. But the 5% will kill you.
reply
upvoteby _hugerobots_1 days ago|
prev|
[-]
Local models can be widely used as productive assets. Yes the infrastructure of SOTA API models is engineered specifically for you to be that utility, but the blanket statement that local isn't up to par is intensely short sighted. Billions of tokens per month on local pays for the hardware when compared to sota costs per month.
reply
upvoteby slowin1 days ago|
parent|
[-]
I believe they can currently be used productively for non-coding tasks (classification, light summary)... but they definitely are not even close to SOTA when it comes to software development.
reply
upvoteby _hugerobots_1 days ago|
parent|
[-]
Defining productivity is a use-case scenario, and a wildly generalized assumption for most people in this argument. Local infrastructure doesn't need to be sota for absolutely every single need for a dev lab, but it absolutely can be delivered with non-api frontier class models.
reply
upvoteby slowin1 days ago|
parent|
[-]
Just to be clear, I'm specifically talking about coding. I think local models can help with productivity today, just not coding.

I'm also a huge fan of local models and think it's absolutely imperative that they continue to advance so we can move off of the Anthropic/OpenAI hosted models. It's important to accurately asses where we are in that journey though.

reply
upvoteby srcreigh1 days ago|
parent|
next
[-]
I think the issue is generalization, if you were more specific about which local models aren’t good enough for which tasks compared to which frontier models in your experience, it’d be a lot more informative
reply
upvoteby slowin1 days ago|
parent|
[-]
I can't just go into any codebase and ask a local model to "Implement this feature: xxx" and get acceptable output. I hope to someday soon though!
reply
upvoteby _hugerobots_1 days ago|
parent|
prev|
next
[-]
Like the other commenter, I'm confused about the 'just not coding' conclusion. I'm using Qwen 27B on a 5090 at > 100tk/s with 150k context (which isn't enough admittedly), and DeepSeek v4 Flash with 1million context on a gb10/spark. Both of which are performing surface level, and deep needle precision infrastructure architecture. They code 24-7, stupendously.
reply
upvoteby fhub1 days ago|
parent|
[-]
It would be interesting to hear more about how you’re actually using them. Do you have sophisticated feedback loops around the models so they can verify their work and converge on good solutions? And how do you decide what to give the 5090 vs the Spark vs a frontier model?

Correctness matters much more than speed to me, but if I can get both, that’s obviously very interesting.

reply
upvoteby brandon2721 days ago|
parent|
prev|
next
[-]
Local models are undeniably capable of "helping with coding" today.
reply
upvoteby slowin1 days ago|
parent|
[-]
I so want this to be true, but for the kind of coding I do (not Flask apps), it's definitely not the case. Like I said, SOTA models just barely, barely work for me. My projects are usually 100k-1M lines of Rust or Go.
reply
upvoteby brandon2721 days ago|
parent|
[-]
Out of curiosity, what do you find the SOTA models are simply incapable of when it comes to your Rust and Go projects?
reply
upvoteby slowin1 days ago|
parent|
[-]
The SOTA models now work really well in my codebases, but that's only been since Opus 4.5/4.6-ish. Prior to that, and with current local models, they simply couldn't work holistically and would just thrash around. Now I feel as if SOTA are approaching my coding levels if not surpassing it. I still need to guide on architecture, but I can see that going away within the next year or so as well.
reply
upvoteby brandon2721 days ago|
parent|
[-]
Thanks, that makes sense. When you said they “barely, barely worked” for you I assumed that meant something different.
reply
upvoteby slowin1 days ago|
parent|
[-]
Oh yeah, that makes sense, sorry! I meant they just started working well and did not until relatively recently.
reply