There was a 3.5 122B 10A release -
replyI tried it for a bit, and It was not really worth its size. It got swept up in all the other AI news recently, but laguna s 2.1 I think is the best ~100B moe model right now
replyI didn't mention it above, but Laguna S is my other favorite model. I use Qwen a lot more, it's smaller and faster, but I like to switch to Laguna when I feel like I need a "heavy hitter" for certain huge or complex tasks.
replyWhat on earth hardwares do you guys have to be able to run 100gb models locally?! That's crazy! I'm here struggling to even get 27b models to run in somewhat usable way
replyStrix Halo, 128GB RAM. I got a refurbished Corsair AI Workstation for a smoking price ($2100) about two months ago. Lucky timing that it was in stock.
replyStrix Halo as well. Bought it for $1,800 new on sale and shoved an extra 4tb drive into it. Been amazing for local AI. Maybe not the absolute fastest thing (usually around 30t/s depending on the task) but has been awesome for a local AI box that I can solar power.
replyNice. Mind sharing the solar side of your setup?
replyCouple of rack mount batteries and roughly 5kw of solar panels. Feeds into a subpanel so I can flip it when I want a couple rooms of solar on the house, or hook a generator up if needed. Can't power the entire house, but works well for thinks like computers, lighting, etc. And if I want to expand, just throw on more panels, or realistically, just throw on more batteries to store the juice.
replyHaha I'm on an Mac Studio with an M1 Ultra, 64gb ram. I bought it when it first came out, it just happens to be good for local LLMs. I have to use a smaller quant of Laguna S though (I think 4-bit? Not at my machine to check), as 8-bit and full size definitely don't fit in the 64gb I have.
replybrb, going to see if 2nd hand mac studios are available!
replyYeah, a good rule of thumb is that the weights take up ~100% of the size of the model, so 100B bytes (8-bit quant) would be, well, 100GB and a 4-bit quant would be half that.
replyI don't know if that's, well, a rule of thumb, it might be, well, straight multiplication.
replyOh, that is a useful rule to know! Thanks!
replyThat's not exactly the math. Theres also vram needed for context. I operate several 72-128 GB machines and the larger the context the slower they go.
replyAnd the context takes space +kv cache. KV cache drives usefulness as your context grows, it needs to pull the kv cache.
Simplified, the context has to be run on every turn, so the KV cache supplies the processed tokens, so it just needs the new inpute.
Yeah, here I am sitting deeply deeply deeply regretting not buying couple CMP 170HX at $200 or $350, knowing I could just flip them ethically at purchase price if nothing came of it... I could have just casually built a 128GB dual A100 local AI monster
replyI'm working with a lab that has a few Ampere GPUs on infiniband and they are just not compatible with the latest quants and vLLM updates. FP8 is about as low as you can go.
replyBut they're reportedly a soft nerfed GA100 64GB/40GB at $1200, that's not more expensive and certainly can't be slower than a Mac Studio.
replyquantized + offload
replyI have an RX 6700 XT with 12gb vram and 64gb system ram. running dense models like 27b is difficult, but i can run IQ4/IQ5 qwen 122b-a10b or 35b-a3b at ~20tok/s
IQ4 qwen 122b-a10b would mean 61GB total size and 5GB active, so about 5GB of the model loaded into GPURAM plus any generated context, and 61GB of weights loaded into system RAM? I don't know if that math is correct, but does that run well? Wouldn't that only leave 3GB of system RAM?
replyUsually theyre quantized. Also, there was a window where AMD 395+ W/128GB was just a high end $2500 hardware with unified gpu memory.
replydgx spark, nvfp4 so I have spare room for KV cache (context)
replyMoE models can use system memory along with a GPU.
replyand get high token bandwidth?
replyNot badly so because MoE models(identifiable by "CoolName-xxxB-AxxB" naming scheme) have bunch of branches in the middle that only one out of all gets non-zero values. Each of branches aka "Experts" as well as top/bottom parts are significantly smaller than the whole, and so CPU emulation of CUDA operations mixed with GPU taking as much as possible become not so out of question, unlike for dense models("CoolName-xxxB" without "-AxxB")
replySimilar to a spark, which isn't blazing fast but usable.
replyUsing qwen 3.6 27b for local coding as well and downloaded Laguna s 2.1 but haven't had time to give it a full spin yet.
replyCurious for any more experiences
I agree. 27b dense really did seem like the sweet spot.
replyA 35 A3B as smart as previous gen 27B would be a sweet point
reply> 35B MoE is certainly a good and fast local model. I find 27B dense to be quite a bit smarter
replyIsn't that just the definition of MoE vs dense ?
Full name is 35B-A3B. 3B Is the token generator thats selected out of the 35b available in the model, which is some layered jazz.
replySo it can be dumber but its quite capablr.
I use 27B in plan mode and 35B MoE in act mode. I noticed that is the best balance for me for consistent tool calls and intelligent planning. Takes some time to switch, but it's worth it for me.
replyWonder if it's possible to share a common cache via cachyllama between 27B and 35B-A3B
replyI've heard 27B is smarter! I tried it some time ago but couldn't get it working with my oMLX. I need to try it again.
replyHonestly the 27b dense one punches way above its weight in a lot of domains, especially coding in my testing, so I think you will probably be disappointed.
replyIn my case I would say they are comparable but moe models are looping and getting lost a lot more than dense models.
replyOn the other hand having 90t/s with any local model is nice and Pi with loop police extension can prevent looping a lot.
Looping seems related to quantization and not the model itself. If youre digging deep into quants to get working context then yeah.
reply