undefined

points

[-]

The reason for running those insanely low quants is to fit in extremely limited memory budgets. The first thing you sacrifice is speed, then context and accuracy (up to you in which order). IQ2_XXS and below is desperate/proof of concept territory. If you have a spare half gig for the MTP drafter, run a larger quant instead, it will be less incoherent, and damn the speed, it won't be garbage at least. Only around Q4 I'd allocate the comparative luxury of more memory for a speed increase. At least on a dense model. MTP makes a lot more sense (but helps statistically a bit less) on an MoE.

Qwaiting for that 3.8-35B-A3B

by danielhanchen13 hours ago|

prev|

[-]

Hey we did not remove the MTP for sizes above 8GiB - but yes for small GGUFs under 8 ish GiB, we removed the MTP module (IQ2_XXS and lower), because it's 500MiB to 750MiB in size, and on small 8 GiB machines, even 500MiB is needed.

As someone in the comments said we made a separate Q4_0 MTP if that's helpful so you can use that.

But I would suggest using UD-IQ3_XXS for 10.9GB for 16GB machines or Q2_K_XL

by xlayn12 hours ago|

parent|

[-]

Daniel, question I got the Qwen3.8-27B-UD-Q2_K_XL.gguf from https://huggingface.co/unsloth/Qwen3.8-27B-GGUF?show_file_in... and continue with my testing, but the model quickly felt into a loop of asking the same thing over and over again, I have seen the MOE do that but not the dense ones.

And I had similar experiences when Qwen3.8-27B unsloth images just came out with the full Q8_K_XL, I'm using an AMD setup which has modifications to save to disk the kv, but your (assuming you are part of the unsloth team) for some reason have been giving me similar issues.

I tried https://huggingface.co/mradermacher/Qwen3.8-27B-Uncensored-G... the 8 bit, 6 and 2 bit... the 2 bit almost use the complete KV doing it's thing and didn't loop itself.

It can be something in my setup, there is a very high chance of that, but the previous 3.6 images from qwen, the 27B, the 31A3 and 122 they are all unsloth and did work on my setup without issues...

Again could be my setup... let me know if there is any data I can supply to you to debug if needed.

by kamranjon6 hours ago|

parent|

[-]

Are you using the recommended settings for temperature and such? https://unsloth.ai/docs/models/qwen3.8#recommended-settings

Often times I run into issues like this it’s because I am using settings for a different model or just forget to set them up.

by mike-the-brain15 hours ago|

prev|

[-]

you can still have it, no?

> We also removed the MTP module from smaller quants under UD-Q2_K_XL (8.37GB and lower) to converse around 500MB of disk space - you can use the Q4_0 MTP separate module if needed

by xlayn15 hours ago|

parent|

[-]

my bad, you are totally right, thanks!

by walrus0112 hours ago|

prev|

[-]

Q2 quantization is basically giving a capable model a lobotomy. It will not accurately represent how smart or capable something like qwen 3.8 27B in Q8 will be.

by zenoprax9 hours ago|

parent|

[-]

Sure, but this is true for all lossy compression (audio, images, etc.)

Given 16GB of VRAM, what will give me the best experience in OpenCode? Currently using Qwen3.8_Q_3

by walrus015 hours ago|

parent|

[-]

probably the best experience would be deepseek v4 flash 0731 (it takes about 170GB RAM on the server side for the full thing and RAM reserved for 1M context) via opencode's $10 a month plan until you use that up, it's either Q8 or full precision. Assuming you're ok with doing things with external inference.

by kennywinker3 hours ago|

parent|

[-]

Why would they have listed how much vram they had if they were looking to rent gpu time on someone else's machine?

by walrus013 hours ago|

parent|

[-]

A casual review of my comment history would show that I've been nothing but the biggest proponent of running models locally, and I do so myself a great deal. But one also has to be realistic about the capabilities of what you can do in a 16GB GPU these days. I already said an extra small Q2 quantization was effectively lobotomized so I didn't want to repeat myself.

This person has basically run into the limit of state of the art for even a modestly sized local model (this isn't deepseek v4 flash 0731 Q8 which I am running myself locally on a great deal more hardware), this is a 27B dense, but they're just not going to have a good time if they expect good quality results out of a Q2. The choices are either upgrade hardware or pay for external inference.

by kennywinker2 hours ago|

parent|

[-]

Fine, but they already said they are using a Q3, so Q2 being unusable (disagree, but whatever) isn’t helpful new info.