Unsloth has been dethroned by ISTA:
replyhttps://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
The 3-bit quant is lossless based on benchmarks.
There's a table on the HF page that compares it against Unsloth's UD-Q4_K_XL and IQ2_XXS: https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#fu...
replyOh, wow, they think it's just a smidge below the q4? That's crazy good if true.
replyThe benchmarks they chose are rather cherry picked to not include long context or difficult ones that involve long horizon work or many agent turns, as I suspect this is where the model shows more differences compared to the full fat one
replyYep more hops from the lower Q is likely going to skew the vectors further over time.
replyI wonder if there's a way to mitigate this by running it through an original Q8 draft model, attuned somehow for the PTQ1 quant, but giving it a higher threshold for the acceptance linear with the context length itself?
The longer the context, the higher the multiplier on the threshold, and more likely the draft result is used. Not ideal but it may extend the usable max context.
This model might, even without this, be amazing for short lived agents that work via generations / have changing tasks.
deleted
replyCame to ask the same. From my really rough understanding, it seems like Unsloth's method allows a slightly higher precision at a higher file size, while PrismML's uses a different approach to achieve a smaller size (and presumably less precision).
reply