They were given day zero access to release their quantizations[0]. The lowest quant will still require 256GB VRAM to run inferences, but as always I'm amazed by the Unsloth team by how much they can reduce the requirements for these powerful LLM's, and the easy-to-follow guides to get started with using them.
replyUgh, ok that's out of my league :) Almost though, another 32GB and it would work.
replyThanks for the link, appreciated!
Not sure what league people with a five-year old M1 Pro with 16GB of unified RAM fall into :)
replyhttps://huggingface.co/unsloth/Mistral-Large-4-1T-A52B-GGUF
replyI have no idea how you could shrink this down to fit on any type of prosumer hardware though. I doubt it's possible.
Yes, you are right, and I was too optimistic. I'd have to level up.
replyThis is the stuff I've put together and it is clearly too small for this:
https://jacquesmattheij.com/local-inference-with-large-langu...
Could you mention the exact model of the PCIE switch/host adapter you are using? The Aliexpress links are always a bit ambiguous. I am working with the c-payne switch https://c-payne.com/products/pcie-gen4-switch-5x-x16-microch...
replyThe system is a $25k custom build with 8 video cards (4x RTX PRO 4000, 2x RTX 4090, 2x RTX 3090). That price was only valid for early 2026.
reply