This model is for researchers who already have racks of equipment, inference providers who sell access to models, and for businesses who must (or choose to) use on-prem models. All of which have to pay to use the model if their revenue rises beyond certain thresholds: https://huggingface.co/moonshotai/Kimi-K3.1-Pro/blob/main/LI...
Why? Americans are always yapping how China growth is basically fueled by IP theft, are you saying that it works both ways?
Sign me up!
We will break even from our old Claude spend in about 2 weeks.
It services 30-40 analysts and staff concurrently and is faster than any of the SaaS out there for us.
Also what harness do you use? Any problems you encountered on the way?
EDIT: per project, not per month
EDIT 2: digging out an old bill from 8 months ago when we bought the server, $14k per month is what we were running and it was going up, a lot.
Enterprises also need very stringent IP protections that you don't get through the subscriptions. Even the Anthropic-API per token license isn't great - sometimes you need to go to 3rd parties like AWS Bedrock to get guarantees regarding no multi-tenant GPUs.
Plus on prem can go find abliterated models.
It's an advantage that keeps me coming back for local models when I can't get around Claude or Codex's little blockers.
I regularly use together.ai with models like this, where they're fully self hosting them and I can turn off all data retention, at a fraction of the cost of proprietary models. I recently discovered abliterated models (models edited to not refuse to work on security stuff) and hosts for them like refuseless.com, and that's an option for open source models, not for proprietary ones.
Unfortunately due to their licensing requirements for modal-as-a-service providers, the pricing can’t go lower than Moonshot’s from what I can tell.
This is still great if like your example you trust together.ai more, or if different providers have better speeds/cache rates (though usually the original lab has the best cache hit rate).
And obviously if you’re a large enough service to afford to self host or finetune it then this is good.
All the service provider needs to do is provide the option to rent out VM's with access to the inference cards.
Have the customer provide the model.
Maybe provide a bit of a speedup, by letting the customer mount the model by hash.
Sure, that all sounded way too technical 10 years ago, and somebody would have said that's already too much annoyance to deal with for average customers.
Reluctantly i'd agree.
That was 10 years ago. Nowadays, with some proper docs, this entire set up can be up and running in 5 minutes.
Not a trolling question, genuine interest.
Usually there are some static experts in each attention layer, plus a whole bunch of dynamically-routed experts. Typically some form of LRU cache is used for the dynamic experts.
So you can keep only the static expert(s) and routing layers in memory and then stream the dynamic experts from SSD, at a modest inference speed cost driven by the latency required to load layers from SSD when they aren’t already in the memory cache.
The LRU cache works reasonably well because a given task will often hit the same experts on each iteration of the forward pass.
For example, on a Mac Studio, Qwen3.8-flash-next (180b total parameters) with 50% of experts in memory and n-gram offloading has about 1/4 to 1/3 the inference speed of everything in memory. That requires 32GB of vRAM for the weights (plus a little extra for context, obviously)
I can't, however, run DS4.1-1017, for example, but maybe I didn't find proper options and quantization yet.
So I'm really interested how to be sure what can/cannot fit in specific hardware (for example, I want to buy a Mac Studio with 64 or 128 GB RAM, what MoE model it would run at comfortable speed? Will it be qwen-flash-next or something bigger? And so on). So far websites with "benchmarks" were really terrible in that regard (and huggingface calculator even worse!)
This is expected if you're actively steering the agent, but what about running unattended tasks - perhaps multiple such tasks in parallel, such that you can stay engaged by "attending" to each of them in turn? Then you're not restricted to such a requirement, and can benefit from much slower (individually, not wrt. total throughput) inference streams.
> I currently have 192GB and 24GB VRAM
I'm not familiar with beellama, but this seems like it could be the perfect setup for Strata which people are saying good things about, including for running DS4.1F (at least quantized). The whole point of it is to effectively leverage heterogenous RAM/VRAM setups. Of course llama.cpp itself is doing a good job of trying to catch up, but they've yet to support efficient SSD streaming (mmap doesn't quite cut it) in the main branch.
How so? This is not obvious to me; AIUI Strata does support streaming weights for larger models via SSD. Of course the speed would drop quite a bit once you rely on SSD, but it ought to be workable. Since you're on a HEDT/server platform, you can also benefit by adding more SSD hardware to be accessed in parallel via the abundant PCIe lanes.
This depends on how you plan to use the machine... but Qwen-flash-next is right at the top of what a 128GB Apple Silicon can do.
QFN with 4bit quantization will run "comfortably" with all weights in memory and a 128k context on a 128GB machine. On M5 Max w/ 128GB, that will give about 75 tok/sec (using oMLX with MTP enabled). It spits out text much faster than I can read.
However, this truly uses all the RAM. You'll have like 8-12GB left for the OS and apps, which isn't really enough to actually use the machine as a workstation while the model is loaded. But it would work fine as an inference server with Opencode running on your laptop. (Hence, "how you plan to use it?")
Alternatively, you can offload QFN's n-gram table to SSD (this is just a settings option in oMLX), which drops generation speed on the same machine to ~60-65 tok/sec (assuming the model lives on the internal SSD) and reduces memory usage to ~90GB with even a large context. That leaves plenty of RAM to use the Studio as your main workstation. Also, you may run much longer contexts because of the extra memory headroom.
Second alternatively, you can use oMLX's expert offloading feature to keep 50% of experts in memory (with n-gram offloading also). This reduces the RAM footprint to a little over 32GB + context. Offloading the experts drops generation speed to around 20-25 tok/sec on the same machine as above, but it would allow you to comfortably run on a 64GB machine, while still using it as a workstation.
I have tested the 50% expert offload case on an M1 Max / 64GB, and the inference speed is only a little slower (more like 18-20 tok/sec) than M5 Max / 128GB because in both cases the bottleneck is SSD read bandwidth and the GPU is spending a lot of time idle.
One last thing: GLM 5.3-flash doesn't use an n-gram table, so you can't use that trick to reduce memory footprint. You'd need to use expert offloading to make it fit a 128GB machine. Since it has over twice as many active parameters as QFN (which doubles the required SSD read bandwidth), my guess is that a 128GB Studio would not be much faster than your current setup for that model.
Perhaps a flood of 8x B300 will be homelab ready in a few months
Haha, that's beyond wishful thinking, I'm afraid. We're not even seeing a flood of A100s (6 yo and several generations old). Every GPU is still being squeezed right now, and the trend will probably continue for the foreseeable future.
I wouldn't be surprised if RAM cartel or nvidia will attempt something like that, plus nvidia already talks about license-locking GPUs.
Also, again, IIRC, IBM hardware already works this way. There are almost no aftermarket when I last checked for them because it's all locked up and useless.
Perhaps one option is building data centers along with solar power plants in sunny countries that have a lot of space. I'm not sure about the water/cooling situation though.
It would certainly fit the pattern of Bitcoin farms clustering near dams.
Bonus points when the morons are doing so because they get all their political news from a Chinese platform. The [Americans] have gone insane.
The political system just does whatever the captains of industry tell them to do.
A lot of the data center build out has been bipartisan. Unless Dems realize they need to listen to their constituents to avoid a 2028 bloodbath (they probably won't), I wouldn't count on that to save us from RAM/GPU prices.
A few weeks before Black Friday a couple years ago, I noticed a listing on Amazon for a high-end AI card for around $40,000 (maybe a B100, don't recall). The listing had the automatically inserted Xmas season note "Free returns until Jan 31." I spent a serious minute pondering if I could float the cash for the ~40 days between the credit card bill being due and returning the card. :-)
I cant say how well that image works as I'm still waiting for them to approve my gpu access
It seems more plausible to me than wherever the frontier is at any given time getting small enough to serve on one local card directly
I usually keep under 200k if I can help it for Claude/ open ai
And lower locally on my Mac to save on prefill.