In addition to the library bindings, we have a small library of tools (workspace for view/edit, scratchpad for persistence) and making your own is registering a Go function. And in recent weeks, I added the Vision and Qwen support, as ds4 added them.
Even if you don't use the Go library, the ds4go binary makes it really easy to download the libraries off of HuggingFace with a TUI available vie Homebrew.
Here's some TUI toy screenshots, sorry I still haven't released that code; it's of different quality than the others. [3]
EDIT: add ds4go TUI screenshot gist [4]
[1] https://github.com/NimbleMarkets/ds4/releases/tag/v0.8.20260...
[2] https://github.com/nimblemarkets/ds4go#install
[3] https://gist.github.com/neomantra/ae47422c8daf7a458212c93992...
[4] https://gist.github.com/neomantra/40180ade13df93290250ce8c6d...
The project GitHub page is a much better introduction for the hn crowd.
From the github repo it seems like you really don't need a big Mac with huge amounts of RAM but SSD is sufficient.
If this is anywhere near 50 TPS, that would be a game changer in the personal LLM space!
I'm also looking into expanding the protocol and the engine to support various steering techniques.
I just had to do a little patch to support my iGPU device that is a bit newer than Intel Xe-LP, maybe I'll do a PR.
On a side note the other day I was experimenting with Sonnet 5.5. I gave it the llama cpp repo and told it to extract in a single file inference for a single model + backend (qwen3.5 4b mtp + sycl) and (after a long time) it actually worked! It produced a ~1400 lines file with no deps. I need to check the quality of inference yet but I think this is still a great achievement.
I'm pretty sure 2027 will be a very interesting year for local models and inference.
If your GPU supports XMX we could also explore using it to improve the prefill kernel, but I don't have the hardware to test it myself.
Maybe Intel and AMD should help them with that.
I heard antirez saying that he designed DwarfStar also to be forked and tuned to everyone's specific needs. Do you have a specific machine/spec in mind?
Now Ive been running qwen 3.8 flash next for more than a week and it’s doing great, really fast and super long context windows. Sometimes the model is behaving stupidly by not remembering something I said earlier but it could be also a problem from the agentic AI harness. Im using oh my pi but Im wondering what people are using ds4 with here ?
small native inference engine optimized first for DeepSeek V4 Flash (including the experimental vision model), DeepSeek V4.1 Flash (Metal, and text inference on CUDA), and additionally GLM 5.2 and 5.3, GLM 5.3 Flash and DeepSeek V4 PRO, and Qwen3.8 Flash Next (Metal and CUDA)
For our mere mortals that were kids not long ago and can't really believe we've got our hands on a x090 series targeting Qwen3.8 27b, https://github.com/noonghunna/club-3090 is the way to go.
I'm maintaining a web frontend for this, trying to at least. You can follow it here: https://github.com/gchamon/club-3090-server
DwarfStar's selling point is that it only supports a small set of carefully chosen models, but it supports them really well.
Also the goal of the project is to squeeze the absolute maximum performance and capability possible out of limited hardware resources (compared to clusters of B200s or something).
Does Rust even give you good access to low-level code on different platforms? And if so, how much extra work do you need to do to make it acceptable to the compiler? And is that work worthwhile if you are not going to get the security guarantees of normal Rust code? Is it a worthwhile tradeoff when the goal is performance?
Those are real questions by the way, not rhetorical. If Rust could work well for this type of project then I would like to know.
It's about as good as it can get for this kind of code.
[1] https://www.reddit.com/r/rust/comments/1ixt1ei/zlibrs_is_fas...
This is explicitly an AI-coded project, Antirez argues that LLMs are worse at writing Rust than C because so much high quality systems code (think e.g. sendmail) that ends up in AI training sets is C, not Rust. Another related argument is that the more detailed syntax and compiler feedback found in Rust compared to C are really a negative for LLM workflows.
There's plenty of room to disagree wrt. this of course: without the strong typing checks of Rust around e.g. indirect references, safety and correctness ends up being a global property in typical C programs, and LLMs are terrible wrt. reasoning about global properties. You're better off forcing them to adapt to a different local syntax that does a more complete job of enforcing modularity, since this is comparatively foolproof.
Much easier to work with a language you are most comfortable with right?
The video is in Italian but has an auto-dubbed English audio track: https://www.youtube.com/watch?v=sOt0WpQG5eU\&t=526s
the ds4 quants were very good beating the unsloth quants https://github.com/michaelasper/benchmarks/blob/main/deepsee...
Tensorfold is getting 100%+ speed increases on both prefill and decode for models like Qwen 27B. oMLX has followed them and have had similar improvements in the past week.
There's lots of different techniques like letting CPU help with prefill, DFlash specualtive decoding etc.
I'm really excited for this as I'll be receiving an M5U in about a month. Expect to be running Qwen 4 27B or Flash (it's a 96gb machine), and they may come close in performance to DS 4/4.1 Flash, and should be able to hit 100 tps. Local is really becoming viable, especially considering that GPT 6.1 has been running at 20ish tps the past week.
What are we going to name the company, how about Dwarfism 2.0? What happened to 1.0 Jared?