upvote
It's less than 24 hours old. The inference engines haven't been optimized for it yet.
reply
I’m getting ~6.7 tok/s out of it on an a laptop RTX 4060 with 8GB VRAM — over twice the speed I can get out of qwen, and arguably fast enough to turn it loose on real work. For a minimal hardware setup, this is easily the best speed for performance I’ve found in a model. The nemotron models are the next best, they are much faster, much more verbose, and much less accurate.
reply
I'm so sorry you're getting how many tok/s?
reply
According to TM, the creators, they benchmarked it to run at 5x the speed of Qwen 27, so there is probably some issue on your end. It runs a fraction of the active parameters, even with looping, so it should be much faster.
reply
This has been observed by the broad community for llama cpp.
reply
llama.cpp is one of the slowest these days
reply
What alternatives do people use instead? I hadn't heard of any others (except I guess ollama, which I thought no one used anymore)
reply
Depends on your hardware, but this build of [ninfer][1] runs inkling-2-mini for me at 77 tok/s versus llama.cpp at 30 tok/s. So far it has been pretty good. Solved some things that qwen3.8-27B could not.

[1]: https://github.com/iamwavecut/ninfer-all

reply
Ollama is just worse (well, depends on your use cases) pre-configured llama.cpp, I believe. I barely know anything about local hosting, but you might wanna look into vLLM (cross-architecture) or MLX (for Apple Silicon).

Also, although not necessarily for this case, there are some inference engines made for just one model/model family, like antirez’ DS4.

reply
Can you autoresearch optimize it?
reply
The NV in NVFP4 stands for NVIDIA. :)
reply
It’s just not optimized in the same way. A little tweaking on my vllm fork and it runs faster on my 7900xtxes.
reply