What harness/quant did you use for testing?
replynvfp4 mlx, literally barebones pi.
replyedit: on bigger tests, got it to loop pretty easily unfortunately, probably local settings.
> edit: on bigger tests, got it to loop pretty easily unfortunately, probably local settings.
replyBeen playing around for a few hours with the poolside/Laguna-S-2.1-NVFP4 + poolside/Laguna-S-2.1-DFlash-NVFP4 + vLLM, been seeing the same behaviour. Usually new model releases are plagued with issues at release though, best to wait 1-2 weeks then retry, or better yet, investigate yourself :) Personally I haven't found any obvious issues.
What inference server are you using? They have a custom branch for llama.cpp, but I wouldn't be surprised at all if it still needs fixing.
replyThis:
https://github.com/Blaizzy/mlx-lm/tree/pc/add-lg
...and this is what I should probably wait for (not sure why it's in vlm):
https://github.com/Blaizzy/mlx-vlm/tree/pc/laguna-s-nvfp4
...or perhaps I should've just used the gguf with the provided llama.cpp instead of trying to run the nvfp4-mlx from the get go, but where's the chaos in that :)
replyRunning deepseek flash on something locally now, this will have to wait a bit. I still stand by my initial quick assessment - looks capable. Some people on r/localllama also reported loops. We'll see in ~10 hours. Hopefully I haven't terribly mislead people.