I think it is due to the new website? it now looks like every other vibe coded site,the only upside is that is looks more saleable for people unfamiliar with it, e.g., explaining OSPO,IT the stack you are using. they should also add a pricing page for eenterprise where they promise 99.9% uptime for local models*.
replyWow it’s aggressively vibe coded. Nothing inherently wrong with that, but it looks a bit amateurish which is funny.
replyI’m still waiting on 98.css to become the standard for vibe coded sites. You don’t have to read docs anyway if you’re just using LLMs! All you have to do is say “use 98.css” and you have a 10/10 site
Wow, gonna use that. Thanks
replyAt this point the options are llama-server or vLLM if you're serious about running things at your desk in the under 256GB RAM size class (70B, 120B size models). In addition to, of course, 27B to 35B size things. With of course a ton of compile time build customization options for whatever specific hardware platform you want to run either llama or vllm on.
reply> At this point the options are llama-server or vLLM
replyWhich last time I checked, both use different formats of the weights, the former GGUF while the latter .safetensors. I mostly end up using vLLM these days and I'm a bit more performance sensitive than what I used to be. Just a shame it's a hassle to share the weights between them with conversion and what not, either batched or on-startup.
possibly hitting front page because this website is fairly new? For me, it's certainly the first time I've seen a one-liner curl|bash installer for llama.cpp, which was basically the only reason to use ollama.
replydoes your comment depend on the OS? I thought MLX has better performance on MacOS than llama.cpp
replyThe gap was MUCH larger in the past, but in my tests, oMLX and llama.cpp are now very similar (within 10%) in both prompt processing and generation speed. GGUF ecosystem provides a better selection of quants, in my experience Unsloth ones are excellent.
replyI thought the main advantage of oMLX is it's less likely to invalidate the KV cache when working with coding agents, which is key when working on a Mac because of the slower prompt processing.
replyVanilla llama.cpp leaves a lot of performance on the table. I'm reaching 120 t/s with a custom inference engine for a model that llama.cpp can barely run at 70 t/s. Theoretical maximum on this hardware is around 147 t/s according to measured memory bandwidth.
replyJust run /goal to optimise it and you should be good in less than an hour. Also best to use models that support speculative decoding.
replyOptimize llama.cpp? Hmm.
replyWrt speculative decode, basically zero finetunes keep it. I'm testing with some ridiculous abliterated amalgamation so spec decode has been gone for most of its ancestry.