I just used their `Bonsai-demo` repo like this;

`cd ~/Code/Bonsai-demo && BONSAI_CTX=65536 ./scripts/start_llama_server.sh`

then used it in a very minimalistic pi with a very small system prompt.

Didn't spend much time to try to optimize it tbh, but my issue was not the speed. it just could not make a decision on how to implement the task, kept going on an on.

30B is useless on 24 gigs of ram as there's ~4 gigs of ram left for everything else even with unsloth quants