It bundles tools, model selection, and overall management.

It’s portable in the sense it will install on any of the supported OS using CPU or vulkan backends. But it only supports out of the box ROCM builds and AMD NPUs. There is a way to override which llama.cpp version it uses if you want to run it on CUDA, but that adds more overhead to manage.

If you have an AMD machine and want to run local models with minimal headache…it’s really the easiest method.

This runs on my NAS, handles my home assistant setup.

I have a strix halo and another server running various CUDA cards I manage manually by updating to bleeding edge versions of llama.cpp or vllm.