The person above is running the OS from a separate GPU, so the command can use all of the VRAM of the 4090.
replyIf you're trying to use the 4090 for your desktop at the same time, you will need smaller context size. You could remove the '-c 170000' and try turning fit on to see what fits.
>But nvidia-smi only showed Xorg using 200M out of the 24G, so why would a 911M alloc fail?
replyThat was just the last buffer allocation request that failed, it didn't tell you by how much it failed by. It could have failed it by a few kilobytes, it could have failed it by 910MB. One would guess it probably failed it by a couple hundred megabytes in the end judging by your results.
There has to be a better way of sharing config , a profile of sorts you can upload and share
replyYes, llama.cpp supports presets (an ini format). Though it's not always clear which llama-server flags it does not support.
reply