The RK3588 is a bit interesting because of its NPU. You can find models that have been converted to take advantage of that on HuggingFace: https://huggingface.co/models?search=rk3588 .
replyNo clue how performance compares. Not sure it's worth dealing with the lesser software support compared to getting an AMD mini PC and using Vulkan on llama.cpp for standard GGUF models.
I'm going to attempt to get it running on the BeagleY-AI https://www.beagleboard.org/boards/beagley-ai
replySimilar form factor to raspberry pi but with 4 TOPS of performance and enough RAM.
It depends on your idea of decent speeds and what you would use it for. I just tried it on a laptop with an AMD HX 370 running on battery in power save mode and it's not especially impressive, although it runs much better in balanced or performance mode. I gave it the prompt "write a fizzbuzz program in rust" and it took almost a minute and a half. I expect it to be pretty terrible on an SBC. Your best bet is to try it out on the oldest hardware you have and figure out if you can tolerate worse performance.
replygood idea, will test that out
replyTried with S25+ (SD 8 elite). 0.82tok/s(4B L model). It's barely useful speed but it's pretty impressive either.
replyRK3588 uses a 7 year old CPU design and OrangePi 5 looks expensive (well over $100).
replyA used sub-$100 x86 box is going to be much better
you're right. For my purposes, I was thinking of something I could use if I wanted to manufacture a new (smallish) product
reply