Not if it's 5-10x slower than a remote inference server. Mac prefill latency is exhausting.
replyM5 changed that a lot though - it could still be better but 4x improvement made it cross the frustratingly slow barrier for me.
replyOh tell me more about prefill latency.
replyreply
Thank you very much for this link! Extremely fascinating.
reply