I don't consider investing $20k into hardware to run SOTA open models, that are far behind proprietary SOTA, to be competitive.
replyEven if open models were competitive, it's still typically going to be more expensive than a cloud provider because of low utilization and higher purchase price.
Last time I checked it costs closer to 500K to run SOTA open models at any usable speed.
replyHow do I know that a private LLM system won't have a degradation of quality similar to this or worse? The only thing I can think of for the proposed scenario is some sort of homomorphic encryption system? But not sure.
reply>How do I know that a private LLM system won't have a degradation of quality similar to this or worse?
replyYou use benchmarks, you test, and because YOU'RE the sysadmin you know what weights are running at what time, it's very visible. You can airgap the hardware and be guaranteed it won't change over time. And, frankly, degradation over time doesn't seem to be what's happening with the open models.
Right now, I would be willing to pay ~$20 extra per month for strong privacy, assuming the same capabilities as Opus 5.5. I don't see local models making sense economically any time soon unless you value privacy at $1000's per month or are fine with much lower performance on hard tasks.
reply