> say you are forecasting traffic for how to scale a load balancer, it can be a bit off

Problem is you really don't even need ML for the curve fitting tools. Just ARIMA or Holt Winters is enough and likely faster. Heck, Kubernetes HPA doesn't even implement full PID, just P.

It seems like the main challenge is going to be finding domains where accuracy is important enough to spend inference token money on, but not so important that you'd hire a team to build a custom model. Or maybe that team borrows Timestral weights to fine tune to the domain, but they probably already have a different model customized to the problem?

Agree on the share-buying point. I keep a side project that tracks about twenty published bitcoin price models side by side and grades each one on its own forecast record. Two lessons carried over to any time-series benchmark for me. First, freeze every forecast as it stood the day it was made; refitting on later data flatters everything. Second, put the naive baseline (tomorrow looks like today) in the same table. The best-known model in my set, stock-to-flow, sits more than 100% off over the past year, and it looked brilliant in backtests. A 0.03 gain means little to me until I see it next to persistence.