But why would you throw away that data at all? These are tiny models compared to domain-specific audio/music/image models, and any capacity spent on recognition is capacity that could have been used for prediction.

Is there a reason you don't throw in something like cross-attention on an embedding of the sequence description/data labels? I've trained numerous generative models for audio and images and it's always the case that the unconditional model performs worse, trains slower, etc. than the conditional one (with the best performance from a hybrid using classifier-free guidance or similar scheme).

Again I don't have experience with the domain of time-series forecasting directly, but I would think there are generalities between autoregressive generative models and time-series forecasting, given that both involve predicting the next sample(s) given N regularly sampled past inputs.

Do you actually find that for some use cases foundation models work better than AutoARIMA or etc? I do a lot of ecom projections and it’s never been the case for me, most of my cases are best with seasonal naive.

Would love to know actual time series foundation model use cases!

Exactly this. It's a zero-shot model that can understand what a time series is with very little a-priori information. I would assume they've trained across a wide diversity of real world signals.