It's not a matter of knowledge, it's a matter of resources. It takes billions of dollars of hardware to train a SOTA LLM and it's increasing all the time. You cannot possibly hope to compete as an independent or small startup.
reply> It takes billions of dollars of hardware to train a SOTA LLM and it's increasing all the time.
replyTrue, but it's also true that the returns from throwing money to the problem are diminishing. Unless one of those big players invents a new, propriatery paradigm, the gap between a SOTA model and an open model that runs on consumer hardware will narrow in the next 5 years.
Presumably, the hardware to run this level of model will be democratized within the timeframe of the parent comment.
replySee https://amppublic.com and Stanford CS153, https://www.youtube.com/watch?v=mZqh7emiz9Q
reply