upvote

points

by Schlagbohrer3 hours ago |
comments
upvoteby aurareturn2 hours ago|
next
[-]
It is both a software and hardware problem. Software because you can train LLMs that get better at very large contexts. Hardware because no matter what you do in software, you still need faster and bigger chips.

Yann LeCunn has been very wrong in the past about LLMs.[0] The approach he wants to take is to train using sensor data in the physical world. I think it's going to fail because there's near infinite amount of physical data down to Schrodinger's equation on how particles behave. There's too much signal to noise. My guess is that they'll need magnitudes more compute to even get something useful but they do not have more compute than OpenAI and Anthropic. In other words, I think LLMs will generate revenue as a stepping stone for OpenAI and Anthropic such that they will be the ones who will ultimately train the AI that LeCunn dreams of.

[0]https://old.reddit.com/r/LovingAI/comments/1qvgc98/yann_lecu...

reply
upvoteby zaphar1 hours ago|
parent|
[-]
I don't know. Some of those statements still look correct at the time they were made and then people found out how to work around them. I don't think anyone has shown his general assumption is wrong really. The issue is we don't know what the ceiling is for these things is yet because we haven't hit it. But that doesn't mean there is no ceiling.
reply
upvoteby aurareturn1 hours ago|
parent|
[-]
His generation assumptions were wrong. That's the point.
reply
upvoteby zaphar1 hours ago|
parent|
[-]
I haven't seen any indication that they are. Can you point me at some?
reply
upvoteby AntiUSAbah2 hours ago|
prev|
[-]
Nvidia has so much money, it would be a waste if they wouldn't attack current problems on multiply points at once.

People, Researcher, Investor etc. probably also want to see what would be possible and someone has to do it.

I can also imagine, that an inferencing optimized system like this could split the context for different requests if it doesn't need to use the full context.

Could also be that they have internal use cases which require this amount of context.

reply