undefined

points

[-]

The amount of faith a person has in LLMs getting us to e.g. AGI is a good implicit test of how much a person (incorrectly) thinks most thinking is linguistic (and to some degree, conscious).

Or at least, this is the case if we mean LLM in the classic sense, where the "language" in the middle L refers to natural language. Also note GP carefully mentioned the importance of multimodality, which, if you include e.g. images, audio, and video in this, starts to look like much closer to the majority of the same kinds of inputs humans learn from. LLMs can't go too far, for sure, but VLMs could conceivably go much, much farther.

by red75prime3 hours ago|

parent|

[-]

And the latest large models are predominantly LMMs (large multimodal models).

by D-Machine2 hours ago|

parent|

[-]

Sort of, but the images, video, and audio they have available are far more limited in range and depth than the textual sources, and it also isn't clear that most LLM textual outputs are actually drawing too much on anything learned from these other modalities. Most of the VLM setups are the other way around, using textual information to augment their vision capacities, and even further, most mostly aren't truly multi-modal, but just have different backbones to handle the different modalities, or are even just models that are switched between with a broader dispatch model. There are exceptions, of course, but it is still today an accurate generalization that the multimodality of these models is kind of one-way and limited at this point.

So right now the limitation is that an LMM is probably not trained on any images or audio that is going to be helpful for stuff outside specific tasks. E.g. I'm sure years of recorded customer service calls might make LMMs good at replacing a lot of call-centre work, but the relative absence of e.g. unedited videos of people cooking is going to mean that LLMs just fall back to mostly text when it comes to providing cooking advice (and this is why they so often fail here).

But yes, that's why the modality caveat is so important. We're still nowhere close to the ceiling for LMMs.

by throw3108223 hours ago|

prev|

[-]

> you think that enough of the complexity of the universe we live in is faithfully represented in the products of language and culture?

Absolutely. There is only one model that can consistently produce novel sentences that aren't absurd, and that is a world model.

> People won’t even admit their sexual desires to themselves and yet they keep shaping the world

How do you know about other people's sexual desires then, if not through language? (excluding a very limited first hand experience)

by red75prime3 hours ago|

prev|

[-]

> Can ChatGPT access that information somehow?

Sure. Just like any other information. The system makes a prediction. If the prediction does not use sexual desires as a factor, it's more likely to be wrong. Backpropagation deals with it.

by naasking1 hours ago|

prev|

[-]

> So you think that enough of the complexity of the universe we live in is faithfully represented in the products of language and culture?

Math is language, and we've modelled a lot of the universe with math. I think there's still a lot of synthesis needed to bridge visual, auditory and linguistic modalities though.