If you think about it, this requires recognizing something common in things which are very different: e.g. book and towel have different texture and color, and you somehow need to separate what's depicted from the background. So that's pretty much innate, core brain function. I mean, any animal with vision can recognize an object from the background - otherwise vision is useless. But for humans (and some animals) this translates to depiction of object on a flat surface very easily.
So, yeah, analogies-all-the-way-down seems plausible. Even object-vs-background and depiction-vs-paper is itself an analogy.
This is so funny to me, because as many people know, sharing an analogy with another person is the fastest way to LOSE an argument with someone, or otherwise spiral it into an unproductive place.
I think it’s Scott Adams who used to say analogies work well for explaining. They work terribly for persuasion.
This is true if the arguer is hostile, but as I've gotten older, if I get the sense that someone is entering an argument with the primary goal of "winning", I'll try to avoid that framing or just look for an offramp entirely.
Sure, the other party might think they "won", but they were going to think that anyway. For those more inclined to feel that the point of talking to each other is to learn from each other, I'll continue to use analogies and other things reasonable people understand.
I just experienced this in a conversation. An analogy offers an opportunity to engage with the straw man and miss the forest for the trees.
Ex:
Explainer: "Getting a spleen means cutting open the patient and taking it out. It's just like how I'm going to unzip this section of the patient-shaped doll, and remove this little purple bean. In both cases a hole is necessary in a similar location."
Hostile listener: "Nonsense! I can just buy beans at the store! So just buy a spleen! No hole!"
I’m not sure if Hofstadter puts it this way, but to me even the core aspects of your sentences in this post have roots in analogies. What does it mean to lose an argument or to spiral it to a different place? There is no place, there is no lost item, but we talk about these abstract ideas in ways that largely depend upon understanding things like physical objects and space and movement.
If he had accepted the "Bitter Lesson", I think he would have been at the forefront of LLMs.
The path forward all big llm providers ("ai" labs) have gone is neuro-symbolic (even though they publicly would never labeled it as such to not admit critics like Gary Marcus were right - even though all their actions actually point in that direction).
Smolensky's latest paper posted here the other day has some thoughts on how modern neural networks might beconsidered neurosymbolic, or rather "gradient symbolic processing," from another perspective entirely.
I wouldn't say the bitter lesson has given out! If you haven't noticed, these things keep getting bigger and bigger.
If the workings of those circuits are obvious to you, I'd really like to learn. Do you mean the level of analysis at https://transformer-circuits.pub/ ? (That looks like good work but not a deep understanding.)
Hofstadter referenced this back in the day as a promising beginning: https://en.wikipedia.org/wiki/Sparse_distributed_memory which sounds kind of similar in style to the embeddings you bring up.