Maybe you haven't noticed that the "Bitter Lesson" had itself a "Bitter Lesson" - that scaling pure data and compute did not lead to AGI: diminishing training returns, GPT-5 disappointment, even openAI stating it was the last 'pure scale' model.
replyThe path forward all big llm providers ("ai" labs) have gone is neuro-symbolic (even though they publicly would never labeled it as such to not admit critics like Gary Marcus were right - even though all their actions actually point in that direction).
Neuro symbolic, rly? Can you please elaborate what it is that made you conclude that?
replyI think me just means neural network models RLed to Chain of Thought reasoning? The thinking tokens are the symbolic bit.
replySmolensky's latest paper posted here the other day has some thoughts on how modern neural networks might beconsidered neurosymbolic, or rather "gradient symbolic processing," from another perspective entirely.
I wouldn't say the bitter lesson has given out! If you haven't noticed, these things keep getting bigger and bigger.
Quite a leap to call a random embedding a neurosymbolic representation
reply