upvote

points

by rahen12 hours ago |
comments
upvoteby WarmWash3 hours ago|
next
[-]
The quasi-mythical memristor would be choice for bread boarding a brain. However I suppose you could train a model and then manually place fixed resistors to build the network
reply
upvoteby Schlagbohrer5 hours ago|
prev|
[-]
Amazing and anachronistic to see something like that from 1960. And then it makes me wonder why there wasn't more progress on neural nets being used for many things prior to the 21st century. (I haven't read the history of the AI winters but I have heard of them)
reply
upvoteby rahen4 hours ago|
parent|
next
[-]
The first AI winter was largely triggered by Minsky in a book he published in 1969, which mathematically proved that single-layer perceptrons couldn't solve non-linear problems. Favorite quote: "Our intuitive judgment is that the extension [to multilayer systems] is sterile."

Yet we had the computational power to run backpropagation in the 1960s and small Transformers in the 1970s (I'm the author of both):

https://github.com/dbrll/Xortran (backprop on IBM 1130, 60s)

https://github.com/dbrll/ATTN-11 (Transformer on PDP-11, 70s)

What was missing wasn't the raw processing power, but the ideas and algorithms themselves. Because funding and research were completely discouraged during the AI winter, neural networks research was left dormant and we lost two decades.

reply
upvoteby WarmWash3 hours ago|
parent|
[-]
I wonder had we invented transformer architecture back in the 70's or 80's, if the pace of hardware innovation would have naturally slowed AI progression, and given humans decades to slowly adapt, rather than the current tidal wave (that seems to grow in size daily) bearing down on us.
reply
upvoteby mr_toad5 hours ago|
parent|
prev|
next
[-]
> why there wasn't more progress on neural nets being used for many things prior to the 21st century

They were simply too computationally expensive to train for the limited things they could do. It wasn’t until we had the ability to train large neural networks on commodity hardware that things really took off.

reply
upvoteby j_bum5 hours ago|
parent|
prev|
[-]
This doc on Ilya Sutskever & Geoffrey Hinton gives a great background on the progression of deep learning over the past decades [0].

Tl;dr - compute was the bottleneck.

I am not associated with this channel/video, just love it. I’ve shared it here before.

[0] https://youtu.be/glWvwvhZkQ8?si=XjcwWWy43305tl6O

reply