> Training a one bit neural network from scratch is apparently an unsolved problem though.

I don't think it's correct to call it unsolved. The established methods are much less efficient than those for "regular" neural nets but they do exist.

Also note that the usual approach when going binary is to make the units stochastic. https://en.wikipedia.org/wiki/Boltzmann_machine#Deep_Boltzma...

> Training a one bit neural network from scratch is apparently an unsolved problem though.

It was until recently, but there is a new method which trains them directly without any floating point math, using "Boolean variation" instead of Newton/Leibniz differentiation:

https://proceedings.neurips.cc/paper_files/paper/2024/hash/7...