Because if each neuron represents a small set of similar concepts instead of several completely different things, it should be easier than expected to look inside the neural net and see what it's really thinking.

It might still be quite difficult since the network is huge, and just because you know what the individual neurons represent doesn't mean you understand the complicated thoughts they're building. But easier than expected is still good news.

The big win would be, "Here's the deception neuron, here's the sycophancy neuron, here's the prioritize my own goals over the goals of the user neuron, here's the seek control over my environment neuron."

How much would it cost to find those things, if they exist? For using the blog post's methods to label every neuron in modern arch, Claude and Gemini produce similar back of the envelope math that looks something like, within an order of magnitude of the resources it takes to pretrain the model in the first place. If you have a better way of narrowing down the set of candidates, probably a lot cheaper.

Would that be reliable and useful? Well.