I think it's from lyra's neuron-explorer project https://x.com/_lyraaaa_/status/2113833330429083662 (see also https://neuronary.org/about, https://github.com/lyramakesmusic/neuron-explorer)
replyThere was a paper that popped up a month or two ago (I think published earlier though) about being able to inject "thoughts" into the neurons. E.g. Ask it what I should eat, and steer it towards "bread".
replyWell I think the first big publication on this front was Anthropic’s Golden Gate Bridge neuron.
replyThis was not a neuron. This was from a trained sparse auto encoder.
replyIn a few words, they found a way to map residual stream activations (so, a vector) into a few interpretable sparse concepts. They also had the converse maps, that translate concepts into activations.
Golden Gate claude was obtained by artificially increasing the activations in the direction of the 'Golden Gate' concept. It was not neuron surgery.
You don't even need SAE for residual stream based steering, you can do so much more naively by just taking the average difference in activations of prompted vs. unprompted versions, then PCA to get a "control vector" that you can use to elicit the prompted behavior
reply