Both of your comments are illuminating :p
replySo, we could technically debug a prompt's output? I get that there are too many steps to actually step thru, but what if there were checkpoints? At least you could isolate behaviors to specific sections of a neural network?
Of course. And mechanistic interpretability research is a thing.
reply