I agree, this is important context. They call it out explicitly. However, even identifying clusters that specialize in certain fields automatically and then finding and fixing learning “errors” is incredibly impressive to me. Especially for medium sized models between 20B an 70B parameters, this could yield huge capability and efficiency improvements, not to speak of the potentially enormous downstream effects that come with interpretability.

Good points - I’m still shook