Nothing Astra did is even slightly surprising or novel in mechanistic interpretability, which has been a fairly big field for a few years now.
replyYou are unfamiliar with abliteration?
replyThere are a 1000 papers published on arXiv in ML every single day. It's hard to keep abreast of everything that's going on in this space.
replyablation is not manual, and comes after evaluating which parts to turn off against given desired result.
replyit is manual, just automated via running inferences on a dataset of assumed refusals, and is a small rank-1 weight edit that doesn't require fine-tuning or training. Perf hit is much smaller than baking in these refusals in the first place.
reply