I don't think its that crazy that you could look at activations and tweak weights directly to get a certain outcome, but isn't the question whether you can do that without side effects or degrading/breaking other parts of the model?
replyIf I want to do that in a generalizable way that preserves performance across all domains then am I not just back to training again?
Yep youre right, I don't see an eval on this.
replyI think you could still essentially have a really cheap Lora-Post training on a frozen base model by doing this. And instead of creating 200 checkpoints for different post-trained directions just adjust a single one on the fly.