You could use PEFT? Operating on only a subset of weights is fairly standard practice nowadays …