Sure it could be extended to support LoRA finetuning but this implementation has the goal to be as lean and efficient as possible for a pre-training stack as you can be.