Yes, but doesn't the token change mean that?
replyYou can train a tokenizer on old data just like you can train a model on old data.
replyBut you can't use an old model with a new tokenizer. Changing the tokenizer implies you trained the model from scratch
reply[dead]
reply