The "new data" is: person A prompts an LLM to create or modify a tool, person A distributes code person B, person B's LLM uses the tool via docs/help/error. That is a direct path for an LLM to "know more" from a human than what's in its training data.

If we get a new programming language not in the training dataset, we could give an LLM a decent compiler with compile errors, and some sample code and it would be able to write code in the new language without training.