The LLM is the model + context. The output depends on both.

You're making an artificial distinction. The LLM sees a new programming language for the first time, and can immediately code in it. That's learning by any reasonable definition.

If you go past the context window, it forgets, which is a limitation of current LLMs. But as long as it learned how to code in the new language within its context window, it has gained that new ability.