Not necessarily. Language models can do stuff—well, figure out or determine stuff—purely through inference. Give them the input on their context window and look at the output, no need to generate or run any programs.

You can take it further with a reasoning model and let it generate + consume a bunch of additional tokens before getting your output. Still not writing or running any programs.

To my understanding, that's how the examples of determining which game a small neutral net was trained to play. The weights of the neural net for into the context window, and then Astra was able to analyze them purely by reasoning rather than running external tools.