Surely if you end up relying on a given prompt to produce the exact same code every time you should instead just check that code into source control the first time you generate it?
replyA deterministic LLM isn't going to behave appreciably differently from a non deterministic one if your input or context varies by even a tiny bit (pun intended) each time.
If nothing has changed, caching the result would certainly be cheaper. But if you're doing that as part of a test, it's not really running the test and it might defeat the purpose of the test.
replyi tried to create a makefile driven workflow based on this idea and ended up with https://github.com/khimaros/enc -- it suffers from the issues you raised
replyi'm hoping that it becomes more useful as models improve and become more reliable at producing working code (though determinism would be great for improving prompts).