Wait; So you say you outsource development to a LLM fully knowing that it will not write sound/deterministic code and your quality requirement are your test cases, right?

I further suppose you are not writing the test cases in its whole by hand. But you try to specify them before you hand out the development task to the LLM, right?

It sounds like you just introduces a new team member which is not trustworthy yet. Normally you would review each change of him and explain how to improve hisself and the code. But that's not possible to a fixed-state LLM. That sounds exhausting.

If your Markdown Files should aim at improving the LLM contributions, you are again stuck with the fact that it did not follow in the first place.

So I conclude: You pay for an Intern who is not trustworthy and pay additional input token on Markdown Files to still no be certain about future contributions.

I just don't grasp how we as engineers are accepting this and integrate it into our craft. And: Everybody using External LLM Services, possibly providing the entire project as context, is allowed to let the source code of your company be leaked to some third party. I hope that party is trustworthy and does not have a track record of copyright infridgement. Because this would be fairly naive and reason to be fired. So we as Engineers knowing the implications should therefore point to these issues at the correct management level.

At least that's what I am doing shrugs

I think testing is really important (even without LLMs). Currently I have 2247 unit tests across 132 files in a ~150K LOC codebase (test code included in that count). It takes about 50s to run. There could be more, but it's not nothing. There is playwright for it to test drive the UI, although if I'm being perfectly honest even before LLMs I thought that kind of test tends to be brittle and annoying to write (I guess I don't have to write them anymore, but they are still brittle). I'm honestly trying to give it as much structure as I possibly can -- I'm not trying to setup the agent to fail so I can be like "gotcha!"

I'll also just point out my philosophy for using LLMs for this project, which is that I'm not trying to go as fast as I can. (I want to go at a good pace, but this isn't an experiment to just finish something over a weekend). The 150k LOC have come about since February, with some mix of me writing code and LLMs, so on average I'm probably bringing in about 800 LOC per day, which I imagine a lot of vibers would find to be glacial. To me that's the sustainable rate of what I can do when you factor in that I need to test drive every feature, make sure it doesn't conflict with another feature, check for bugs, check that the code looks reasonable, and debugging. (I also think that rate limit is specific to this project: I could see easier to test things going much faster, and harder to test things going slower)