To reduce system prompt and tool bloat, use a minimal harness (I wrote my own, but I've read that pi.dev is okay, too).
To make your prompts more precise, tell the LLM which files it has to read (or at least where it should start), so it does not have to search as much. This also reduces the change of misunderstandings and makes the LLM adhere to existing practices.
To keep your code base in check, tell the LLM (in a new session) to review the code and refactor from time to time.
When a task is done, start a new session. If you find that you have to repeat a lot of information in your next prompt, put the information in a file so you can reference it in the future (aka documentation).
I don't have a harness or prompt. Just VS code integration through my company. All my prompts are from scratch and the only context it has.
Lately I haven't been saying go to X file and change Y, I say you have SSH keys to the embedded hardware: plan, deploy, and test latency improvements to the X stage of the image processing pipeline, parity test output images to avoid regressions, and target a 5ms latency improvement, ideas to look at are M, N, and P image processing techniques, I think latency is in this domain is mostly caused by Y, but verify, and let me know when you have a plan ready to implement.
Then it churns for 4 hours, it builds and runs it's own test harness and profiler, and autonomously works until it has a changeset ready to go.
That's how I get to 600k, and it seems to work fine? It got us 4.8ms today, through SIMD improvements and using a technique I hadn't thought of in place of a more expensive operation.
Eventually, the context window got so bloated that they resorted to hiding function bodies in large files, which is of course a stupid idea because then the LLMs have to use other tools to read the files, wasting even more tokens, or hallucinate the content. Honestly, it is amazing that LLMs work at all in VS Code.
You can inspect the context by pressing F1 and then selecting "Developer: Show Chat Debug View" in VS Code (https://github.com/microsoft/vscode/wiki/Copilot-Issues) and marvel at all the garbage that is in there.
Make sure to regularly validate that what you think you understand is still what you actually understand. One can only safely offload so much.
By breaking the problem into discrete steps and aggressively restarting the prompt from the current state after completion of said steps