config.toml

  model_context_window = 27000
  model_auto_compact_token_limit = 17000
  model_auto_compact_token_limit_scope = "total"

Same. I've told @tibo many times about these issues but apparently OpenAI's focus is on features that buy them the most users ASAP instead of increasing harness quality.

Start with the big idea. Pin down the product manager-level description, and maybe some details about which features are in or out, how the phased design/roadmap will work, maybe what the basic tech stack will be.

That document gets saved, and has the big picture context.

Next, we go a little deeper, flesh out schemas, APIs, a little more about what the code should look like, the overall testing strategy. This document gets saved, and the AI can reference it for context.

The next level is a very specific implementation plan. Think individual small JIRA tickets in an epic. The AI writes this based on previous context. I make sure each individual step has very explicit instructions about branch names, which branch is branched from where, the testing and documentation that will be done for that step. It should be obvious from the higher level documents, but that helps keep the AI to only worrying about the local document in its context window.

Then have a DAG of stacked pull requests designed for subagents to work through.

All of this keeps the context window down (as well as costs down, and accuracy up) and works super well, and I can get a tremendous amount of code that is easy to review through stacked PRs. Invariably I will want to fix earlier ones then percolate that down the stack, so I have some skills for that.

Every large feature has at least one of each file created for it in that order. I start by analysing stuff and making reports that might also contain design thinking and decisions. Once I'm satisfied we have properly analysed and thought though whatever it is we're working on then I might make a plan or just jump into implementation depending on how complex the issue is. And then I might have zero or multiple reviews of different kinds by different agents depending on how much verification I think the issue warrants.

This is my super simple worse-is-better development process. I make up what is needed on the spot.

It's hard to believe that sort of confusion doesn't hurt performance a bit, the question is if that degradation is worse than the performance fall off from long-context (which is highly task specific)

P.S. Love the "bonsai" name btw!

Multiple (slow) conversations seems to be the efficient path for me the past week or so

it's hard to modularize a monolith without keeping large chunks in context at first pre-delegation; the orchestrator -- however you implement it -- needs to carry as much of the original thing as possible into context and big monoliths make this task heavier.

so, tl;dr : I use a lot of tokens re-writing legacy codebases that were constructed with very little CS training in some huge spaghetti fashion by random people around the world.

that's because you and your workloads probably fall into a bucket of users where compaction is tuned well.

but the reality is that those processes need to pick what to carry over, and that's a tough thing to get right for everyone, so for me compaction is a signal to restart the session or to tell the LLM to reference existing design docs lest we go off the rails.

large context limits is one of the things that make the open model competitors very attractive to me.