> occasionally we brainstorm.

Seriously, how do you brainstorm with an entity that would 180 if pushed a bit..How can you take anything it say at face value?

I've been on a similar path, and I used to put time in high level design, as it was basically required in some cases. But not a lot of time, because smaller agents research, perform small tests, whatever to inform the high level design. Working incrementally seems to help, maybe prompting a bit more often, but the same can be accomplished with well defined checkpoints.

Then generalize + standardize that process, get an agent to replace you as high level designer so you can manage a team of high level designers, etc etc