Much of the cognitive work in writing emerges in the labor of writing prose and challenging assumptions against the model you build in your head as you go. I've found this process is inverted when working with an LLM: it in effect emits a provisional structure first, and I discover what I actually think by finding where its output is vague, overconfident, incomplete or outright false.
Accepting surface-level coherence as finished thought is the failure mode to avoid.
1. Throw some notes, PR links, etc. into copilot and ask for a first draft
2. Go back and forth for a while asking for revisions
3. Trash all of the AI-generated text and rewrite the whole thing myself from scratch, which I can do quickly in a typing frenzy because I now have a clear idea what I want, by way of seeing what I don't want.
I was thinking more when it comes to stuff other people send me (like a freelance writer) and “editing” is basically writing the first draft because no cognitive effort has yet been spent on the piece
LLMs lie in the unhappy-medium between an abstract machine we can reason-about versus a person we can instinctively model and simulate.
Instead it's a complicated machine that evades both reasoning and intuition.