upvote
I'm hoping that by "gets it," he meant that if you start a discussion about the design, it doesn't misunderstand and immediately go off to do the work. Some models tend to do this.
reply
Whether or not a distinct "plan mode" is needed, upfront planning remains essential in my experience, even with Fable (albeit not the 5.1 version). I agree that, as the models get better, you can skip planning on increasingly complicated tasks.

But there is still a ceiling above which it is necessary to "preload" the context window before starting to call tools and get into the meat of the work. You want to establish domain language (especially with Claude models which otherwise will invent their own, and it will be inscrutable) and key requirements and assumptions. You want to do a Q&A iteration cycle with the LLM. You definitely should do a sanity check that the LLM actually "understands" what you were trying to achieve, and then make sure that understanding is coherently and plainly stated in the prompt. All of that seems to be necessary still for just about any serious task, if you actually care about the quality of the results and/or don't want to burn hundreds of thousands of tokens on flailing around to get to a good quality result.

So no, you don't "need" plan mode. But you do still need to do all of the things you would do with plan mode.

reply
But in your example you never even asked it to plan so you could check the implementation before writing it you just asked it to write it directly, so this isn't even a comparison to plan mode
reply
This is a pretty obscure and in-the-weeds benchmark, but to me the models’ interpretation feels quite reasonable.
reply
Apologies, I didn’t mean to imply it’s a benchmark, I just wanted to provide a reproducible example of where I see models make decisions that seem to be fine initially but might paint the software architecture into a challenging corner. I don’t expect models to read my mind, but I do see them produce a lot of verbose output, none of which is used to say “here’s a simple response to your ask, but have you also considered...”
reply
It's obscure to use common functions from the standard library?
reply
deleted
reply
It’s just an odd way to judge whether an entire model “gets it” in my opinion. There are so many variables.
reply
I think knowing which models use the standard library and which ones pull in dependencies as highly useful. As a dev I've always favored staying as close to the standard library as possible as it makes refactoring, understanding, and deletion much easier. Not everyone has the same preferences as you, it's nice to understand what matters for others too.
reply
It can't read your mind that you have an HSM, can it?
reply
[dead]
reply