> invalidates my initial idea using some obscure criteria than changes direction
replyYou might be giving it too much leeway.
Opus and Fable are more trustworthy, but even today I find they very quickly go off the rails for research tasks if I don't give them short chunks of work.
They're not ready for prime time automation of research (and before someone mentions the math results, that required absurd amounts of computation time and has immediate feedback in the form of Lean)
Yes, I find DeepSeek can be very precious about its ideas and I have to insist multiple times to do the experiments I ask for despite it thinking them not worth doing. Often my experiments do give good results and I swear the model gets salty about it.
replyI guess when you ask these things to take on the role of a senior scientist it picks up some natural stubbornness in the training data.
When it comes to difficult optimization/search problems, doing research in a single Claude/Codex thread is definitely a struggle. There are ways to bypass some of the pitfalls by creating a agentic swarm with hard guardrails such as context resets and changing the way the harness works with memory, etc.
reply