upvote

points

by redox9916 hours ago |
comments
upvoteby rbehrends16 hours ago|
next
[-]
If I'm not "demanding enough" from models, it's precisely because they are imperfect. Having them do long, complex tasks autonomously usually comes with more cleanup work afterwards IME.

My usual workflow (not counting throwaway code) is to use agents as virtual pair programmers, which gives me a good balance of quality and velocity.

Each interaction with the agent remains individually comprehensible and makes progress towards one of the following goals:

- Understanding the problem/solution space.

- Implementing a solution.

- Understanding the solution, especially any agent-written parts and interaction with existing code.

- Evaluating the solution (tests, code analysis, what-if scenarios, consider edge cases).

This does not just involve the model generating code, but also explaining or analyzing it as needed.

This process allows me to retain a mental model in lockstep with developing the written code and have some assurance both of its functionality and non-functional requirements. It also fits in nicely with most existing team-level workflows.

But it's also something that I can do with pretty much any of the modern models.

reply
upvoteby redox9915 hours ago|
parent|
[-]
Well that's kind of my point. Models still have a lot to improve. They are still imperfect enough that you don't trust them and have to use them as pair programmers.
reply
upvoteby rbehrends15 hours ago|
parent|
[-]
Yeah, but that's also a large part of why they remain interchangeable, I would say. I feel like we may be talking past each other?
reply
upvoteby redox9915 hours ago|
parent|
[-]
They are interchangeable maybe in your case. In my case for my tasks and for my product, Opus and GPT are at a level where I can vibe code with them and trust them, whereas the others aren't really there. When models get better they'll reach whatever point you need to be able to trust them.
reply
upvoteby rbehrends15 hours ago|
parent|
next
[-]
You said above that "Opus 5.5 still makes a lot of mistakes". This seems to contradict what you are saying here?
reply
upvoteby redox9913 hours ago|
parent|
[-]
Sorry just woke up, I wasn't being clear.

There's two thresholds.

1. Tasks models can do. 2. Tasks models can do with enough reliability that you can YOLO merge without looking at the code.

Most models can do most of 1. That's no longer the interesting part.

2. is quickly expand. I can trust opus 5.5 enough with easy to medium tasks that I can YOLO merge them. For open models, I can only trust them with easy tasks if I want to YOLO merge.

I'd love to have a stronger model that I can trust to YOLO merge hard tasks.

reply
upvoteby rbehrends56 minutes ago|
parent|
next
[-]
I think this is where we may have to agree to disagree, because this not something that I'd do outside of toy projects. Not necessarily because I'm worried about the correctness of a single MR (in fact, it may very well be more likely to be correct than a human-authored and -reviewed one), but because I'm concerned about the accrual of technical debt. Even if each individual MR is correct, the trajectory for code quality can trend downwards.

The recent update to SlopCodeBench that includes Astra and Opus 5.5 (including the insight that higher reasoning levels may not improve the situation) seems to match my intuition. [1]

[1] https://github.com/humanlayer/advanced-context-engineering-f...

reply
upvoteby dca887 hours ago|
parent|
prev|
[-]
I am genuinely curious, at what point did you find yourself comfortable “YOLO merging without looking at the code”?

I struggle to find myself ever feeling comfortable doing this in a production environment at my day job.

reply
upvoteby redox992 hours ago|
parent|
[-]
Probably around GPT 5.6 Sol.

These days your average Opus 5.5 PR with multiple passes of review by Opus 5.5 and Sol 6.1 agents is much less likely to contain bugs than your pre-AI PR.

reply
upvoteby agile-gift026213 hours ago|
parent|
prev|
[-]
But sounds like Opus and GPT are interchangeble for your case.
reply
upvoteby redox992 hours ago|
parent|
[-]
Both have good models. My workflow these days is usually Opus plans and codes, then Opus and Sol review, finally Opus verifies the review and applies fixes.
reply
upvoteby NoDodgeQuestion16 hours ago|
prev|
[-]
Do you have example, open source you make using non-interchangeable model, that you can not make using other model?
reply
upvoteby redox9915 hours ago|
parent|
[-]
That wasn't my point, my point was that open models can technically do everything but they make mistakes more often. As an example imagine Opus makes a mistake every 5 prompts and Deepseek every 20. Because you still have a human in the loop, you tell it to fix that mistake and it does, but it's time wasted or bugs sneaking up on you.

Having said that, I was working on an AI for a certain video game, and I had Astra 6 and Sol 6/6.1 work for 30 hours straight, with some human steering to get it unstuck every now and then. It could not get it to work right, and the code it created was a mess.

When Opus 5.5 came out in like 4 hours it cracked it.

I didn't try but I'm very confident if Astra failed, the open models would've failed too.

reply
upvoteby NoDodgeQuestion14 hours ago|
parent|
[-]
Less time wasted or bugs sneaking are also good example, can you share video game AI example?
reply
upvoteby redox992 hours ago|
parent|
[-]
Sorry it's proprietary but imagine a bot for a multiplayer videogame that can replace real players who disconnect mid game.
reply