undefined

points

[-]

I think it’s because maintainability is a factor of software design, and good design is squishy and hard to articulate. There’s no good metrics for design. All the tools that purport to measure it are doing trivial things like cyclomatic complexity, and can only tell you when things have gotten really really bad, but not distinguish between “good” and “pretty bad.”

Design skills take about 10 years to develop, in my experience, and a lot of smart engineers go through a phase where they make gawdawful messes in the name of “magically easy” designs around the 5 year mark.

Design requires wisdom born of experience, isn’t measurable, and requires forming a mental model of how the system hangs together as a whole. It’s just not stuff LLMs are good at.

by jadar41 minutes ago|

parent|

[-]

> It’s just not stuff LLMs are good at.

But then you're back at the question from the OP of "why not?" Is it lack of good examples in the training data? Is there something of code "goodness" that goes beyond phenomenon that are measurable? Is it that LLMs lack in the ability to organize and think abstractly?

by Fordec1 hours ago|

prev|

[-]

I have a hunch it comes down to the training data. We have spent decades as an industry talking up the new shiny thing, deliverables, frameworks, features, performance improvements, the one thing you don't read copious amounts of prose about is "how I maintained our system so that nobody noticed".

by dhorthy2 hours ago|

prev|

[-]

fair point, this is the thing I struggled most to extract out while writing it - if you can propose an RL environment that penalizes a model for bad design, then I'm all ears - right now there's no fast oracle/verifier for this (as stated in the post)

My current evolving take on "how would you build such a thing" is you need to tee up a roadmap of 20 features and feed them to a model one at a time, so it can't design up front for what's coming.

That way if it builds the first 10 features and the codebase goes to slop, it get's penalized when it can't build features 11-20, or when those features take wayyy more tokens/time/cycles than a model that maintains a clean codebase can do.

This is how most real software is built by most teams - incrementally, getting feedback from users along the way, and steering goals in response.

by jadar1 hours ago|

parent|

[-]

Isn't that how AI written software gets better too? By steering the model towards the goals of the user?

I don't know if it's a question of how many features to feed to the model, either. Of course, overwhelm the context with too many features and it will get confused. But that's where the memory management idea that G. Huntley talks about is helpful. You're trying to steer the model within its memory limits towards a certain goal.

The problem is getting it to produce "good" code. Formalizing what that means is the task of the programmer. How do you steer a model towards always, or more often, producing good code so that you don't have to do rework? That's the same problem as with a junior engineer, but the way you do it is different. Right now we're trying to do it with mountains of prompts — which sort of works but has diminishing returns — and with onerous code reviews. We've seen this get better over time, but I think some more mechanical methods will help as we figure out how best to steer the models.