upvote
When I worked at a particular fruit company in Cupertino, my boss had been asked repeatedly by the developer teams to provide a tour of the data center we'd recently finished building out, which housed the servers their code ran on.

He gave them a lengthy, grueling, hyper-detailed tour of the entire facility, encompassing the HVAC systems, electrical systems, network and computing systems, finishing with about 20 minutes where he had them stand inside a hot aisle that he was just outside of, giving a fantastic soliloquy on the importance of code efficiency, and the consequences of ignoring it. It was hilarious to watch from the comfort of the cold aisle, knowing full well what he was doing.

reply
What did the software engineers hope to learn from this tour?
reply
The scale of a cloud datacenter is pretty awesome to witness. I had the opportunity to visit a large cloud company’s data center years ago, it was immense in scale and (mostly) very well thought out.
reply
What was he doing? Showing them that bad code will make them feel hotter?
reply
Busy servers generate more heat. Inefficient code makes servers busier. There’s a real physical outcome for bad performance code at scale.

But you don’t really want lots of idle or underused servers, aside from burst capacity management. So if they universally and consistently wrote more efficient code I’d expect a smaller datacenter. Full of still busy servers but just less of them.

reply
He was showing how complex and expensive a data center is and giving them real-world visceral experience that inefficient code has real-world implications.
reply
Maybe he got PTSD from being a former tour guide and this was his time for vengeance.
reply
Exactly. For most of my career, bad engineers (or juniors who were earnestly learning) would struggle to even create output that compiled, let alone ran. And it would take them a long time to implement something poorly. So the blast radius from their incompetence would be limited. Today, a bad engineer can churn out ten thousand lines of garbage that actually compiles and runs, before my first cup of coffee. If not on a tight leash, they'll sink the whole codebase or inundate every senior person with garbage code review requests. The blast radius is unlimited!
reply
> Today, a bad engineer can churn out ten thousand lines of garbage that actually compiles and runs, before my first cup of coffee. If not on a tight leash, they'll sink the whole codebase or inundate every senior person with garbage code review requests. The blast radius is unlimited!

The term for that is "10x AI engineer." Anyone who has anything negative to say about such people is just jealous of their insane productivity and speed.

reply
The supposed “10x engineer” may simply be someone stealing productivity from everyone around them.

You measure the output of one person while ignoring the cost they impose on their team.

If I generate 10 PRs in a day but three engineers now have to spend the next two days reviewing them, figuring out what I changed, correcting bad assumptions, debugging regressions and explaining why half of it needs to be redone, I haven’t become 10x more productive. I’ve just moved the work onto other people.

Worse, I’m consuming the time of the people who are usually the hardest to replace and whose attention is already scarce.

That’s why PR count, lines changed or features “completed” are terrible measures of productivity. You can make your own numbers look incredible while reducing the throughput of the entire team.

reply
FYI, your comment showed up as [dead] like you were banned until I vouched for it, but I don't see any reason for that. Just thought I'd let you know in case you want to follow up.
reply
> You can make your own numbers look incredible while reducing the throughput of the entire team.

Taken how some companies award promotions and bonus this is actually a double win. You not only improve your own numbers but also make this of your competition worse! /s

reply
If your company has bad engineers cranking out 10,000 line PRs your engineering culture and product was already bad, I guarantee it.
reply
It's not the whole culture and product, it's specific teams that do it. And trying to push back on teams that are "producing" is not a simple task.
reply
Sadly AI also amplifies both the good and the bad managers and tech leads. If you have managers who fail to set a clear direction or tech leads who fail to define a good framework for developers to operate within then AI just means the poorly directed developer effort goes further in the wrong direction faster. Meanwhile teams with well-specified goals and better structure and processes can exploit the benefits of AI tools much more effectively.
reply
> We are pushing tons of code and now our CPU usage has grown exponentially over the past year because the bad engineers just ship whatever Claude gives them and do not think about the consequences.

Sometimes this can be a death by a thousand cuts. Any individual change may not impact performance to a noticeable degree but when they're pumping out a 10x increase in commits it can be a slow decline.

Just look at how they're merging ~300 commits a week into bun.

https://github.com/oven-sh/bun/graphs/commit-activity

reply
That really sounds like the kind of people who will complain in university about being forced to study calculus.

Natural consequence: Then they never grasped the concept of computational complexity.

O(n) Vs O(n²)? They have n, what's the difference? Python is fast enough. The only thing that matters is shipping features fast! Features! Our competitor will have this next week, we need to write code fast, everything else is a matter of adding more compute, which we will pay with revenue!

reply
As a side note; There is plenty of useful math that many people need (or would benefit from) in the real world. Much math curricula is arguably poorly aligned to needs, as it emerged from various historical flukes of design-by-committee. Popular math curricula is not perfect by any means. I recommend looking at high school or college textbooks and considering whether it is well prioritized.
reply
In principle I agree with you.

The missing part of mathematics education, IMO, would be to focus more into developing the intuition of what something means, instead of the current focus on getting some (numeric) results.

reply
computational complexity was never part of calculus. Calculus is the math of continuous change.
reply
You don’t learn Big-O in Calculus because it requires CS algorithmic concepts.

But the core question of “how does this behave as N -> \infty?” is asymptotic behavior (ie: limits) which were developed for calculus and are very much part of the foundational calculus canon.

reply
I should add that text to the rant at my last paragraph!
reply
A hackathon for performance improvement screams to me "you'll be re-interviewing for your own job." I don't foresee this going well for the company. Best case scenario: you wildfire churn all the chaff engineers which limits the company's throughput until they're rehired... with no expectation the new hires will be any better. You also lose A LOT of domain experience and are self-inflicting brain drain.

I'd try and push for some "lunch and learn" meeting where the engineers get lunch catered and in exchange sit in on a meeting where you explain your point of view. Without monetary incentive it'll be hard to change the culture, but not impossible (and food goes a long way in greasing the wheels).

reply
> A hackathon for performance improvement screams to me "you'll be re-interviewing for your own job."

Not necessarily, great engineers do also ship temporary code they didn't have the time to trim.

Our process is of 1) make it work, 2) make it right and 3) make it fast; not necessarily that engineer had time for the 3rd step.

reply
they meant application performance
reply
If your company has sketchy process and minimal checks on the software quality, of course it naturally follows that they hire bad engineers as well. fortunately, that also opens up an opportunity to be a emergency leader with an eye towards promotion.
reply
> If your company has sketchy process and minimal checks on the software quality, of course it naturally follows that they hire bad engineers as well. fortunately, that also opens up an opportunity to be a emergency leader with an eye towards promotion.

AI disease is encouraging "sketchy process and minimal checks on the software quality." QA has been eliminated from my team, and the QA engineers that are left have been declared to be developers now.

Gotta move fast, and I guess making sure the stuff we ship works was "slowing us down."

reply
More like developers are now testers and nobody is doing QA.
reply
> a emergency leader with an eye towards promotion

i've gone this route a few times in my career, it's very stressful and involves angry/panicked people and many all nighters. Also, the glory fades fast. would not recommend.

reply
Also you don't get the money. Why pay for a job already done
reply
The bigger issue (I find) is that the pipeline for finding those engineers is completely fucked.

Hiring pipelines that tested the wrong thing have existed for years but the problem is magnified 10x when you test for something that weakly correlates with ability at best which an AI can do better than a human.

This is leading to stuff like incompetent junior-level engineers being hired as principals.

reply
My company's hiring process is basically: Can you code a product quickly with AI? Can you understand the generated code?

That's basically it. It is surprising hard to find people that can do both, but engineers are becoming much, much better at the first gate while flaming out on the second.

reply
>be a emergency leader with an eye towards promotion..

You assume those people haven't already left, been kicked out, or were hired to begin with. We're not in a rational job market right now.

reply
Do you have SLO/SLAs defined? Are you monitoring performance? CPU usage? Cost increases?

You should have plenty of data to review regularly and push back on any teams that are causing problems. That's a process problem and you need a process for it.

Trace the increase back to specific deployments, call out those teams, and make them fix their shit.

reply
Yes to all, but it's just me responsible for monitoring performance/cost. I either push back or fix it myself, but I am behind. So this is an attempt to give/push more ownership to the teams.
reply
The handling of backpressure is such a good example to distinguish bad from good engineering. A good implementation even presents all the typical "clean" code indicators: DRY, KISS etc.
reply
Mind sharing where you're at?
reply
Sounds like bad management to me
reply
My company got rid of line management. I now report to a director with over 20 reports and barely any time for career development. Our performance reviews are AI generated and we’re losing engineers. The industry has gone completely insane.
reply
This has less to do with bad engineers than bad policies and procedures. You don't performance test before shipping to prod? you get what you get.
reply
Startups have limited resources, we prioritize the best we can. As a staff engineer it's my responsibility to bring this to my company, but it's not an overnight process.
reply