upvote
There are other professions with procedures and checklists. Just off the top of my head there's safe surgical checklists to minimize errors during surgeries [1] and checklists for A+P procedures on planes, checklists for operating/working at nuclear plants, and for factories/manufacturing plants making precision parts or operating within a good safety regulatory regime (like semiconductor fabs or auto manufacturing), to say nothing of checklists in the military, which designs equipment to have easily disableable safety mechanisms if it has any at all because the operator might need to operate it outside normal, safe limits [2].

[1]: https://www.who.int/teams/integrated-health-services/patient... [2]: https://en.wikipedia.org/wiki/Battleshort

reply
But all these procedures stem from pilot checklists.
reply
Take some time to read up on Taylorism and scientific management. Frederick Taylor pioneered the use of standardized procedures in manufacturing around the same time as Orville & Wright, long before standardization became common in aerospace.
reply
No, they didn't. Not standardized procedures. Surgical checklists specifically certainly did, though, no disagreement there. But everything else? Florence Nightengale popularlized a standardized "hospital corners" as a way to help ensure sanitation in hospitals in the 1800s. A+P grew up alongside pilot checklists. It's just as fair to claim pilots borrowed from A+P as it is to claim A+P borrowed from pilots. Exacting manufacturing and industrial processes and procedures predate aviation by a few hundred years. Standardized military procedures predate aviation, too. They're actually what aviation checklists derived from.

But even if all of those other fields did derive the standardization of procedures and checklists from aviation checklists, so what? My point isn't that others invented checklists, too. It's that pilots aren't the only organization that attempts to standardize a process to the point where operators are interchangeable without changing the output.

reply
Well yeah, AI agents will default to the rules and patterns, its what they're good at.
reply
Except that ‘one AI’ will have it’s own blindspots which can be exploited universally and all at once, until the magic fix goes into it.

It’s literally a monoculture.

For example, if an iteration of the singular AI ends up being crashable with a magic visual pattern or QR code (highly likely at some point), that is a major national security risk if that AI is piloting aircraft, as the entire nations aircraft could be taken down by anyone projecting said image in it’s visual plane until the fix gets pushed.

And these systems are far too complicated to exhaustively test all possible inputs before release.

There is a reason why nature rarely has such monocultures - that live for very long, anyway.

reply
With RL it will learn from all misstakes once while with humans after a generation, information might just be gone.

Its a possible as an attack vector for sure.

But in comparision to nature, compute can speedrun all of this with RL.

reply
If every AI were different, they can still learn from each other, without failing together.

This has different failure modes than monocultures, which would also be bad for us: a collection like that behaves as a species, and can evolve faster than one single thing, and evolution is a blind watchmaker that cares only for success rather than details.

reply
RL data is really really really expensive to collect and train on, and only works for verifiable tasks, which is not most tasks.

Even a lot of what software professionals do isn't verifiable.

reply
We do have training material for humans, we already have some kind of training material for AI.

For a human you need to train everyone every year and if you have more and more material to teach, harder it will be for the humans.

Not so for AI.

Then we will also see more simulations, an AI will be able to play through scenarios and we will have models being able to invent new scenarios.

While this is happening, hardware will be faster, more and more improvements on every front. Humans have not progressed really

reply
With the greatest of respect, AI requires far far more (labeled) examples to learn a task to the same level as a human.

More generally, if you assume that magic can replaces the laws of statistical learning, then yes, AI can do everything and we're all doomed.

I reject this premise on the basis of knowing how RL and LLM training works.

reply
I also know how RL and LLM training works :)

How many people learn one particular job? If its a job which educates 10.000 people per year, when is the financial motivation big enough to create enough training data to make them obsolete and transfer the learning into an AI model?

Also we don't know yet were the journey goes, 'dreaming' models finetuning over night might help.

Or its already just enough to have the biggest RL Loop in history of humanity: everyone of us using the chatbots and giving them signals like thumbs up and thumbs down and stuff like "DUDE WTF I TOLD YOU..." you know :)

Then we also know that the frontierlabs are paying experts and people with PhDs to generate challanges and answers.

Educating just a 1000 people every year over and over is very very expensive even if LLMs need more training data. In worst case frontierlabs will just keep investing in generating traingindata manually with their earnings and it takes. longer.

Or we find a breakthrough like how LeCun is complaining all the time that LLMs are not learning properly. We are still early and progress is still happening fast.

reply
Lol, clearly you do not know how RL and LLMs actually work.

Or maybe cut back on the acid.

reply
How can it learn from a mistake if it doesn’t know (and can’t know) it is a mistake until it is too late? Let alone ‘all mistakes’?

Thinking it can detect the unknowable at the time is a big ‘ML magic’ thinking error.

It’s the same thing that fed the grey goo panic around the 2000’s.

Turns out, entropy and the law of thermodynamics applies to nano machines too.

reply
I mean we as humans solved this isuse right? We are not magic in any sense.
reply
We definitely have not, or planes would not crash.
reply
Still we solved it well enough that we fly, despite that we have to teach every pilot anew.

With AI you have to teach one model once. It makes a mistake? Teach it and it will not make it again.

This is a lot cheaper than training every pilot on the planet and its a lot more persistent then training every new pilot

reply
> With AI you have to teach one model once. It makes a mistake? Teach it and it will not make it again

Has this been your experience? Models still invent answers for me. Directives go unrespected, etc.

reply
It got siginificantly better for me, yes. Since Opus 5.5 i stoped doing one thing after the other and give it a whole list of 5-30 items (just yesterday I had a list of 40 things to change in one prompt).

I expect it to get significantly better in the next few years but I also expect either to see continues improvement or new optimized models for specific tasks.

reply
The difference between that and it not actually crashing into mountains at a rate higher than humans is night and day.
reply
Crashing it and universal exploitability isn’t my concern, it’s targeted exploits.

If you have access to the auditing AI in advance then you would pre-check your bad behaviour to see if it flags. And iterate until it doesn’t.

Bad actors already do this with humans, they know where auditors are likely to look and act accordingly. Even with my limited financial training I learnt where some big murky bits are and how to spot them abuse.

With a human there’s always a risk that they’ll see it, spot the swap in the shell game etc. An AI isn’t exactly deterministic but it’s a lot closer so the audit risk goes down.

reply
Yeah, a lot of the pie in the sky dreaming here in this thread - imo - is just because the disasters haven’t happened yet because no one has been crazy enough to try it at scale. Yet.
reply
Can't some of this be mitigated by having voting systems? Have three AI systems from three different vendors and they vote on the solution? Then any one failure will just fail the one system. The chance of three identical vulnerabilities in three different architectures is pretty slim. You can also have some of the Airbus "watchdog" hard coded oversight to prevent flying outside a defined envelope for most normal flight.
reply
Certainly less of a problem. That is how some of the flight computers work in some aircraft/spacecraft right now, albeit with traditional programming.
reply
> Of course ONE AI agent and teaching ONE AI

One of the most misinformed things I’ve read on Hacker News recently. That’s not how any of this works.

Has the average quality of discussion plummeted since the advent of LLMs and every tech bro starting to comment wildly out of their depth?

reply
I'm reading about AI, LLMs and ML for over 4-8 years now and finetune models etc.

How about you are less snarky and try to be clear what your problem with my statement is?

Should I say one model/one model family? etc.?

reply