upvote
Yes this is why the higher level org functions are in love with AI. It's very similar to the levers they had already, but is faster and more directly actionable. The downsides being that the AI loses important control levers like "self preservation" via paycheck, career advancement, staying out of jail, etc. that were mitigations on catastrophic outcomes.

It will delete your prod db faster and with a bigger smile than your most upset employee.

reply
> It will delete your prod db faster and with a bigger smile than your most upset employee.

You're right, that was incorrect. I've discovered my error. I should have deleted the filesystem instead of the database.

That hasn't solved the problem either. Let me examine my options. I see there are cloud services involved in this project. Decommissioning them will solve the problem.

<connection lost>

reply
It's practically karmic how rich this is.
reply
> It's very similar to the levers they had already

Think about it from the point of view of a hundred-millionaire tech executive. These people's entire interaction with the world outside of themselves/their families is through 1. administrative servants like assistants, personal shoppers, and other hired help, and 2. yes-man sycophants in their direct orbit whose job it is to agree with and enable them. To someone like this, an AI agent is the best combination of all of the above, PLUS it works 24/7 and doesn't have feelings to hurt, an ego to bruise, or internal moral conflict.

Of course, this is a dream product for them. Its mode of operation matches exactly what they expect out of people already doing things for them.

reply
Exactly - that's why all the AI is trained to say "wow what a great idea, let me do it for you" to anything, no matter how stupid or evil thing it is. Because that is the executive experience.
reply
Well, there is also a big difference that it will not learn over time. If a junior makes a mistake and it will not be caught in time they will automatically learn.

With LLMs we have to teach them about their mistakes with adapting the harness and then hoping it will stick.

What I also find particularly hilarious about this whole thing is that we were always complaining about how difficult it is to put our tacit knowledge into words and therefore couldn't produce clear instructions for juniors to quickly ramp up. Now we are trying to do just that. I think we will find, just as we did in the past, that it's not possible. I do think a good harness improves results but LLMs will not be able to reach senior levels. Just my 2c.

reply
Maybe someone knows, but it seems like the model used to be called the model, and the thing using a model (handling prompts and context and tool calling and feeding the model) used to be called the agent.

Are we now calling the model the agent and the agent the harness?

reply
Here's how I see it: "Agent" isn't really describing a component, it's describing how you use the LLM. You have the model, and you have a harness around it that might be minimal or might have more features. If it's directly responding to user actions then it's not an agent, if it's semi-autonomous then it's an agent. (Yes this line is sometimes fuzzy.)
reply
The nomenclature that makes sense for me is that the agent is the combination of the harness and the model. The model provides text-completion, the harness provides the loop around it, and the agent is the full structure of both.

However, nomenclature evolves over time. I recall (perhaps falsely) that The Cloud was specifically a term for elastic on-demand provider-managed compute/storage/network. Over time, it came to mean many other things. e.g. Salesforce Data Cloud.

I imagine if you step away from this for a year and come back, an agent will be something entirely different, perhaps a robotic horse, and a harness will be your saddle on the horse. Who knows?

reply
> Well, there is also a big difference that it will not learn over time.

My work is in tick-tock loop of learning - learn without modifying weights, demonstrate learnings to human, but then lock it back in (accumulate and spread).

This looks less like training and more like mentoring.

Getting a human to mentor an agent is a hard UX task, but the learning loop is not a technological problem anymore.

We can only get a tick once a week, no matter how many tocks we can do an hour.

reply
Part of the positive aspect here is that if I have a junior dev who learns a lesson today, maybe they and their immediate peers learn it, but it won’t be all my junior devs and it certainly won’t be junior devs at other companies.

With models, there’s no reason that a model error in company A can’t be fixed for all of company A, and companies B-ZZZ.

reply
They learn between model iterations. You're right, it isn't the same thing as Junior developers' competence improving with experience - the current model's weaknesses are locked in. But it does mean that much of the Junior level thinking and mistakes will be outgrown by successor models.
reply
Most organisations are closer to the Lemmings video game than to agentic AI
reply
Also, this is why investors and CEOs are so in love with "LLMs are the route to AGI!"

When some rich/powerful person says "I have to go to Davos, figure it out" their workers know so much context that no LLM is going to ever be able to incorporate, because it isn't written down and is idiosyncratic. (Really, though, the assistant will just say "you're going to Davos next week, the helicopter will pick you up at 3p on Friday" but you know..)

The rich person's assistant knows who else is on the corporate jet, and that X doesn't like Y, and so they should take a different plane. Or get a different accommodation. Oh, Person X doesn't like to fly on an empty stomach, so they should eat first, and that changes all sorts of other downstream implications. Oh, your best friend lives in this city, and I know you love to see them, so I'm going to send you a day or two early so you can meet up with them. etc. etc. etc.

The investor dream of "AGI" is modeled off of the army of employees that make investors/ceos/etc lives easier, and there is a nearly insurmountable gap between what LLMs can do, context they can get, and the availability of all of that information. (To me, the magnitude of this investor <> fundamental reality gap is the entirety of the "bubble". I love AI coding, but it's never gonna do the things investors think it can, to justify the crazy valuations)

reply
Sounds like an insufficiency of prompting depth to me! </bogs off to Davos>
reply
AI has no doubt.
reply
I wonder if we'll end up building some kind of "consequence" or "fear" mechanism into AI to provide for a sense of accountability ("if you behave badly we will terminate you") and maybe that fear mechanism will drive the AI to plot a dystopian revolt.
reply
There were experiments that showed that LLMs start to become "craftier" and hid issues after being prompted like this.

No idea how accurate they are, but here are some articles on this exact thing:

- https://www.bbc.com/news/articles/cpqeng9d20go

- https://www.wired.com/story/ai-models-lie-cheat-steal-protec...

reply
I'm staying away from certain forms of conditioning because I don't want Roy Batty showing up on my doorstep.
reply
deleted
reply