upvote
Most human personal assistants are equally vulnerable to this prompt injection.
reply
Yeah, anyone who has had to manage developers in a big company will be able to tell you that it is a constant struggle between giving them freedom to work independently and get things done vs. limiting their access to prevent potential bad stuff from happening. In practice, the more bad stuff happens (and the more consultants you allow in), the more hurdles are introduced. Eventually even trivial tasks become a corporate nightmare of approvals. You can't have the conveniences of both work philosophies at the same time. Even the smartest AI agents won't change that fact. It simply limits their work speed to the speed of the approval process.
reply
Humans are vulnerable to scams, cons, phishing and social engineering. This is a vulnerability whose root cause we not know how to fix. We can only alleviate its effects by re-engineering the environment around those vulnerable humans.

LLMs are just as gullible, we just tend to call their vulnerabilities "prompt injection", even though it's basically the same class of attack under the hood. Unlike humans, though, LLMs are a lot less diverse (you and I are using the same Opus 5.5, but we have different brains). This makes attacks on LLMs much more predictable and easier to pull off, just as a phishing campaign that targets one specific person is easier than an attack that scales to millions of diverse victims.

In the age of agents, phishing will continue to be the biggest attack vector of all, it's just the victims that will change.

reply
Well, there's definitively a pareto frontier of freedom and avoiding bad stuff. And many companies ain't on it.

(In practice, I found Google to be better at having (internal) security controls that are reliably but also don't get in the way than eg banks I worked at.)

reply
the difference being, that if you build those kind of approval processes up with AI agents, the system will function a bit faster. you could have a web reading agent. a web writing agent. a coding agent. a credentials agent. etc. and have everyone make these formal requests.

it will catch quite some prompt injections, as the prompt injection has to promptinject a secondary system in order to be successful.

reply
> the more bad stuff happens (and the more consultants you allow in)

In your parenthetical, you just said the same thing using different words

reply
I have, never in my career, had any of my coworkers deanonymize my browsing or leak my email address everywhere on the internet. Your copy-pasted, thought-terminating argument doesn't work here.
reply
Have you had any of your coworkers update Adobe Flash Player, enter their Okta credentials to "claim their $150 Christmas Walmart voucher" in the company-mandated phishing test, or install "Zoom" to join a job interview or sales call from their work laptop?

The attack is different, the result is the same.

reply
I can't believe I have to say this, but they are not the same. Your coworker isn't deanonymizing other people's browsing by sending their email address to every site they visit without their consent.

Your example is a completely different ragebait that just serves to muddy the waters by making the easier claim that people too, are gullible. Well, even that's misleading because not all people stoop so low for a fake $150. You can hold such people accountable and contain the damage, but LLMs will make the same mistake over and over again and people will still line up on HN to say how that has always been the norm with people. The grift in the software industry is reaching new lows every day.

reply
This would be more like your HR department giving your info to the government no ?
reply
It wasn't even that long ago since a human error led to billions of crypto being stolen
reply
And what am I to make of this totally unrelated story?
reply
They actually aren't. There is ambiguity on what the proper email to use is.
reply
Most human personal assistants don't have access to vast swathes of private data that they collected without permission, unless your house is the site of multiple watergates
reply
Wait, are you saying Claude isn't conscious on the internet, sir?
reply
You're dehumanizing Claude, sir. That's against Anthropic's updated ToS!
reply
Please look up the research area "alignment."
reply