For what it’s worth I don’t think you can call this social engineering since there was no human on the other end, even though it appears similar.
The question is, if there were actual human support agents, would they have built additional safeguards to prevent social engineering in this manner?
Even if humans failed at the same rate, if you tried to exploit at scale you’d be throttled by the size of the support team. The failure would happen at human-scale time frames and throughput.
- instead of the ai context dying.
in the ai case, information only survives to the extent where the ai is empowered to store a note or notify a manager of an observation. Anything that does not result in sending a message/storage is wiped
The reason it worked there is that the designers of the system didn't anticipate that the AI will agree to accept any email (maybe they even put guardrails against it in the system prompt, we don't know). It's more like social engineering than bad-security-code, except that like the sibling comment said an actual human will probably not approve that.
These are contradictory cases. If you put guardrails into the system prompt, you've anticipated that the AI will take the action you're guardrailing against. And since AI prompt compliance is at best stochastic (and realistically just crap, over large sample sizes), every guardrail is an explicit recognition of a failure -- the guardrail will be ignored, and you can't pretend you didn't realize it was a problem, since you put it in.
The best comparison I can think of is that it's like validating dats on the frontend; it can make for a better user experience and he more efficient than hitting the backend when you know it will be an error, but it's not protection in any meaningful sense, and if you're not also enforcing invariants from behind the API, you're going to have a bad time. This is pretty similar to the type of issues you might run into with an implementation like that, where someone might make a request with data that you wouldn't expect from your frontend and perform operations you didn't mean to allow.
It might be bad to have it if the user can obtain the system prompt and make note of any advisories as potential weaknesses.
This looks like a terrible design rather than an AI problem to me, though.
An AI enabled terrible design. AI acted as a black box of stupidity, that obscured the stupidity of the design.
Humans do get fooled but it usually takes far more effort than that because a human service rep can learn and is worried about having a job tomorrow.
Do we actually know that a human was in the loop before and that the human judgement was replaced by an LLM? Or is that pure speculation?
I have certainly seen account reclamation flows that allowed providing a new email address (but usually with better safeguards).
https://www.meta.com/account-recovery-support/ai-support-ass...
Now, it’s possible that they instead moved it to human workers and simultaneously forgot everything they’d learned about security or training, but that seems unlikely.
I can think of several pre-2000s chat rooms that did EXACTLY this. It is how I lost several chat accounts as a teenager.
But had never been until it was wrapped in a chatbot. It’s just about unheard of for a major site in the modern era, isn’t it? I think the AI factor is essentially essential. All but.
Like, flagging VPN endpoints is bread and butter for this kind of thing and must already exist. But it's been bypassed
Until I remember seeing someone saying "MCP is dead, we just give agents command line access now". Then I start to think that looking at this in the context of ai is helpful.
If you'd do a retrospective and ignore how AI has shaped expectations and a company's culture to allow this to pass through into production, you'd be complicit/perpetuating what led to this debacle in the first place.
It's not the end of the world, and water isn't going anywhere, but saying AI has essentially nothing to do with it is just a bad take.
Also I've used Meta's old password recovery system. It's not possible to do this in that version. The chatbot is what makes this possible.
I mean this particular auth flow has been a well-known pattern, even before Ai came along.
I guess the only way they got away with this is due to the Ai in the loop. They kind of social (artificial) engineered the Ai, which prolly overlooked the well-known password recovery pattern.
dontake excuses for the greedy