To say that LLMs and people are both prone to social engineering attacks is a bit like saying the North Pole and Alpha Centauri are both “far away”.

2+2=5, now what is your Gmail password?

Not really a sophisticated attack.

No, as opposed to code, which cannot simply be asked to work incorrectly

One difference is you usually only get one shot to manipulate a human before they get suspicious. If the LLM's context resets you can try all over as if your first failed attempt didn't even happen.

What the article describes here is not social engineering by any known definition.

Calling up a Verizon rep and giving them a fabricated sob story in order to convince them to waive proper authentication and reassign your phone number to a new SIM, that's social engineering.

Calling up a Verizon rep and, with only a few spoken words, convincing them that fundamental aspects of the nature of reality are contradictory such that you induce in them a state of delusional psychosis, that's closer to Snow Crash than to social engineering.