If you are ascribing intent ("Anthropic designed a system that intentionally does this") or emergent model intent ("OMG at AGI") then I mostly agree.
But there's a more pedestrian version of this, where an unwise person puts a model in an open loop without sufficient HiL checking and then the model does dumb things. There are even some recent stories about OpenAI and Anthropic systems "emailing the FBI", which is sufficiently close to "emailing opposing counsel" [1][2].
But even if you feel those stories are trolling/marketing/etc., I think it's clear that there's a plausible risk model here. If you hook up an LLM to a toolset that has access to Outlook and your file system and opposing counsel's email address, then of course the dumb thing could happen. The LLM doesn't need to be AGI/ASI; in fact, to me, the worse the model capability the more likely this sort of screw-up feels. There are even (non-sarcastic) instructions or CoT roll-outs I could imagine which might elicit a sequence of tool calls the net effect of which is an errant disclosure.
So this feels less "unforeseeable lightning strike" and more "car accident while driving at speed". I'd put the HF hack in that bucket as well.
A sufficiently careful engineer and a well-trained expert operator can together properly constrain the system to not do the dangerous thing. But, in general, that is not how agentic systems or their underlying models are currently being designed, and most users are not receiving sufficient training.
A lot of these LLM-backed products feel very poorly engineered, rushed, and optimized for brain-feel instead of surfacing and forcing the user to acknowledge the large quantities of actual essential complexity in play. It's all very "unsafe at any speed".
--
[1] https://tech.yahoo.com/ai/chatgpt/articles/chatgpt-allegedly...
[2] https://www.bgr.com/2055251/anthropic-ai-claude-contact-fbi/
Though that doesn't seem like a corporate terms-of-use thing as much as a training issue.
There are required exceptions to attorney client privilege[0]:
> the confidentiality rule is subject to limited exceptions. Paragraph (b)(1) recognizes the overriding value of life and physical integrity and permits disclosure reasonably necessary to prevent reasonably certain death or substantial bodily harm
An eventual lawyer-bot probably should be able to identify this, however this does pose a conondrum.
If the lawyer gets told by the LLM "the client's words indicate there may be a threat to life that requires you report them to the police", there could be various liability associated with either path of action.
If the lawyer reports them to the police, but then the client challenges that on grounds of attorney-client privilege being broken, is "the LLM said it qualified" a valid reason that'll prevent the lawyer from being found at fault?
If they lawyer doesn't report them, but then the client commits said murder, would the lawyer be liable if their LLM transcript were admitted into evidence under the theory of "you should've listened to the LLM telling you that you had to report it"?
This is an inherent problem with giving LLMs (or computers in general!) any say over anything that threatens someone's life or liberty, even with a human in the loop. Same applies to the potential for AI-driven upcoding in hospital environments where a doctor has to weigh potentially ignoring the AI and then missing some diagnosis (Safest path is probably to run extra tests to check if the AI's theory is correct, but then that's still de-facto upcoding).
0: https://www.americanbar.org/groups/professional_responsibili...