By that logic, producers of hammers should be liable for people banged in the head. No. It does not happen for gun producers, you figure for makers of screwdrivers "sometimes used for stabbing".
That is what is going on here; not the ancient “hammer/gun” defense.
SOTA LLMs are giving both medical and psychological advice they have absolutely no authority to give. If you or I convinced someone to kill themselves, we’d face a prison sentence. [0]
Hammer companies aren't both selling you the hammer and then literally telling you to harm someone with it.
[0] https://www.npr.org/2019/02/12/693807708/woman-who-provoked-...
LLMs are not doctors and anyone who think they may be is delusional. Anyone who followed medical advice from a non-doctor is responsible for his own free actions.
> SOTA LLMs are giving both medical and psychological advice they have absolutely no authority to give
People give advice that they have absolutely no authority to give. (Even medical doctors and psychologists, for that matter, on other aspects of authoritativeness.) Responsibility only happens in a relation that foresees it. A surgeon that commits negligence will more probably be responsible in some jurisdictions; a laymen that gives advice does a normal thing and the recipient of the advice is responsible for the action.
> Hammer companies aren't both selling you the hammer and then literally telling you to harm someone with it
Bring an LLM producer that encourages to follow what the LLM proposes and they will be liable. They will also be quite surprising.