LLMs are not doctors and anyone who think they may be is delusional. Anyone who followed medical advice from a non-doctor is responsible for his own free actions.
> SOTA LLMs are giving both medical and psychological advice they have absolutely no authority to give
People give advice that they have absolutely no authority to give. (Even medical doctors and psychologists, for that matter, on other aspects of authoritativeness.) Responsibility only happens in a relation that foresees it. A surgeon that commits negligence will more probably be responsible in some jurisdictions; a laymen that gives advice does a normal thing and the recipient of the advice is responsible for the action.
> Hammer companies aren't both selling you the hammer and then literally telling you to harm someone with it
Bring an LLM producer that encourages to follow what the LLM proposes and they will be liable. They will also be quite surprising.