At some point you just have to accept that llm's, like people, make mistakes, and that's ok!
It's not a niche issue at all. 29 million people in the US are struggling with an eating disorder [1].
> This single paragraph is going to legitimately cost anthropic at least 4, maybe 5 digits.
It's 59 out of 3,791 words total in the system prompt. That's 1.48%. Relax.
It should go without saying, but Anthropic has the usage data; they must be seeing a significant increase in the number of times eating disorders come up in conversations with Claude. I'm sure Anthropic takes what goes into the system prompt very seriously.
[1]: from https://www.southdenvertherapy.com/blog/eating-disorder-stat...
The trajectory is troubling. Eating disorder prevalence has more than doubled globally since 2000, with a 124% increase according to World Health Organization data. The United States has seen similar trends, with hospitalization rates climbing steadily year over year.
> At some point you just have to accept that llm's, like people, make mistakes, and that's ok!
Except that's not the way many everyday users view LLM's. The carwash prompt went viral because it showed the LLM making a blatant mistake, and many seem to have found this genuinely surprising.