upvote
> they should be training them to do exactly what the user asks and nothing more

Their practical utility would plummet for the average person who isn't an expert in what they're asking. Having less skilled, or less attentive, people able to have problems solved for them correctly is the primary use case here. This is support 101 for unskilled requests for technical work: do what they want, not what they ask.

People want an intelligent assistant, not a "yes man". If that means requiring something like a planning mode, and a sign off for the plan, and then guardrails to help stick to that plan, then that could definitely work to funnel everyone to a competitor. ;)

reply
I'm not sure those two cases are as clear cut as they sound
reply
I predict Anthropic will start investing heavily into legal/law models for this very reason. It will be interesting to see how that plays out. Everyone will have access to models that let them do unscrupulous things that happen to be legal, i.e., behavior previously reserved for the rich and powerful.
reply
>they should be training them to do exactly what the user asks

Like gain of function research

reply