I think some of it comes from that they do not publicly let you see the random seed. So each time you ask the answer is different (like a slot machine) and if they let users use the random seed it would let people much more accurately assess if an underlying model changed somehow (same seed and same input will always have the same output).
replyOf course the closed Anthropic would never share this, it would definitely take away the 'magic' feeling of the AI
With models there are a bunch of other dials that can be tuned even if the model itself remains exactly the same.
replyAre those dials set the same across all hardware configurations and clusters? Does model behavior average out the same across different hardware?
There are just too many different buttons that can be set to really trust a provider either not to directly commit fraud, or indirectly commit fraud with system complexity affecting the output.
To my understanding, with batched inference and other "optimizations" you wouldn't get the exact same token predictions even with temp=0.0.
reply