upvote

points

by vessenes1 days ago |
comments
upvoteby Dylan168071 days ago|
[-]
Even then it's deterministic in the way a hash function is deterministic. Change one letter and you can get a completely different output. What people actually want is something continuous.
reply
upvoteby vessenes1 days ago|
parent|
next
[-]
Agreed on the desire for continuous behavior. That said, in a modern LLM, is this hash analogy accurate? I would be surprised if a single letter changed most zero temp force ranked outputs.

E.g:

“Where is the Eiffel Tower Located? One word only.”

“Where is the Effel Tower located? One word only.”

“Where is the Eiffel Tower located? One wor only.”

I’d be very surprised if those got different answers from even a small local model at temp 0.

reply
upvoteby knome1 days ago|
parent|
next
[-]
For a single word response, perhaps.

But for anything else I wouldn't.

The entire chain will be affected from the different tokenization on down. Even if it lands in roughly the same semantic area, it doesn't mean it will land there with anything like the same syntactic selections. Anywhere there were multiple near-tokens could easily select a different route based on even minor fluctuations in the starting conditions. It's chaotic.

reply
upvoteby sobellian1 days ago|
parent|
prev|
next
[-]
I don't know about single letters, but single words?

"Score this resumé. Applicant: Jim ..."

"Score this resumé. Applicant: Greg..."

Is it obvious to anyone that these will have the same modal response?

reply
upvoteby vessenes23 hours ago|
parent|
[-]
I believe there's some data that they will have different responses if the names signify different cultural / race / gender affiliations. Here be dragons.
reply
upvoteby forlorn_mammoth1 days ago|
parent|
prev|
[-]
"Your are a helpful/less assistant"

Give it a try. 4 letter difference. Add a few 100 tokens describing the task, such that the change becomes a tiny fraction of the input.

Discontinuities everywhere.

reply
upvoteby vessenes23 hours ago|
parent|
[-]
But those are VERY different types of assistant. It is correct behavior that you would get different outputs in this case.
reply
upvoteby guhcampos1 days ago|
parent|
prev|
[-]
This is it. People mistake deterministic for precise/exact/correct. It's not.
reply