upvote

points

by satvikpendem12 hours ago |
comments
upvoteby layer811 hours ago|
next
[-]
You still can’t deterministically guarantee anything about the output based on the input, other than repeatability for the exact same input.
reply
upvoteby exe3411 hours ago|
parent|
[-]
What does deterministic mean to you?
reply
upvoteby layer810 hours ago|
parent|
next
[-]
In this context, it means being able to deterministically predict properties of the output based on properties of the input. That is, you don’t treat each distinct input as a unicorn, but instead consider properties of the input, and you want to know useful properties of the output. With LLMs, you can only do that statistically at best, but not deterministically, in the sense of being able to know that whenever the input has property A then the output will always have property B.
reply
upvoteby peyton9 hours ago|
parent|
[-]
I mean can’t you have a grammar on both ends and just set out-of-language tokens to zero. I thought one of the APIs had a way to staple a JSON schema to the output, for ex.

We’re making pretty strong statements here. It’s not like it’s impossible to make sure DROP TABLE doesn’t get output.

reply
upvoteby layer86 hours ago|
parent|
next
[-]
You still can’t predict whether the in-language responses will be correct or not.

As an analogy: If, for a compiler, you verify that its output is valid machine code, that doesn’t tell you whether the output machine code is faithful to the input source code. For example, you might want to have the assurance that if the input specifies a terminating program, then the output machine code represents a terminating program as well. For a compiler, you can guarantee that such properties are true by construction.

More generally, you can write your programs such that you can prove from their code that they satisfy properties you are interested in for all inputs.

With LLMs, however, you have no practical way to reason about relations between the properties of inputs and outputs.

reply
upvoteby satvikpendem8 hours ago|
parent|
prev|
[-]
And also have a blacklist of keywords detecting program that the LLM output is run through afterwards, that's probably the easiest filter.
reply
upvoteby tsimionescu9 hours ago|
parent|
prev|
[-]
I think they mean having some useful predicates P, Q such that for any input i and for any output o that the LLM can generate from that input, P(i) => Q(o).
reply
upvoteby exe344 hours ago|
parent|
[-]
If you could do that, why would you need an LLM? You'd already know the answer...
reply
upvoteby tsimionescu2 hours ago|
parent|
[-]
Having that property is still a looooong way away from being able to get a meaningful answer. Consider P being something like "asks for SQL output" and Q being "is syntactically valid SQL output". This would represent a useful guarantee, but it would not in any way mean that you could do away with the LLM.
reply
upvoteby idiotsecant12 hours ago|
prev|
[-]
You don't think this is pedantry bordering on uselessness?
reply
upvoteby WithinReason11 hours ago|
parent|
next
[-]
No, determinism and predictability are different concepts. You can have a deterministic random number generator for example.
reply
upvoteby satvikpendem12 hours ago|
parent|
prev|
next
[-]
It's correcting a misconception that many people have regarding LLMs that they are inherently and fundamentally non-deterministic, as if they were a true random number generator, but they are closer to a pseudo random number generator in that they are deterministic with the right settings.
reply
upvoteby 11 hours ago|
parent|
[-]
deleted
reply
upvoteby albedoa8 hours ago|
parent|
prev|
[-]
The comment that is being responded to describes a behavior that has nothing to do with determinism and follows it up with "Given this, you can't treat it as deterministic" lol.

Someone tried to redefine a well-established term in the middle of an internet forum thread about that term. The word that has been pushed to uselessness here is "pedantry".

reply