upvote

points

by 9rx9 hours ago |
comments
upvoteby xigoi7 hours ago|
next
[-]
While they’re technically deterministic, they’re still chaotic, in the sense that changing irrelevant details in the input (such as writing “color” versus “colour”) can make the output completely different.
reply
upvoteby vlovich1239 hours ago|
prev|
next
[-]
Even with temperature and a batch size of 1 and fixed seed LLMs should be deterministic. Of course batch size of 1 is not economical.
reply
upvoteby troupo8 hours ago|
prev|
[-]
with temperature=0 and no context. That is, a clean run with t=0, pk=0 etc. etc. will produce the same output for the same question. However if you ask the same question in the same session, output will be different.

To say the least, this is garbage compared to compilers

reply
upvoteby 9rx8 hours ago|
parent|
[-]
> However if you ask the same question in the same session, output will be different.

When isn't that true?

    int main() {
        printf("Continue?\n");
    }
and

    int main() {
        printf("Continue?\n");
        printf("Continue?\n");
    }
do not see the compiler produce equivalent outputs and I am not sure how they ever could. They are not equivalent programs. Adding additional instructions to a program is expected to see a change in what the compiler does with the program.
reply
upvoteby troupo5 hours ago|
parent|
[-]
If you ask the compiler to compile the same input, it will produce the same output.

With LLMs the output depends on the phases of the moon.

reply
upvoteby 9rx5 hours ago|
parent|
[-]
> If you ask the compiler to compile the same input, it will produce the same output.

As with LLMs, unless you ask for the output to be nondeterministic. But any compiler can be made nondeterministic if you ask for it. That's not something unique to LLMs.

> With LLMs the output depends on the phases of the moon.

If you are relying on a third-party service to run the LLM, quite possibly. Without control over the hardware, configuration, etc. then there is all kinds of fuckery that they can introduce. A third-party can make any compiler nondeterministic.

But that's not a limitation of LLMs. By design, they are deterministic.

reply
upvoteby troupo47 minutes ago|
parent|
[-]
> But any compiler can be made nondeterministic if you ask for it. That's not something unique to LLMs.

Not unique as in: no one makes their compilers deterministic, and you have to work to make a non-deterministic one. LLMs are non-deterministic by default, and you have to contort them to the point of uselessness to make them deterministic

> If you are relying on a third-party service to run the LLM, quite possibly. Without control over the hardware, configuration, etc.

Again. Even if you control everything, the only time they produce deterministic output is when they are completely neutered:

- workaround for GPUs with num_thread 1

- temperature set to 0

- top_k to 0

- top_p to 0

- context window to 0 (or always do a single run from a new session)

reply
upvoteby 9rx8 minutes ago|
parent|
[-]
> no one makes their compilers deterministic

Go (gc) was specifically designed to produce reproducible builds, so clearly that's not true, but you are right that it isn't the norm.

Some of the most widely recognized and used compilers, like gcc, clang, even rustc, are nondeterministic. If you work hard and control all the variables (e.g. -frandom-seed), you can make these compilers deterministic, but, hey, if you work hard you can make LLMs nondeterministic too.

reply