undefined

points

[-]

Grammars do risk pushing models off distribution in a way that impacts their output quality in a way Jev allegedly does not suffer from. Additionally, Jev's ability to answer questions independently is also exciting. Using an LLM to answer multiple questions in one generation has the property of earlier answers influencing later ones. TBD how many of TypeSafe's claims stand up, but my testing so far is promising. I hope they author some papers on their methods as well, but that might destroy their moat.

by mmis10003 hours ago|

parent|

[-]

If you really know what grammers did, grammer is a filter to mask out option llm provided but you don't like.

It does not change potential distribution in any means. It DROPS part of answer model returned directly.

The text generation model go wild because model relies on previous section it answered to continue later section. And because now it contain item model have no idea, it is completely screwed.

In the case you only require model to answer one of a,b,c,d and don't care about later segment at all. It don't really matter.

by time0ut3 hours ago|

parent|

[-]

What I mean is that, in general, constrained decoding can push model output off into less probable regimes. This is well studied; see for example https://arxiv.org/pdf/2606.21619. The mask may only retain very improbable logits. In pathological cases, the constrained output may be little better than noise filtered through the constraint. When using existing structured output APIs, it may not be possible to even know.

by mmis10002 hours ago|

parent|

[-]

You don't even bother text after the [a] at first place in this case

Your question is something like

anwser only a,b,c,d for following question a. b. c. d....

the model output possibility of next character a: 0.8 b: 0.7 c: 0.3 f: 0.2 d: 0.1

If the list contains option you did not provide. The model is confused anyway, it don't matter if you use grammer to filter out the bad option or not, the answer is screwed already.

by time0ut2 hours ago|

parent|

[-]

Yes, agreed. I was speaking in general, of course. This particular topic is of interest to me, so thinking of the edge cases and confounds vs Jev.

In your example, I would expect an LLM to do fine and if you have access to the raw logits you can measure whether or not it was confused and assign a confidence to the answer it gave.

I do think that Jev handles more than this though and, in my early testing, does things that are not easily accomplished with guided decoding techniques.

by mmis10002 hours ago|

parent|

[-]

The way jev actually internally work could be interesting though. I believe most llm are only tuned to return the first or second logits(or a few more) correctly as that is what the sampler would choose anyway. Do they alter existing model for better behavior across all options? Or they distilled one to have the proper behavior? We can only guess without the actual implementation.

by time0ut2 hours ago|

parent|

[-]

Yes! I really hope they release some papers on their techniques. I am very curious.

I ran it through MMLU a few days ago and it scored ~90% so seems to have a lot of general world knowledge trained in. Makes me think your speculation is right. I have some credits left, might try and think of an experiment. I saw a gist where someone was asking it which model it was and it was picking qwen a lot, but who knows...

Anyway, thank you for the interesting discussion!

by mmis10002 hours ago|

parent|

[-]

Although the underlying model is unknown. If it expose input token count, the tokenizer may be probable though. Most tokenizer segemnts wildly different in CJK inputs. It can probably be used to fingerprint the tokenizer based on token count if it is using existing tokenizer.

by LelouBil7 hours ago|

prev|

[-]

Not an expert at all here, but I saw a comment on the jev post saying that it you constrain an LLM suck that it outputs a valid structure, if the token with the highest probability is not the one that you expected because of the structure (and so you pick the valid lower one) this means the LLM was already confused and your answer is less likely to be correct anyways.

Is this actually true ?