Yes, agreed. I was speaking in general, of course. This particular topic is of interest to me, so thinking of the edge cases and confounds vs Jev.
replyIn your example, I would expect an LLM to do fine and if you have access to the raw logits you can measure whether or not it was confused and assign a confidence to the answer it gave.
I do think that Jev handles more than this though and, in my early testing, does things that are not easily accomplished with guided decoding techniques.
The way jev actually internally work could be interesting though. I believe most llm are only tuned to return the first or second logits(or a few more) correctly as that is what the sampler would choose anyway. Do they alter existing model for better behavior across all options? Or they distilled one to have the proper behavior? We can only guess without the actual implementation.
replyYes! I really hope they release some papers on their techniques. I am very curious.
replyI ran it through MMLU a few days ago and it scored ~90% so seems to have a lot of general world knowledge trained in. Makes me think your speculation is right. I have some credits left, might try and think of an experiment. I saw a gist where someone was asking it which model it was and it was picking qwen a lot, but who knows...
Anyway, thank you for the interesting discussion!
Although the underlying model is unknown. If it expose input token count, the tokenizer may be probable though. Most tokenizer segemnts wildly different in CJK inputs. It can probably be used to fingerprint the tokenizer based on token count if it is using existing tokenizer.
reply