Oh, I have one of those use cases, matching people in genealogy trees. You can ask all sorts of questions: do the names match? Do they match within some edit distance? Do they match according to soundex/ metaphone rules (which are themselves a ginormous set of rules for letters and letter combinations which may or may not result in the same sounds, hand-coded as a huge if tree by a linguist not a programmer)? What about their relatives, do they match by the same rules? Should we incorporate domain knowledge about local naming customs? Etc etc.
I pointed a coding agent to this problem, and it aggressively started coming up with complex scoring rules and testing them against real datasets. Which led to sort-of acceptable results, but it still missed lots of cases which were obvious to a human, and had false positives which were obvious to a human. Which I could trade off, and slightly improve, with more back and forth with the coding agent.
Pointing a good LLM to all the information about two people, would of course give great results. Maybe even better than human judgment. But I can't do that for 100000^2 people, it would be too expensive in all sorts of ways. I need a fast, reliable scorer. I could maybe train an embedding, but that would be a huge job and where would I get the quality data?
Seems like a more accurate title would be "Jev: Trading general purpose generation for fast typed inference" or something like that.
This is interesting, but the speed comparison seems misleading? A generative model that can output code in a Turing-complete language can do anything a computer can do.
Jev can only generate structured output, right? This is probably super useful for classification/routing/scoring, but it's nothing like the code generating models we're all using today for code and automation.
Also "can't hallucinate" seems wrong? Sure, it can't emit an invalid type, but it can still emit a completely wrong valid value. You can enforce structured output from an LLM too, with an appropriate harness, etc.
Assuming there's no funny business, the Doom demo is cool.
The Doom demo seems very funny business. They're not feeding it video, they're feeding it a text description of what's going on in the game. It's not reading pixel data.
I think LLMs would play a lot better with that input too but Jev does seem to have a huge speed advantage; I don't know if the other models could do that in real-time.
llm hallucinations are high probability tokens that are incorrect vs the real world
I would be happy enough with: only produces what it can verify with sources.
If you eg try to remember a court case (ie produce the reference via LLM token generation only), it's easy enough to check with your data whether it really exists. Similar for following links and other references.
If your data or sources are wrong, obviously your report about them will be wrong. But I wouldn't call that a hallucination.
And humans also behave differently in different contexts. A conversation at the pub has more such hallucinations than a formal deposit in court. For the latter, a good lawyer will look at her shoes, when you ask him what colour her laces are.
Humans are known to hallucinate a lot. Ask 10 different witnesses at a crime scene what they saw and they'll all report different things.
A good, non-hallucinating LLM would only report things for which it has evidence. It would consult the facts every single time.
It's a pain in the butt for humans to fact-check everything but LLMs can quickly look up all kinds of stuff. That's what makes them useful.
A calculator either gets the right answer or doesn’t answer.
It wouldn’t have to be all knowing as long as it knew perfectly what it doesn’t know
System One says they use RLCD, Reinforcement Learning for Calibrated Decisions, which presumably has accurate probabilities as an explicit optimisation goal.
System One hasn't said how RLCD works, but they do say it is explicitly training models to output "calibrated" probabilities, which makes it distinct from RLVR. This is how they describe it:
> System One models are trained for calibrated decisions: their probabilities are optimized against outcomes to reflect uncertainty.
I believe their "accuracy" metric (sonnet 5 level) is where "right/wrong" is measured.
People out there are so resigned to the models being unreliable that they are really doing things like hallucinating deliberately, and then matching the hallucinations to embeddings -
https://softwaredoug.com/blog/2026/08/10/hypothetical-classi...
You could do that or you could just... use a model that will never produce unreliable outputs in the first place.
Even non-frontier models are absurdly good at this in a broad sense.
Which would make it hard to judge "a model that will never produce unreliable outputs in the first place" against something that is already really, really good and exceptional in domain-specific areas with the tiniest amount of elbow grease.
Speed and cost look good though (for now)!
So if the generated schema is for a tool call for calculator, then the numbers will be valid numbers for sure (and not random words).
To me, it looks similar to BNF schema already introduced and implemented few years ago: generally speaking - it limits the next token that is allowed to be generated, probs are drawn from a subset tokens.
(tbh, I'm not sure why it didn't pick up as a more standard interface to LLMs, as it made a lot of sense back then, and now.)
There's also a cheeky "one of the models hallucinated a link" in the wiki jump example that most likely could have been avoided by properly using grammars. You can setup constrained gen so that only valid options (say from a list) can be outputted. Their own inference lib likely does that. So comparing to one that doesn't is a bit cheeky.
That being said, after a brief look at the site I could see this working. Especially if this can be ran locally, the speed and cost can enable some workflows where you have this as an "overseer" layer over say a cli agent. After each step you run through a list of "questions" ("is the task completed?" -> yes -> "does the edit touch files it shouldn't" / "does the edit follow our code writing policies") etc.
edit: extra points if the "question" rubric is also generated by a higher abstraction model. Say "/goal Build out auth" -> generate_rubrics(goal) -> "Is auth implemented on all endpoints" / "Has code touched anything else than auth" / "is this following the best practices" / ...
For all we know this might be a non-language-generative transformer e.g. a transformer where the decoder produces confidence scores rather than language. Please provide more likely architectures if you know them, I'm genuinely curious.
And if you buy their RLCD claims, this might be even better than huge models that know a bunch of irrelevant things.
"Jev: New frontier model 40-400x cheaper and 20-200x faster"
I'm not the gatekeeper of who gets to call themselves a frontier model, but I don't think most people would count Jev in that group. It sounds false.
If their specific claims hold up, then it would make more sense to say something like:
"Advanced the speed/cost frontier for structured decisions"
I think AlphaFold and related are also frontier models.
Being an LLM does not seem like the qualifier for frontier.
questions = {
"refund_requested": Noul(
instructions="Does the customer request a refund?",
),
}I mean, imagine comparing this thing to Astra, it's hilarious. They don't even tell you what the max input size is, and they only allow 10 possible answers to choose from for the Choice mode. It's probably like a 1billion param model. They say it's "not small", but there's zero reason to believe that.
I suspect someone will be able to recreate this within a week by piecing together open-weight models.
Frontier LLMs are expensive jack of all trades. You can absolutely compare them to purpose-built tools on any domain they touch. Engineering is all about assessing tradeoffs.
> Also "can't hallucinate" seems wrong? Sure, it can't emit an invalid type, but it can still emit a completely wrong valid value.
that is likely true of all ML! perhaps we could debate semantics, but I don't think it's fair to say a random forest "hallucinates" in the way LLMs do
The original title before it changed less than an hour ago was:
"Jev: New frontier model 40-400x cheaper and 20-200x faster"
I'm going to agree that was misleading.
And on the second point:
>>Also "can't hallucinate" seems wrong? Sure, it can't emit an invalid type, but it can still emit a completely wrong valid value.
>that is likely true of all ML! perhaps we could debate semantics, but I don't think it's fair to say a random forest "hallucinates" in the way LLMs do"
Also going to disagree here, and I don't think it's semantics.
Type safety is not factual correctness.
I very much agree with this and want to hone in on where do actually disagree. Would you say a linear classifier hallucinates?
IMO your system can make mistakes that are similar in spirit to hallucination (i.e. answering with a false answer instead of abstaining to answer).
From the intro blog
"Hallucination and type-safety are intrinsically related"
I'm not entirely sure why we're conflating type safety with, I guess, value or output safety.
"Would you say a linear classifier hallucinates?"
No, but it can be (and often is) mathematically correct and functionally incorrect. It doesn't help to say "a linear classifier can't hallucinate" when you get even 99% accuracy. That's 100% a semantic play, and it doesn't help when the picture of a dog is labeled cat and the response is "yeah but that's not a hallucination, only stupid LLMs do that"
Model input: "Does the user want to speak to a human support agent?"
Output: Yes.
I imagine that your model would produce this, and I think it's fair to say this is a hallucination. A human would caveat it with: "Yes, but not right now.", your model is incapable of that. Yes is technically correct, but within the context of being in a live chat, a human would understand that the caveat is required.
You’ve already agreed that this doesn’t establish correctness. An approve for an unauthorized action still meets the schema guarantee.
That’s why I find the messaging misleading. You’re acknowledging the limitations in these replies while defending the broader reliability pitch.
Even granting that each answer is calibrated individually, that doesn’t establish calibration of the decision that combines them.
Sure, I can threshold a composite score, but there may be many wrong answers with the same score. An unauthorized action doesn’t become acceptable because it scores highly on the other dimensions.
I still have to define the constraints and test which wrong actions get through the complete workflow on my own data. That’s a substantial part of the work being pushed back onto the developer.
linear regressions hallucinate in the simpson's paradox.
the model output can be quite confident and not representative of reality
- Give Jev and LLM the same input
- Lock down both to approved/rejected/unknown (LLM restricts on decoding)
- Both can be wrong, but neither can hallucinate (invent an another option).
For e.g. classification tasks, even in 2026 people are doing things like hallucinating deliberately, and then matching the hallucinations to embeddings -
https://softwaredoug.com/blog/2026/08/10/hypothetical-classi...
With TypeSafe it just picks the class (actually probabilities across classes), reliably every single time.
(To be clear, this one raw model does not support outputing a full AST directly - it wants to output "choice" among fixed options, "score" on a sliding scale, or a true/false answer (all of these with confidence scores attached), so building the AST/structure would be a code-driven (or even perhaps outside LLM-driven in some more challenging cases) multi-step affair where the model would essentially be playing a "game" of building the structured output step by step and getting a revised partial state back. But one could expect this to lead to interesting results.)
- @CompleteSkeptic
Very strange.
My first thought was that it would be ideal for robotics? As in control of limbs, general planning, route finding, etc.
I foresee this to be the path moving forward - giving AI models understanding of the computation directly(as well as compositional rules) This feels like a short path towards total software in many areas.
The comparison between LLM speed and Jev speed is misleading, because they're using autoregression to generate all of the type names, all of the schema, etc. A closer comparison would be if the LLM was purely outputting the raw numbers. Even then, comparisons to LLMs are pointless because you could train a transformer on the same sort of task that Jev is doing and get even better performance yet again, and a smaller model. I suspect this is some form of stripped down diffusion language model.
You really have to do a lot of hand holding here, and map out your problem space manually, and very carefully, to get any sort of accuracy. For example:
> Keep each Score to one dimension. If a description says “punctual and smart and experienced”, the question is measuring three things, and an input that is high on one and low on another can’t be placed. Confidence drops and the score means less. Split it into one Score per thing and combine them in code
If you don't perfectly represent the distributions of possible answers then you'll likely get garbage results. As far as probabilistic state machines are concerned, I'd say creating the distributions of possible answers, and their hierarchy, is the actual hard part.
One of their examples is:
- "state": "I have asked three times now. Can I please just talk to a real person?"
- "Is the customer asking for a human agent?"
Imagine the users request is: "I want your human agent to call me tomorrow at 5pm."
Human conversation is fuzzy, getting useful reliable results out of this is going to be a challenge. Of course, you could add follow up checks like: "Do they want that now, or later?" -> if later -> "Do they want that tomorrow, or the day after?" and so on... But now you're building an LLM out of if statements. I am skeptical of whether this model has much utility for fluid language interpretation - I suspect it'll only be useful for scenarios where you've tightly constrained the answer space but want to use fuzzy language to describe it. Like:
- Question to human: "Would you like a support agent RIGHT NOW?"
- Their response: Yes | Yeah | Mhmm | ye sure (any possible yes signal)
Model input: "Did they ask for a support agent?"
Still... a tiny LLM could accomplish this sort of thing without problem. And that doesn't stop someone from saying: "No, not right now. But tomorrow." - and the tomorrow would get missed. I think this is why people haven't really tried this approach much already.
Also their Doom demo is on structured state, not on images. Meaning, the enemies must be being served to the model as coordinates (or the exact angle of projectiles that hit the player), otherwise it'd have to scan every pixel of the 360 degrees to know whether an enemy is in front of the crosshair or not. You can see from the map below that it's also choosing travel checkpoints/destinations through walls. So they've severely cooked this to make it look far more capable than it is in practice, and any speed advantage that is offered here is not factoring in the shortcuts it is taking, the training on the map, and the fact that it can cheat because the structured state it is using is not bound by obstructions.
Here is their docs by the way: https://docs.typesafe.ai/ - so you can understand how it works.
Seems really cool.
Side note - just like most people don't need an intelligent personal assistant to manage and respond their emails and book their flights, most people also don't need smart homes. Century old toggle switches are more than enough in a 3 room apartment or 5 room house unless you have a mention.
There's no reason to not do both.
Also a quick NFC sticker in each room taking you to a small HTML site containing settings (temp, ventilation, lights, shutters, setting a alarm by the lights) has been golden.
No one wants to: download Shelly app + AC app + look for ventilation IR controller + figure out how casting works for the TV + figure out how to use the Shelly app to turn lights into an alarm. It's too much friction for little gain. But a quick tap? Great.
But tapping your phone on a NFC sticker bringing all those controls together per room in stead of per category (all lights in Shelly app. Person in room #1 has no interests in the lights in room #4 at the same time.).
IF you tap it while not on Wifi yet it just tells you to connect to Wifi. :-)
One "all house" sticker next to the front door allows any last person leaving or first person entering to put the entire house in active / idle mode.
Works wonders. And as soon as local AI is quick enough the stickers will be a microphone!
We have 8 light buttons in our living room/dining room/kitchen space. It is very convenient to us that we have 1 button for turning all of them on/off at the door to upstairs (at night turn off all lights and go to bed upstairs, in the morning come downstairs and turn on all lights) - but also have 1 on/off button near our back door for when we leave/come home.
Next to that: on/off toggles a schedule where the lights are bright and cold-ish by day, and low and warm by night without us having to manually adjust each light every hour or something.
Again, need is a big word. But it's very convenient and pleasant.
I agree that it was notable that they delegated to an existing LLM, but I don't think it detracts much from the value proposition (not yet proven) of their demo.
For those who don't know (which is probably everyone but me), I ported the design-by-contract pattern in Python and combined it with LLMs. This was early 2025. I originally wrote about it here: https://leoveanu.com/2025-03-01-dbc/ . Contracts are a core feature of SymbolicAI ever since. The community seems to have loved it too (https://news.ycombinator.com/item?id=44399234).
I think I'm starting to glimpse the implications and it's gonna change agentic workloads if it holds up to scrutiny. It's too early for me to tell anything other than jot down some rough thoughts.
In short, you get blazingly fast semantic branching you can use in control flows. For contracts, I can now directly take the data model that you have to design and convert it into Jev's expected format. Or I can use Jev for semantic branching in postconditions.
If my understanding is correct, that should be doable, but I need to think more about it. It could be that with Jev I can finally “compile contracts” and better chain them into workflows, which is something I always wanted but didn't know how to do properly.
Eager to test. On the waiting list.
GP's first sentence isn't arrogant (at worst displaying a bit of false humility) because it's saying everyone but him doesn't know about a thing he did. Your second quote you apparently mis-parsed because of a minor English error (he should have said "to Python" rather than "in Python"), but to me it was pretty clear what he meant.
Unfortunately, none of this is explained in the announcement, but the documentation [0] is pretty good.
[0]: https://docs.typesafe.ai/concepts/how-to-build-with-system-o...
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
with TypeSafeClient() as client:
response = client.system_one(
state={"document": "I was charged twice. Please fix this ASAP."},
questions={
"billing": Noul(instructions="Is this ticket about billing?"),
"tone": Choice(
instructions="What is the customer's tone?",
criteria={"calm": None, "frustrated": None, "angry": None},
),
"urgency": Score(
instructions="How urgent is this ticket?",
criteria=["can wait", "this week", "today"],
),
},
)
print(response.nouls["billing"].noul)
print(response.choices["tone"].choice)
print(response.scores["urgency"].score)
Here's how this would have likely been made.
- Tiny transformer or equivalent model (maybe a few bn or so?), explaining latency and cost
- Questions are sent in parallel to multiple copies of it (I'm sure they're edge located)
- The model is post-trained for calibration in a wide variety of data (the recipe is relatively simple, and likely targeted on distillation of logprobs / confidence of a bigger model)
Notice how cost is ONLY for input tokens as output is merely numbers (few tokens) because input could be huge (questions and options).
At 0.042-per-million price they have, Astra estimates the model to be 3bn parameters.
One could replicate this by post training Qwen 3.5 2Bn. I expect people to do so soon!
This is something I focus on in a bunch of my experiments - how to get immense value out of tiny models (<1b params). There are lots of different architectures out there and there is so much to optimize if you know what you are asking and have a grammar to constrain with.
Great to see this and I hope this is a lot on top of what is already openly available.
I need to rank 1000 articles and pick the 5 most relevant for the user? Jev.
I need to audit and strip out content because my user is affected by regional privacy laws (without hallucinating)? Jev.
I need to surface the 3 funniest media comments that match the user's sense of humour? Jev.
I need to ...? -> Open-weight model.
I'm sure someones working on this as we speak using an open-weight LLM base (Qwen or something would be a perfect fit).
This sort of task is a perfect fit for a very small model capable of semantic parsing. You can get away with a LOT less parameters without all the autoregressive generation and long-context reasoning.
It looks like the model takes as input a state (structured text? not sure if multi-modal) and a question (as a "Choice", "Score", or "Noul") with some additional augmentations possible. Then outputs the question's answers as appropriate (e.g. a choice, accompanying probabilities, confidence).
Edit: On the AI primer page, it looks like they do the RLCD on a pre-trained base model?
I do agree that the comparison to LLM tokens is hard to understand (also because output tokens are not comparable).
But yes, text or structured state (like a JSON with multiple pieces of text in) -> decisions out (e.g. choice maps to "match" statement, "score" maps to sorting, "noul" short for bernoulli maps to if-statements)
(I'd also argue something like p_yes or just probability might be a simpler name, but I'm sure there's a better reason behind Bernoulli maps).
"instead of autoregressive string output it instead outputs structured type-safe 'decisions' with probabilities/confidence scores, each generated in parallel
so sort of more like a Large Classification Model than a Large Language Model? or, maybe better to think of it as a sort of "shift left" in the LLM's transformer architecture, allowing you to replace the predefined token vocabulary of an LLM with a prescribed set of 'decisions' that need to be made based off the input context; and exposing those probabilities directly so they can be integrated into the system logic, instead of just sampling from top-K.
all of this while still being instruction-tuned (!!!)"
It's always been possible to build classification pipelines using LLM embeddings as the input. seems like this is a much more sophisticated / useful application of that concept
the one nuance I'd get into is I'd call it "zero-shot" over "instruction-tuned" (the latter often implies a particular distribution), but very safe for sharing
The Doom demo really help me, at least, to understand how System One differs from LLMs. However the first demo (Side-by-side demonstration) - I'm struggling to understand what is going on here!
The demo is showing System One producing its output in parallel very quickly and for little cost compared to an LLM generating its answers token-by-token. The "noul" type is used to evaluate a yes/no question and return the probability that the answer is yes.
So this demo is showing System One offering much more nuanced responses and specific probabilities compared to an LLM's more crude responses (e.g. LLM shows "true" or "false" compared to "0.9" or "0.07" probabilities that the answer to some question is true).
(shilling my blog post on that jaggedness: https://www.completeskeptic.com/p/lies-damned-lies-and-bench...)
Do you have an architectural explainer yet for Jev or are you holding that close to your chest and letting the magic rip for now?
I don't want to shill my blog too much, but I will say data is probably far most interesting than architecture: https://www.completeskeptic.com/p/the-bitterest-lesson
Input, and criteria/instructions can both be defined as structured input (JSON). This ends up being pretty powerful because the model is trained to understand structure.
e.g.: https://docs.typesafe.ai/primitives/advanced#structured-inst...
> not sure if multi-modal
just JSON... for now :)
> outputs the question's answers as appropriate
correct!
I can't really intuit how I should think about when the model will be accurate. Is there somewhere to read more about that? I assume customers would just have some tests or talk to you.
Unless they're hackers, no. It's not really a chat interface, it's meant for consumption by machines and composing into higher level systems (pairs great with LLMs).
> Is there somewhere to read more about that? I assume customers would just have some tests or talk to you.
We're going to release some more info on evaluations over time, and yeah, join the waitlist! We offer faster access in exchange for good memes
this is complex, but generating text is highly complicated and requires mode dropping to make long cohesive text
Jev's focus on structured I/O and confidence scores are game changing. If this does at all what it claims, I think this is going to quickly become the new standard approach for agentic systems.
Full disclosure, I am not they :=)
Decision making isn't that of a bottleneck I suppose.
A lot of people have become prompt maximalists, asking for complex multi-part solutions or dynamic workflows in a single prompt. You can get this to work sort of reliably with frontier models, but without much confidence or clarity where things might break in practice. My goal is to strip out as much determinism as possible from prompts so the LLM only needs to handle a narrow, well-informed decision, like "Pick one of these three things" and build around the answer. Sometimes you need to fill out a whole JSON payload and LLMs really actually suck at manipulating and adhering to JSON. They do ok now because labs have put in a ton of effort on making harnesses play nice with structured data. But it comes at a high token and context cost because under the hood I suspect the model is churning invalid text repeatedly until it gets around to passing some internal validation.
I am using the single prompt approach with GPT5.4, which is free, but it’s not reliable. Using Jev I’d decompose the prompt into a bunch of smaller questions, then I’d combine the answers in software. I’m super excited to try Jev out.
But agreed it'd be very useful to see it deployed on other hubs, and it seems worth it to provide the bespoke API format. Perhaps Typesafe's API will end up becoming the standard for a new type of structured model, the way OpenAI's API did.
I guess you can choose your provider still? But isn't the point that the lowest bidder is doing inference?
So, assuming this is not vaporware, this would raise the tide for everyone because it shows what's possible.
We already had encoder models that skipped text generation for giving us a numerical output that could be computed as a probability. we also got no hallucinations and faster inference for free there. So we already had
1. "unstructured state in, probabilistic decisions out" 2. "orders of magnitude faster and more efficient"
What was hard there was to train the model head without ML expertise, and considerable amount of data.
This seems like this is a democratization of those encoders? The addition over existing encoders seems to be coming from being able to specify the output shape (up to a cardinality of 255). It is unclear to me if this is possible using Jev without additional labels for fine-tuning.
If so, that is still very impressive, but I think the faster inference and 0 hallucinations might come for free, from it not being generative.
also "70-500ms vs 3-329 seconds" are apples-to-oranges unless the LLM baseline is doing comparable work (e.g., long chain-of-thought). If Jev is skipping generation entirely for a narrow structured task, of course it's faster.
Nonetheless i want this to be true, so I'm looking forward to Jev
Edit: I really have to say that I like their manifesto https://typesafe.ai/manifesto
It's totally reasonable to compare against LLMs doing chain of thought if it gets comparable performance.
I think this is reasonable if people are actually using LLMs to solve this type of narrow structured task, which they are. The evidence is that every LLM provider has some method of forcing the output to conform to a json schema in their documentation.
can jev play battlefield six for example
Their manifesto: "you only build on top of it if it's trustworthy." - the irony of this while putting out the most misleading, dishonest marketing campaign I've seen in months for their first public appearance doesn't exactly scream "trustworthy" to me.
Yes, I think it's a mistake that many organizations are cramming LLMs inside of automated pipelines where the extreme generality/flexibility of the model is at odds with the fact that you're using it for a very specific task that gets repeated over and over, and needs a very specific structured output to be successful. But specifying your task carefully (as well as deciding what counts as your input state representation etc) seems like a form of programming. Something (a person or a model working in a relatively unrestricted way) will need to produce a configuration/specification for this system.
So rather than Jev vs Claude I imagine that using Claude/ChatGPT/whatever interactively to define / refine your Jev config which then runs in prod might be the happy combination?
> Output tokens: FREE (too cheap to meter).
Insane. The video demos are really compelling, in particular the speed.
> Structured outputs slot into ordinary software as fuzzy decision rules: classify, route, score, extract, or branch where hand-written logic is too brittle. The surrounding code constrains their freedom, making them easier to compose into reliable systems.
I buy this vision. A lot of LLM integration I see these days is ultimately exactly this. OpenAI-style structured outputs works decently but this would be a great improvement in cost, latency.
constrained decoding (OpenAI-style structured outputs) make models dumber unfortunately - the short+dense version is that simply masking logits is insufficient because if ever a model was assigning probability to an invalid token, the model is by definition confused. you'd be better off erroring IMO
I'm very curious how much ressources are needed to run such a model. This could be a complete game changer for local applications.
FAQ: Is Jev just a smaller LLM?
Jev is neither small nor an LLM, hence being off the intelligence Pareto curve.
Image in documentation: https://mintcdn.com/ts-docs/aFVnpmCIX68NpsV1/images/ai-prime...
While this model may share much with GPT-style models on the encoder side, it clearly has a different decoder architecture. So is a high-parameter count language model an LLM even when it doesn't have a GPT-style decoder? The definitions are in flux.
They claim it's not an LLM, which I read as "not an auto-regressive token generator". I assume they are still using a transformer, otherwise they would be talking about the thing that's not a transformer, instead of all the fluff on the linked page. But they emphasize parallel generation, so is it like a text diffusion model?
you pass in your "prompt" and options (described in natural language) that it can respond with, in addition to your input. it gives back that option set with a probability assigned to each one
Rip there goes my excitement. I have a task that something like this would be great for but the list of options is a zero or two larger than that xd
Typesafe.AI sounds like some typescript/structured output type of tool…
What even is “system one” ?
IMO the product/tech is really there, just needs better communication.
I think that this specific part is not super interesting if your harness just recovers from invalid LLM outputs.
The latency and cost - yes, those are super interesting.
That would be the litmus test.
"Does not hallucinate" is not the same as "is never wrong".
So the ATC test could be the benchmark.
The doom demo is quite cool
Instrument your game to output properties of entities near the player and the output is the various control inputs - moment to moment gameplay gets solved. Maybe augment with a tick-by-tick controlled stepping mode if particularly twitchy - an LLM can take care of the higher level reasoning then.
I suppose this is the same video as the one from the parent comment, but I don't know for sure - I don't have a twitter account and the above link doesn't work for me.
I can see the individual tweets in the browser while not signed in though.
It's in the parent article under a section named "Doom" in case that asset URL ever changes.
Just joined the waitlist, excited to try it out!
OpenAI has been teasing how fast computer use is with their models running on Cerebras chips but the difference here is a burning hole in your pocket.
https://developer.apple.com/library/archive/documentation/Ac...
Then, you can serve it faster/cheaper than the frontier LLMs. It's basically distilling a small but extremely common use-case from LLMs and serving it. Then RLCD comes into play to update weights when a new model comes out, etc.
Any thoughts on what the next potential "cheap" win to be distilled from frontier LLMs is? I'm going to need to play around with this.
https://fastino.ai/blog/gliner2-5-span-free-information-extr...
im thinking about how well Jev could be used to replace a current LLM-as-Judge evaluation workflows, specifically on chat transcript data (think ~1,500 tokens) i wonder if the reasoning usually required pushes it a bit out of scope. didnt see anything published about constraints on the state size, so would be curious to hear about that.
And each question is a separate single token model completion done in parallel
If you work at TypeSafe please try this.
Side note: This is probably how LLMs would perform with better encoders and next-latent prediction, so eventually those will beat this architecture out. Still amazing though.
I'd love to do research on this when I have the time.
That's what I was insinuating through "better encoder"; the model creating more efficient representations of ASTs using something like JEPA
we do think there's a lot of potential though and do want coding themed releases soon
As you said, if it worked, they would have demoed it haha
youd ask it to pick a location on the ast to add something from the grammar?
i dont see how this stays confined well enough? make a new output space every time? does that end up auto-regressive?
- is the capital of france paris?
- it is august. is it raining in paris?
(forgive the examples; they're probably not semantically the sort of thing jev is trained to work on. but i figure the point translates to various kinds of questions that come up in "inner loop of agentic pid controller" contexts)
a normal text-generating model if asked to produce a number will also do that just fine. i assume in jev's case it was actually rled to essentially learn to express priors over things using its implicit world model, which definitely ought to help, but can we say more?
However I don't understand how are they claiming zero hallucination, how does giving confidence score fix hallucination? or am I missing something here?
The one downside is that the context window is very small (32k.) So some initial ideas we had for initial evaluation of code reviews won't fit yet in the window.
LLMs are universal simulators, their latents model the world. So I bet if you compare their logprobs with probabilities output by this model, it will be highly correlated.
Someone should do this quick experiment. I bet there won't be enough of a meaningful difference.
Founders response to a similar question on X: https://x.com/CompleteSkeptic/status/2100067328620896408?s=2...
pasting it here: zero-shot + general == programmable
I would assume any extreme scale narrow task could then be fine-tuned for, but we'll see - I suspect putting it all in shared cognitive core has bit maintainability/generalization benefits
Last 32k of connect + Summary of current task: Did we learn something useful here (true/false)? What is the category to file it under? Then notify the LLM to file it away.
What class of memory might be useful here? Model gives probability to each item in the list. Short description of all memories ordered by tagged class is used in the next round. Are any of these memories useful in the current context, such that they will inform the model and help in its task (yes/no)?
I’m sure there’s some fine tuning to be had, but this sure seems like the basis for a substantially better proactive memory system that works around an existing LLM conversation.
If I’m understanding what this does and how this works (generic input, intelligent classification with probabilities, rapid and cheap), this is absolutely nuts.
> Search for relevant information over giant corpuses
Do you mean as an alternative to embeddings?
Thanks for putting this together. I'm surprised the cost saving is so little though. I expected much more based on the post.
Jev input … > Unstructured data (e.g. text) with an emphasis on structured program state.
What pre-training data/model is Jev based on? Surely result effectiveness is dependent (outside of one’s own input as “state”) on that?
> [them] Output tokens: FREE (too cheap to meter).
I'm very confused by this.
[0] https://goodstartlabs.com/research/verification-is-the-bottl...
If self driving is red means stop, green means go, and stay in your lane - then it would work great, but having to actually think and test which maneuver is optimal for a given situation while weighting safety, road rules, random unexpected actions and getting to your destination, I think it's a much bigger problem. A bigger model specifically trained on that maybe would do great, but then the output is not the constraint anymore.
But I haven't tried the model, so I 'm just ballparking and could be very wrong.
Edit: never mind, found https://docs.typesafe.ai/introduction/quickstart by now
It might be boring, but I can see exactly how I could use this right now to improve my agentic rag.[0] In two months I am supposed to deal with a giant corpus, while still maintaining responsive chat UX. I have been working my butt off to make our first big client happy. This could really help solve the chunk ranking problem.
[0] assuming the policies are compatible with sensitive production workloads, some time in the near future.
But in things like programming, most concepts are abstract - 'if hungry eat an apple' in programming terms would look like
'if hunger > 50 {apples--; hunger-=30;}'
and compilers work with 'concept erasure' - to them, tokens (which are like llm tokens) look like
'if var1 > 50 {var2--;var1-=30}'.
They don't care about how these things map to real concepts. So all the embedding directions used to encode real-world concepts are just noise to LLMs when programming. This greatly reduces dimensionality and training costs. So does a token representation tuned for programming constructs, rather than natural language would probably have a more efficient encoding.
On a more serious note, it will be fascinating to see how this different spin on modelling inference will create new paradigms or slot into existing ones.
I think this could be pretty decent in CI? There's a lot of "flakes" I've mediated that this could have handled much more efficiently. Maybe observability as well, triggering elevated logging and other initial measures?
The goal is going to be to use llms to distill operations down to some dsl, and pass it into something like Jev.
Is there anything published on how it maintains calibration? Or when you say "outputs calibrated probabilities" you mean "as calibrated as frontier LLM models, just cheaper" - which is a different claim; as LLM's aren't particularly well calibrated
Given that, can't you just replicate Jev by telling Astra "here is the question, you must make a multiple choice decision / output a score between 1-10, please answer directly in a single word, no reasoning allowed"?
(Edit: Ok, Jev is much cheaper in input tokens so these two aren't directly comparable at all)
we actually use astra (and fable) in this way for our evals: evals.typesafe.ai
someone on the team cooked hard on that and it shows example traces comparing our model to opus/sol
I like the idea of System one models but all LLMs so far work as system 1 thinking because humans generate speech subconsciously with system 1.
System 2 thinking requires consciousness which AI does not have, so even reasoning models are still system 1 thinking as system 1 in humans has reasoning with heuristics.
Its limited but most people navigate the world with it completely, so it's enough for AI.
You could theoretically ask “what is the next appropriate character?” and add the entire ascii charset but i doubt it’d work well and you’d be implementing autoregressive churn across network latency…
[1]: www.expressionsofchange.org
1. Do you provide any kind of largest common subtree caching for cheaper input?
2. Have you tried auto-generating Lisp programs structurally?
3. Have you tried augmenting a Lisp language with a `choice` function that makes choices given a prompt, the environment, and the continuation stack?
(2-3) No, but that's kind of a sick cook ... Want to get access and try it? nathan@typesafe.ai
Maybe I'm just not doing a very good job at prompting Jev, but I think right now it's not quite capable enough to generate Lisp code.
Link: https://console.typesafe.ai/playground?share=shr_148e1248984...
it is a structured data model, but technically not a language model (it doesn't generate language)
I’m guessing it might be able to replace maybe 40-70% of LLM calls for a given pipeline depending on the business task, cutting the API costs on those calls by an order of magnitude.
1. Start with an LLM, so that your model understands natural language.
2. Replace RoPE with a tree embedding scheme, and causal attention with a sparse attention on the graph structure. (You could use full attention... but it's cheaper to use graph attention.)
3. Chop off the final unembedding layer, replacing it with a projection down to two scalars, one for logits and one for confidence.
4. Each option of a choice is represented by a number of tokens in leaf position; average these tokens' logit outputs to get the option's logit. Average all of the confidences from all of the options to get the choice's confidence.
5. Train the logits by KL divergence from a true distribution (or NLL on samples from a true distribution).
6. Train the confidences on a subset of the data in which you know the entire true distribution.
The hardest part is getting real world data for workflows, but I wildly speculate that you can get by with only ~50,000 documents if you first adapt domains using synthetic data.
Except this, this thing looks like revolution.
lol, I bet they would publish them if their score on those benchmarks were good.
Finetuning a language model for decision classification (with probabilities) is already well-understood. What specifically changes in the training objective with RLCD? Are its benefits isolated from Jev’s new architecture/parallelism?
Either way, really cool and impressive.
Yes, that’s the kind of attitude I want to see in these model releases
But also effectively this is a classification model. It excels at specific certain types of workloads, and obviously will fail at others. Not really sure how one benchmarks this tbf. I can see their argument on why this requires a novel specific eval for whatever your usecase is. A consistent "global" benchmark might be hard to do
This website does not inspire confidence at all, it all sounds like a marketing piece. I wish it was true, some kind of text-prompted classifier with LLM performance would be cool, but I can't trust it with what we are given.
(though ideally everyone gets off the waitlist and can try it out for themselves )
Does this imply it's a very small model? I couldn't find anything about the model itself.
> we assume there is a correct compute graph (a “workflow” represented in code) and use the predictions of the largest, smartest, and most expensive external models as reference probabilities. ... Rephrased: every model gets the same workflow. We test how they compare to the average of the smartest models (in this case, Astra and Fable).
They assume there is a correct graph, but they don't compare to that, they compare to the average of the smarts models? So the smartest models are getting it wrong but you compare that anyway as a benchmark? So the outcome is "how much of a Fable am I getting" etc. Why not compare the actually correct thing?
But then even on this hand constructed eval, the first plot is showing Jev at less than Sonnet 5 accuracy. It is barely better than Luna. There are two Opus 5's and two Sonnet 5's without explanation. What is the plot showing?
I gave up.
I think we'll see a bunch of different architectures over the next five years.
we also thought the voice at the end was AI-ish, but apparently that's a real voice actor but slightly sped up
Reminds me of this: https://www.reddit.com/r/ITcrowd/comments/tg05j1/i_cant_beli...
forget LLM benchmaxxing sidequests, I'm sold on the real benchmark
I think the interesting thing would be seeing if prompt injections still work with this kind of model.
edit: looks like a framer export where there is a text stroke being applied :|
Ouh! Any open weights models that can do this yet?? If not, how much longer? I have a Mac Studio coming soon.
> LLMS > Strings / generated text. Strings are flexible and can be anything: chat responses, code, hallucinations, refusals, or even type-safe structured values. To be used by software, responses need to be parsed + validated. There is also always some risk that the AI goes off the rails.
> Jev > Type-safe structured values. Possible outputs and structure are defined in advance. The model never makes type errors. All answers are accompanied with calibrated probabilities and confidence scores.
I mean, this isn't even remotely comparable to LLMs so why compare? Also, why are they bringing up AGI given there approach is so restrictive that what they're building literally cannot have the creativity required for AGI? The video is 100% marketing slop...
The bulk of the application of LLMs is that they generate reasonably reliable text which doesn't need to be defined in advanced. I'm sure there is a niche for this and congrats to the team, but please let's not hype this as if it's the next big thing in AI...
What a sad world would you live in if you don't keep creativity for the humans.
I'd use this as a tool an LLM can use for specialized tasks. It's not AI in itself.