Or, I don't know, make your own vacuums.
One could maybe autogenerate these text planning commands, but it would require a map and the robot's current location, so it doesn't really solve that, unless it can find a specific thing completely on its own. How much of a planning horizon does it have?
The advantage over traditional approaches is presumably flexibility. LIDAR isn't going to solve an instruction like "find the man with the pink shirt".
I would like to know what it did the other 23.4% of the time!
On their page where the result graph is, go to navigation error, that's the one that matters for your question, and you see their model is great at not navigating "wrong", so their failure rate was that it couldn't figure it out.
[/joke]
here's the link to the PIGEON paper - https://lukashaas.github.io/PIGEON-CVPR24/
It may come late but it‘ll be safe and reliable. It also requires a lot of OCR.
Costs of simple tasks grow extensively: OCR with "Mistral OCR" at $4 per 1000 pages vs OCR with Opus 4.8 at sometimes¹ $1 per "page".
Or just the immense costs when burning tokens in an unoptimized agentic coding environment costing tens of dollars for a few simple classes or functions versus a highly optimized "autocomplete" model costing under $10 for thousands of such classes and functions.
Or the, over ten dollars worth of tokens when some "agent" using a general model, tries to perform the task I gave it to "read the event on example.com/event/1337 and put it in my calendar", include commute time as well"
The "general models" currently only become smarter by growing bigger and having larger context windows - by becoming exponentially more expensive to train and to run and to interact with. Whereas "Niche" models can do the things that "normal code" cannot do, and improve by tuning and tweaking only that. Their goal is then to fill in gaps that traditionally are hard or impossible with normal software. Wheras the goal of a general model (with agentic reasoning)is to replace that entire "normal software".
One example: I am not interested in "chatting with my calendar". I'm interested in a calendar because it is a well known view (UI) of my planning and tasks, but I see a lot of opportunities where AI can improve my working with this calendar. I may be interested in a smarter screen when I hit "+ Add event"; one that has knowledge of my previous events and patterns (some RAG vector db maybe). One that maybe has access to content I just copied, or read (though: privacy?) or can open my camera to let me shoot a pic of something that has the event info on it. In such a set-up, Niche LLMs perform dedicated tasks: determine patterns (he always books a Yoga class on wednesday or thursday, two days in advance, so lets suggest a yoga class), determine existing content (event is planned 100Km from his home, so lets suggest the commute based on previous commutes like this). Or an OCR model. Or an autocomplete model. Relatively simple, niche models, called from within software to aid me when "calendaring". Not replace the entire calendar with some chat.
¹Edit: This was a rather unscientific research of mine, where I compared some models to read from photographs, compared purely on costs and timing. "Opus" or other generic LLMS with image input capabilities commonly did better on "performance" esp with difficult input such as a picture of a poster of some rock event.
If I need to pay someone 300k to make the model and infrastructure... then I would need to process many documents to recoup my OCR costs compared to asking claude code nicely.
Perhaps the model zoo is becoming good enough that the cost to find a specialized model is not so high?
Unless you are in military robotics or automotive of course :)
I'm also skeptical that you can cleanly differentiate between "safety critical actions" and "actions", though this is less of a practical concern given how laissez-faire some manufacturers are. For context, I work on safety critical robotics (in automotive).
And even if you use all the tricks in the book to make them work for you, the cost can easily be 1000 _times_ more than the specialized model. Ditto for speed.
This is especially important for things like robotics or navigation.
While impressive at 8B, what would the expectation be in real life, that it's run remotely or autonomously with a strapped on GPU and battery?
All I can think of are robot dogs, Tesla bots, and whatever flavor of the month Japanese robots show up at trade shows.
Robotics (using physics sims)
Cybersecurity (red team / blue team)
Math (using automated proof checkers)
Programming (using compilers)
For the record I think robotics is a totally logical place to use this training approach and this is very impressive. But if we zoom out and think about LLMs in general I’m not sure this inspires confidence in AGI arriving any time soon. I would also propose that this is a form of overfitting / training-test contamination.
Take cybersecurity for example. Through brute force techniques you will gradually memorize all of the possible exploits. So when fable breaks into a DoD network everyone is shocked but in reality it basically memorized all possible exploits including some zero day.
I’d be much more interested to see if fables performance is preserved as new exploits arise (NOT zero day - negative day meaning exploits that don’t exist yet). Would fable still find them? Or would they need to retrain it on the new software stack continuously in order to identify the zero days.
This is an important distinction that I have not seen made before.
This analysis by Toby Ord demonstrates why it’s a problem if frontier improvements are coming from reinforcement learning (brute force methods) from a purely computational perspective: https://www.tobyord.com/writing/inefficiency-of-reinforcemen...
But I'm scared for when those home helpers get drafted to fight in wars, either for or against me...
I imagine the EU would block any attempted takeover of Mistral given recent Anthropic and US govt actions.
SOTA 80% means a practically useless robot. What are they really imagining their ICP to be here?
Posts like this just reminds me of the end to end demos AV companies built in the early days using a single camera - only to realize that it’s harder than it looks years later into development.