undefined

points

[-]

Hey there! If you end up trying out needle on the test suite it would be very useful for us if you could share some failure modes of the model! We are always trying to understand where the model isn't doing good and where we can make it better.

For your question on tool calling, I think you will find that the model is pretty good at simpler tool calls and parallel ones, but can struggle with implied references and multistep reasoning. These are definitely things that can improve with task-specific finetuning but for some things you just have to have a model that is properly sized. That said, we are always trying to improve the model so that it can handle an ever larger set of queries

by viccis13 hours ago|

parent|

[-]

Thanks for responding. I ran it just now and it looks like it could be useful if I change and limit the scope of the kinds of actions I need. Here is a breakdown of where it struggled vs some local models (~8-12B parameters running on a 16GB Mac Mini or my desktop's 3080), bear in mind I had Claude integrate it into the benchmarks and this is its interpretation, not my own:

What it gets wrong:

- It copies numbers instead of converting them. "25 minute timer" becomes duration_seconds: 25, and "twelve minutes" becomes 120. The first one comes with 100% confidence.

- It picks the wrong action. "take the paper towels off the list" became an add. "remind me in 20 minutes" became a timer. "add five minutes to the pasta timer" became a new timer plus a cancel.

- It never declined anything with our full tool set. Background chatter became note_save "blue one" at 0.99 confidence. "play some jazz" became a screen card, and "wake me up at 6 30" became a 630-second timer.

- It can't use household context. Notes, timer names and reminder IDs have no place in its input. Passing them anyway made results worse (5 of 27 single-turn requests right, versus 8 of 27 without), so the backend now leaves them out.

- Follow-ups mostly broke. "take off the last one" removed the whole list.

by HenryNdubuaku13 hours ago|

parent|

[-]

This is extremely useful feedback for us, thanks! I think the easiest thing here that can be fixed with tool definitions is the number conversions. Additionally, the model tends to work better with fewer tools. We will definitely be focusing on better context usage and followups going forward as well.