I'm wondering what's up with the release of Gemini 3.5 Pro, they keep postponing it. For a while, Google was doing pretty well with their releases.
replyHeh, probably something like this.
replyWorks fast - tells people how to overthrow the government.
Follows all rules and conventions Google wants - says corporate speak without actually accomplishing anything.
Can actually do complicated things- apt to tell the user to fuck off and do the hard work themselves.
Training models seems more akin to raising a kid than a computer application.
FWIW: "Baba Is You" is 7 years old and heralded as one of the greatest puzzle games of all times, with guides and solutions shared all over the internet. How to beat this game is 100% in the training set.
replyIt was our original assumption. Yet, we went through trajectories and agents did not recall solution. It is with a sharp contrast with task for which agents magically generate solution, e.g. https://openai.com/index/why-we-no-longer-evaluate-swe-bench....
replyIn a few instances (we covered it in Caveats) Gemini 3.5 Flash "knew" which level it was, but misremembered, and went with a wrong solution.
It turns out knowledge and ability are not the same thing. We should test both.
reply