You trust AA since the sol fiasco a month or two ago?
replyWhat did I miss?
replyThey updated their methodology when the new openai model didn't rate higher than the old one, or something like that.
replyI do t think it was a fiasco. I thought the updates were good. They should do more work on benchmarks. There has been good work on testing benchmarks for example to ensure they are accurate, and as issues arise they should change the benchmarks. The point of AA and Arena is trying to understand how to think about models.
reply[flagged]
replyWhy not just try it yourself?
replyIt costs money to run serious tests. We're way past the time when you could get an idea of the new model in a couple of tricky prompts. Now I need to compare a whole resulting project to get a clear view. If someone else proves the benchmarks don't hold up in reality, then it saves us further tests.
reply