upvote
This likely isn't AA putting its finger on the scale for specific interests, rather in the case of Muse/Astra, it's to avoid looking really bad because it scores things way out of wack with people's anecdotal vibes on model capabilities. It is possible that labs are getting preliminary AA results back and raising concerns about scoring that then get folded into updated benchmarks, but I strongly doubt it's nefarious.
reply
They announced their new Index v5 yesterday, they frequently add new benchmarks and rebalance old ones.
reply