like they have words that are dressed in scientific language on their site like

"We estimate a 95% confidence interval for Artificial Analysis Intelligence Index of less than ±1% - based on experiments with >10 repeats on certain models for all evaluation datasets included in Artificial Analysis Intelligence Index v4.2."

but where's the outcome dataset justifying this? how did they get that probability? what was the specific methodology of the tests? what variables did they account for?

there's a major difference between scientific sounding and being truly empirically rigorous. the 'research' in AI intelligence feels somehow even less trustworthy than supplement-funded studies because those are at least subjected to scrutiny by peers without profit motives

Completely discredits the index if it just gets modified to match social media vibes.

Theoretically it's unscientific to do tweaks like this but in reality this is what actual science is because you need to see the results understand the problems in your experimental designs.

Now the interesting thing is that you could repair the bias problem. The main issue is that you will do these tweaks after a bad result not after they look fine.

Maybe we could commit ahead of time that the experiment will be reanalyzed after results regardless of what they are. Instead of only when the results disprove the hypothesis.

So like artificial analysis committing to a fixed cadence of index updates instead of when the results start looking jank.