Sorry this was confusing -

There are 30 tasks in my eval total:

- 20 of them ran on Nov 1, twice each (so 40)

- I created 10 more tasks and ran them on Nov 5 (also ran them twice each)

- Today I ran all 30 tasks, once each, on Opus 5.5 to see if the performance decreased at all

- I only put the comparison against the original 20 in my post, since I saw in another post that people are claiming Opus was nerfed on November 3

This will all get less confusing when I make my benchmark public (hopefully this week).

It stands as written? The previous test had 20 tasks run twice each, so denominator 40. The recent retest did the same 20 tasks but only once each, as mentioned in the caveat. The 30 is presumably either a typo or a separate count (like the test had 20 tasks related to code with 30 bugs/features).

Also - LLM text? If GP is confusing it's because it's overly succinct and might have typos, which are not exactly the hallmarks of LLM text.

2 alternative interpretations to consider:

1. 30 was a typo, it should have been 20.

2. 30 was the full suite, but they only ran 20 of those 30 on Nov 1st and so ran the same corpus again for a fair comparison.

I think it's confusing and incomplete, but not wrong. Paragraph 2 says the Nov 1 test included 20 of the tasks and ran each twice. That's the 40 denominator.

Why paragraph 1 even mentions the full set of 30 tasks if 10 of them are irrelevant to the comparison, I don't know.

I'm not sure if it's LLM text, or somebody who has been reading way too much LLM text and is now corrupted into mimicking it. My guess would be a mix of human and LLM. "...evaluated by tests the agent didn't see" sounds very Claude. The first line (2 sentences) sounds human to me. Everything after sounds AI, though the last line is ambiguous.

It's kind of sad how our meat brains are getting non-consensually trained on this categorization task....