undefined

points

[-]

I should add a F.a.q. for this question.

The suite is across many categories, not only coding, and most of the tasks are low-horizon (or what the opposite of long-horizon is), where the max thinking time is around 10 minutes.

Gemini models are really smart, unfortunately they don't play well with any harness, so hard to use in practice.

But try them out for one-shot tasks, they are really good. Don't use them for coding in a harness, but you can ask them to generate code/planning (still, for coding only other models are indeed recommended).

by markasoftware14 hours ago|

prev|

[-]

yep, the person you're responding to created the benchmark and is using HN comments as advertisement.

by XCSme13 hours ago|

parent|

[-]

I did, avertisment is a big word, as I gain nothing from the traffic, I run the website for myself, and some other people find it useful too.

Happy to hear what would make the website more useful.

by gaflo8 hours ago|

parent|

[-]

You are displaying ads on your website, there's a clear financial incentive.

by XCSme2 hours ago|

parent|

[-]

True, but the ad is to my own product, there are no advertisers, maybe there will be at some point, but if they were, that would barely cover the costs of testing the models, and likely never get a ROI on the hundreds of hours I've spent building it.

Would I want it to grow and make money at some point? Sure, why not, then I can test even more models at higher reasoning levels. Meanwhile it's just me testing models when they come out and publishing the results for anyone who finds them useful.

I don't see why posting some info and a link with my own findings, in a relevant discussion is considered spam. Should it be?

by WelkinFolk1 hours ago|

parent|

[-]

I believe it is pretty reasonable to show ads about your own product on your website, and shouldn't be considered spam.