undefined

points

[-]

A better benchmark might be asking the LLMs to write GO ai and then comparing that-- the issue is that there will be a HUGE difference in performance that depends purely on this game being in the LLM's training... but training a general LLM to directly play these games would be a waste of capacity and shouldn't be encouraged for benchmaxxing sake.

Programming an engine OTOH is a skill that is more general and they should all have.

Might be useful to have the target of the engine be some specific virtual machine that gets a strict cycle budget-- e.g. execution runs so many cycles, and result is read out of a specific memory address at the end (or when it terminates early).

by theendisney9 hours ago|

parent|

[-]

This is a great idea. It could learn from its mistakes, repeat things that work, abstract complex situations. I would even want humans monitoring the project.

by ForHackernews11 hours ago|

parent|

prev|

[-]

Excuse me? Is this thing supposed to be a borderline SGI or not? We already know LLMs are good at spitting out code.

by nullc11 hours ago|

parent|

[-]

Playing these games autoregressively isn't even the right way to use the LLM for this task (unless it was trained to do so...). It's somewhat like having a creative writing bechmark but requiring that all the input/output be base64 encoded. It can do it-- but no guarantees on the results!

And it's also just bencmaxxing bait: you can get a huge improvement on the task by RLing on it, but make no improvement on anything else. Doing so would just waste model capacity.

If you could tell that every LLM was equally not being exposed to the task then you could justify it as a test of abstract reasoning, but you can't. So it ends up on how much go transcripts ended up in the training, which is ... not a very interesting metric.

by monster_truck4 hours ago|

prev|

[-]

imho doing it on 9x9s kind of misses the whole point, anyone who has gotten to single digit kyu vs humans will understand when I say its effectively tic tac toe with extra steps

by Shorel1 hours ago|

parent|

[-]

It's a very fun tic tac toe!

(I have not reached single digit kyu)