I wonder why they didn’t throw a real chess engine in there for a baseline. There are engines where you can set the elo in the settings, so it should possible to see these LLMs relative to a human 1500 rather than just relative to each other.
reply> so it should possible to see these LLMs relative to a human 1500 rather than just relative to each other
replyAs a 1500 elo human I can tell you that a 1500 elo chess engine doesn't play like anything like a 1500 elo human.
This is true, but I'm not sure it matters? I was poking around at the lichess database recently and those elo calibrated bots are
remarkably well calibrated, their rating variance sticks out like a sore thumb compared to human players even at similar game volumes. So it should still be a decent predictor of how good a human at that level is, even if the playstyle seems alien.
replyI feel like every position is in the database so you could just lookup the most popular move for an arbitrary elo and that's the bot.
reply