"Quirky and obscure" has the functional benefit of ensuring the source question is not in the training data/outside the median user prompt, and therefore making the model less likely to cheat.
replyWe have enough people complaining about Simon Willison's pelican test.
When you program, do you consider using your prior knowledge of programming cheating?
replyWhat would make the prompt a better actual evaluation in your judgement?
replyNot focusing on pokemon for a start. Maybe use something more people can recognize and evaluate. I have zero knowledge of pokemon, I see it as a niche thing for ultra-nerdy people, and not something everyone is familiar with. Nothing about that test can be evaluated by anyone but a pokemon expert. Sorry, but pokemon isn't as mainstream as some people might think it is.
replystill #opentowork huh
replyWhere does one even use that hashtag?
replyIt's a LinkedIn joke.
replyAh yes, also known as C++ enjoyers.
reply