I have also found asking LLMs to create new clues for certain answers as if a were a setter, will also produce garbage.
replyThey're stochastic parrots, cryptics require logical reasoning. Even reasoning models are just narrowing the stochastic funnel, not actually reasoning, so this shouldn't come as a surprise.
A lot of it is due to the tokenisation system though. It just doesn't represent individual letters in words. And at the moment reasoning models aren't trained to know how to spell all words, and to know when to spell them out to unlock other useful stochastic funnels.
replyI find the term "stochastic parrot" super reductive. Like yes, they technically are. But they are (I guess faking) reasoning and intelligence more and more usefully each iteration. Gemini Pro 2.5 compared to GPT 4o is a massive difference in utility.