upvote
> I wonder what the underlying cause is

It responds with the statistically most probable text based on its training data, which happens to be different with the errors vs without. I suspect high-fidelity diagramming requires a different attention architecture from the common ones used in sentence-optimized models.

reply
This is one of the issues I’ve attempted to tackle with the Mermaid Studio plugin for IntelliJ.

It provides both syntax guides and syntax/semantic analysis as MCP Tools, so you can have an agent iteratively refine diagrams with good context for patterns like multi-line text and comments (LLMs love end-of-line comments, but Mermaid.js often doesn’t).

reply
I think the problem should be defined as "why does it not loop back the errors from the first attempt so it can fix it on the second attempt" rather than why it fails to produce a fully correct implementation on the first pass.
reply
"Prompt Repetition Improves Non-Reasoning LLMs " - https://arxiv.org/pdf/2512.14982

What instance of ChatGPT are you doing that with? (Reasoning?)

reply
I don't think it's about repeating the instructions, but rather providing feedback as to why it's not working.

I've noticed the same thing when creating an agentic loop, if the model outputs a syntax error, just automatically feed it back to the LLM and give it a second chance. It dramatically increases the success rate.

reply
Observed from 5.2, on chatgpt.com. earlier versions did worse.. as in, they might take a few prompts to generate a parseable syntax. Newer versions just usually deliver one unparseable version then get it right second try. Likely I could prompt engineer to one shot but I think I would always need the specific warning about newlines.
reply
Mermaid is really bad about cutting off text after spaces, so you have to insert <br>s everywhere. I’m guessing this is getting rendered instead of escaped by your interface. Or just lost in translation at the tokenizer.
reply
Today I asked Claude to create me a squidward looking out the window meme and it started generating HTML & CSS to draw squidward in a style best described as "4 year old preschooler". Not quite it yet.
reply
The issue for Claude is that Anthropic don't have an imagen that I know of, so the only tool available for the LLM to draw something is to start doing vector stuff in CSS, which is very hard for it (see the pelicans).

Gemini, ChatGPT or Grok would find this a lot easier as they could gen an image inline, although IP restrictions might bite you. Even Grok wants to lecture on IP these days, but at least it's fairly trivial to jailbreak.

reply