Human writers also use specific constructions for definite (though not always consciously known) reasons, and mapping those correlations onto writing models is only getting done on accident. This is a big part of why LLMs doing pointless negations (it's not X, it's Y) is so jarring: people use that construction in order to preempt an expected misconception, whereas LLMs try to use linear algebra over the learning corpus to generate it in appropriate quantities and places according to RLHF/fine-tuning/etc.

In other words, it isn't just verbiage that has a "preferred" quantity, but also higher-level language features, and the appropriate contexts for those features feel like the sort of thing that will be extraordinarily difficult to replicate without a theory of mind.

edit after coming back and re-reading: the "in other words," and rephrase is because I'm trying to explain something complex enough that I'm not even sure I fully understand it, so I try typing out multiple explanations and leave the ones that make the most sense to me. This might be idiosyncratic to my writing and not a significant feature of training corpuses, but even if it wasn't it's a second example of the sort of thing I'd suspect LLMs to be bad at using appropriately.

I vaguely remember a study that I was able to find when fact-checking this comment before publishing it [1], which found that music needs periods of high predictability to nurture listener expectations for the sake of later subverting those expectations, and the most enjoyable parts of music are those moments where expectations are subverted. This sounds like a similar effect.

1: PDF warning https://www.cell.com/current-biology/pdfExtended/S0960-9822(...

"A model trained to minimize statistical prediction errors is going to sound ... predictable (aka boring)." -- what if the goal is to be a (good) human author.

In that case, model will in fact strive to keep their writing as close to interesting writing as possible.

Gemini models are so, so much better in this regard. Their outputs are actually readable.

That doesn't follow. Predictability is your brain reacting to the lack of novelty. What an LLM produces is a chain of words following some kind of conditional distribution, whose output could vary significantly from your understanding of novelty depending on its training and priors.

With a single, generalized prompt ("produce something unpredictable"), ChatGPT responded with:

> 00:— / ANOMALOUS RESIDUE > Vekh. > Ìssara. > Nnnn—tq—èl. > > A temporary thickness enters the meat of the signal. > > Not rain. Not a substance with permission to fall. > > UL / 9 / ferrous / ∂ > The measurements are eating each other. > > Someone has left an entire childhood in the wrong unit of pressure.

That is not predictable to me, at least. It could be boring, but I think that is mainly a product of me knowing where the text is from. LLM-generated text is inherently boring, to me, but that has nothing to do with predictability.

Interesting, I was thinking the walls of text are better for the watermarks.