upvote
Pangram's marketing always reminds me of Anchorman's Sex Panther cologne: "They've done studies, you know. Sixty percent of the time, it works every time."

Pangram's "100% AI generated" claims are right 65% of the time. https://link.springer.com/article/10.1007/s40979-026-00226-w

reply
Did you read the paper you are linking to? It records a 0% false positive rate for evaluation of human-authored controls, and fewer than 5% of the hybrid and humanized papers had their AI levels overestimated by pangram. It is not completely clear from the data, but it seems that depending on whether the n=2 overestimates were “100% ai generated” assessments, the study you link says pangram’s 100% AI assessments were right anywhere from 95-100%
reply
Still, I guess if you would care you could let the LLM perform loops with pangram tests, changing the text till it scores well enough.
reply
My citation was correct. To question to ask yourself: For my use case, is it okay that Pangram can't reliably tell the difference between "100% AI generated" and "AI assisted"?
reply
deleted
reply
deleted
reply
It's depressingly hard to find one that isn't. You see the title, think this might be interesting, and puke by the second paragraph.
reply
I'm really open to feedback. I checked your past comments and you posted:

> AI is good at coding if there's an oracle. If the system is ancient, unreadable, untestable, that's exactly the opposite. It won't get the exact set of corner cases.

I sort of mention this in the article, so I'm sure we're somewhat aligned on the core. How would you have worded things?

reply
deleted
reply
First of all I want to say it is not your fault, and also your article is clearly partly human, despite what Pangram says.

The thing that I like the least is the headings and the way LLMs always try to put a punchy line. "Correctness time splits in two" says nothing if you don't know what it splits in. Maybe "making it correct vs. describing what's correct"?

Another trope is short sentences: "Good engineers used to say this before LLMs, and now nobody can argue about the sunken cost of having written that code." instead of the longer and redundant "Good engineers said this before LLMs. It was true then. It is enforceable now in a way it was not, because nobody can argue that writing more code was the hard part".

Another clearly AI paragraph is "Plumbing time collapsed. Scaffolding a service, generating tests, translating between frameworks, writing the first draft of a migration: all of this is fast now, and any timeline built on those costs deserves compression." Instead: "The time to bring up a proof of concept or refactor old code has compressed, and you should take that into account when planning your timeline".

There are videos on YouTube about AI style, you just need to learn them and undo them when they're the most blatant.

reply