undefined

points

[-]

Couple of methodological notes

* pre-chatGPT is not an effective control because language evolves. In the arxiv corpus in particular there are "fashions" in research depending on what gets funded lately, not to mention many new words and topics not invented before a given paper.

* In general, detecting AI from content seems difficult as humans write like they read. To the extent there are unique factors recognizable as AI and to the extent humans read them, they will eventually incorporate them into their writing style. Accordingly, you'd need to model a rolling window of "AI tells" that decay at some rate.

by phreeza20 hours ago|

prev|

[-]

This is a pretty stunning result. The time series looks really convincing. Is the way the detector itself is trained orthogonal to this or could there be some "leakage" in that the pre-chatgpt text is in the (positive) training data?

by dopamine_daddy19 hours ago|

parent|

[-]

I tried my best to avoid leakage. If you're curious about how I trained the detector I have a writeup on it: https://unslop.run/blog/how-our-ai-text-detector-works

FYI this is all relatively new so there might be lots of issues and iterations coming.

by Paracompact18 hours ago|

parent|

[-]

My honest first impressions, since I think the project is well intended: This writeup is itself AI, and I would venture to call it slop. The Calibration section is borderline uninterpretable, and I challenge any non-author who claims to understand it to answer some basic peer review questions about it.

by simonreiff16 hours ago|

parent|

[-]

I have to concur, unfortunately. This isn't reliable, reproducible, or interpretable.

by paxys18 hours ago|

prev|

[-]

What detector are you using? How can you be sure of its accuracy given that every commercial AI detector has been debunked?

by JamesBarney18 hours ago|

parent|

[-]

Almost every. Pangram is pretty accurate on longer texts. I haven't seen any glaring examples of false positives.

by malshe10 hours ago|

parent|

[-]

I’ve heard good things about Pangram but it’s trivially easy to fool it. I generated text using Opus 4.8 and then humanized it using Grammarly. Pangram determined it 100% human-written. Tried it numerous times with the same result. GPTZero claimed with 80% confidence that it was human-written but AI polished. Nothing in that text was human-written.

by adamgordonbell12 hours ago|

parent|

prev|

[-]

This! Pangram is very good. They claim 1 in 10,000 fp rate.

I had to change my mind on AI detectors after playing around with it.

It would be interesting to hear how this detector compares. It also seems to be aiming for low fp rate.

by 0x000xca0xfe17 hours ago|

prev|

[-]

Could you share some pre LLM false positives? Would be interesting to see what is tripping the detection. Did a tiny fraction of authors write like LLMs, before LLMs?

by jmcqk618 hours ago|

prev|

[-]

Extend it to scan papers from pre-2020. That should give you a better baseline accuracy for your detection system.

by cgio10 hours ago|

prev|

[-]

A signal, I put an AI generated of mine (with heavy human guidance on aesthetics mostly and some minor human editing) and got 5% only.

by 18 hours ago|

prev|

[-]

deleted

by andycasey19 hours ago|

prev|

[-]

this is neat! is the model available somewhere for local execution? or even a lookup table with your results for all arXiv pre-print codes? I want to run it on lots of pre-prints and I don't want to kill your server

by dopamine_daddy19 hours ago|

parent|

[-]

Thank you, I plan to release the arxiv preprint codes. Also don't worry about killing my server, let me know if you succeed :D

by exe3419 hours ago|

parent|

prev|

[-]

Wouldn't that provide an excellent signal to tune models to be less like AI? I suppose that's a good thing ultimately.

by economistbob20 hours ago|

prev|

[-]

Thank you for this work.