1. AI generated text works off of whatever model it came from, whatever prompt was used, and probabilities. It’s been described as a “next token predictor” and that, for the purposes of this answer, is how I’m describing it as well. First problem: it’s only as good as the above.
2. Because of the above, its accuracy relies on probabilities. It can confidently give me wrong information, and as someone who would read this book to learn a topic I don't already know, I have no frame of reference to know whether I’m being fed accurate information.
3. Since it greatly speeds up the time it takes to physically write material, there is a possibility the author didn’t closely vet the information provided by the LLM as a next token predictor. That means it could be wrong in subtle ways, and the only way for me to know is to rely on knowledge I don’t have.
4. Had AI not been used, I could conceivably go on the past work of the author, rely on the person who has the expertise who wrote the forward, or the fact that physically writing a book takes prohibitively long enough that the dopamine from having written a book doesn’t outweigh the amount of time it takes, and if you are going to take months to years to write a book, you’re going to not want that effort to be in vain. You’re going to do your diligence and not just “wing it”.
AI makes it relatively trivial to appear knowledgeable and to get that dopamine rush from saying “I’m an author” and appearing to be an expert in a field without being an expert in a field. For some, that temptation is too great and there’s no way to know without the passage of time. Time for folks to review the book who know the subject matter. Time for reviews. Time for folks to put signs up that say “don’t trust this material.” In the meantime we still have that content out there, further infecting new AI models and people into believing it’s accurate.
That’s why the use of AI is dangerous, and is especially dangeorus without disclosure.
Well guided is a qualifier that matters.
I have observed that models are not able to effectively come down to a human level. They are mediocre at approximating what a first-time reader would want to know. If you will forgive the anthropomorphization, they are not very empathetic. In this book for example, the author pauses the flow of thought with countless little anecdotes, anticipating where the reader might have a question or want to know more.
Of course, a well-guided AI is capable of anything a human is capable of. But is that not tautological?
As well, if people were a bit less anti-AI then maybe we can get disclaimers that things are written by AI that don't immediately torpedo any value the writing has.
Tangent: I use AI for code, but I've had to ban it from writing readmes and doc comments; the AI models I've used produce text I'm unable to parse, and assume my readers would not be able to either.
I don't understand why HN cannot ever say the first sentence without the second one.
Most people agree that using AI for research or finding better words is fine, but then can't upvote or have a meaningful conversation because of the performative politics and wild delusions about a scifi future.
Dude, all we have with LLMs is an NLP search engine. Get over it. It's been around half a decade of this nonsense. There is no "step change". There is no "hockey stick" curve. The models have not "dramatically improved compared to last year" regardless of what year it currently is. There is no big bad overlord coming for you. Nobody is getting replaced by AI. Even frontier AI can't replace local AI. It's just the logical next step towards better search that assumes you want the "I'm feeling lucky" button a little too often.