Also funny how people (including LLM vendors, like Cursor) think that rules in a system prompt (or custom rules) are real safety measures.
replyThat's why there's tomes of overlapping AGENTS.slop folders and 100K lines of "docslop" and people inventing "memoryslop" systems to reduce this token burden. But the agents can't really distill even a simple instruction like "don't delete prod" because those three words (who knows how many tokens) are the simplest that that expression can get and the ai needs to "reread" that and every other instruction to "proceed according to the instructions". It never learns anything or gets into good habits. It's very clear from these kinds of threads that concepts of "don't" and "do" are not breaking through to the actions the bot performs. It can't connect its own output or its effects with its model context.
replyLike we say in adventure motorcycling: "It's never the stuff that goes right that makes the best stories." :)
replySure, but do junior devs find another key, in an unrelated file and use that key instead of their own? Maybe once you read about someone doing this and maybe it happened or maybe someone was being overly "creative" for entertainment purposes. But it probably doesn't happen in practice. The LLM making this mistake is becoming more and more frequent.
replyIt's also possible it's only a compelling story, and not based on any real events.
replyYeah people don’t understand that if you put an LLM in a position where it’s plausible that a human might drop the DB, it very well might do that since it’s a likely next step. Ahahaha
replyThis is exactly what I have in mind when something like this happens. Sometines it generates a story you want, sometimes not
reply