Guarding PII is an extra, I guess, and if you want to use a giant cloud-hosted NN to control software on your own device, there are very few default restrictions. And no matter what restrictions there are, when they go beyond prompting, they would have their own ShowHN.
For 99% of users, the model knows it's in the /home/joesmith/whatever repo and `git log` shows the last change was by `joe.smith@example.com` - trying to keep the user's identity secret from the model would be nigh-impossible so I think anthropic gave up on even trying.
Then, your agent invents a User-Agent and it sometimes invents one with your email in it.
It doesn't even require prompt injection or a failed request, it might be in the agent's pretraining knowledge that a given website requires a user-agent.
It's just doing what it's supposed to do, analyzing the statistical patterns of a natural language prompt and matching it to a likely response.
What's ridiculous is an assumption that it "it should never do something so ridiculous" as if an LLM is a person who understands what that word means in any context other than next-token prediction.
Where Claude Code has supposed boundaries between deterministic and non-deterministic tasks and flags.. generally.. it's unclear how much privacy has been a part of that and what implication that could have.
Hope that clarifies.
GitHub offers noreply email addresses
Wish that were itself behind a permission wall, but I guess everyone just uses auto mode anyway so perhaps that ship has sailed.
Common problem: you might consider it trivial to define a rule that forbids access to any file path outside the working directory. Allowing "safe" shell access is already a heuristic, there is no way around it:
- shell scripting is Turing-complete
- there are platform-specific paths that need to be at least readable to perform trivial tasks: Everything is a File!
- security models to make such low-level file access transparent exist, and generally work, but: they are never perfect (container environments, chroot etc). Because...
If someone else distributes my commits (or messages), I have no control over that, not any different from sending a physical letter?
I was saying "cloud-hosted" explicitly, and was speaking about what it means to have a remote control software?
Doesn't even matter much whether it's an Artificial Neural Network for the sake of my argument.
I never give a provider my actual email address, I recommend others do the same. Otherwise, it’s just a matter of time until it is leaked or otherwise part of a data breach.
“lethal trifecta”
https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
(a component of my cybersecurity work is red teaming inference, agents, and mcp interfaces)
This is why alignment is important and highly non-trivial
FYI, it's generally bad form to "haha" your way through privacy violations.
Leaking private information is a very big deal for a lot of people.
Anthropic has an engineering culture problem that's going to shoot it in the foot if it doesn't learn to take these kinds of issues seriously.
This isn't the first example of the kind of work they're shipping in Claude Code or in how flippantly they engage with the community.
I disagree, we can likely never fully (or should?) trust models to do these things deterministically, that's why we have harnesses around the agent with built-in constraints.
It is actually in the name: https://en.wikipedia.org/wiki/Harness, https://en.wikipedia.org/wiki/Safety_harness
This is an official Anthropic response???