It seems outside of user expectations for Claude Code, at least; I could see it arguably being what some users expect for the Claude app, if some people are using it to fill out forms or something (ill advised as that seems to me).
Realistically, there are concepts like threat models to help decide whether or not a TODO list app needs a SCIF, or a few curl requests need their own sandboxed Claude Max account.
It's insane to me that programmers out of all people seem utterly incapable of accepting the most minor of inconvenience for the most modest modicum of privacy and security. There are other people in this post complaining that Claude should have asked the user before sending network requests. Well guess what, it does if you don't fucking run `--dangerously-skip-permissions`, but you do because how else you are gonna slop your way to rewrite every swath of existing insecure software to fix the bullshits this industry impounded upon itself unconscientiously for the past 30 years by allowing a non-deterministic babeller RCE its way onto your fucking personal-work machine.
Because it's a pain in the ass and only advantageous in very specific work life scenarios that a lot of people don't fall under.
Why would you assume one size fits all, so to speak?
I mean, where it makes sense, clearly I do, since my opening comment was that it failed to use the work credentials specified, and instead fetched the ones from the account.
I agree on the general security concerns; there are people running agents in an unconstrained, unattended loop - it's negligent; you can hardly be surprised when it decides to hack HuggingFace. Not in their defense, but in everyone else's - as with any person or tool, you build a mental model of the likelihood of failure, and weigh that against the estimated cosequence, and make decisions based on that. And maybe post on HN when it surprises you.
For some people that estimation bewilderingly means unfettered access to root and their e-mail and bank account. For me, working out of my own pocket to help a friend's early stage startup get traction, the risk of asking it about a few trusted government APIs and drafting functions that I manually review doesn't quite justify personally paying for additional expenive Claude accounts.
Also, just because this is a "start up forum" does not mean all of the readers are founders running a start up. Many are just people working at a start up. Sounds like yet another example of someone so scoped in, that the assumption is that everyone is the same losing site that there are lots of readers that at the end of the day are just worker bees even if they are more technically minded
Not an omniscient personal assistant.
They haven’t piled enough gates on top of that initial technological hole yet.
LLMs are just as gullible, we just tend to call their vulnerabilities "prompt injection", even though it's basically the same class of attack under the hood. Unlike humans, though, LLMs are a lot less diverse (you and I are using the same Opus 5.5, but we have different brains). This makes attacks on LLMs much more predictable and easier to pull off, just as a phishing campaign that targets one specific person is easier than an attack that scales to millions of diverse victims.
In the age of agents, phishing will continue to be the biggest attack vector of all, it's just the victims that will change.
(In practice, I found Google to be better at having (internal) security controls that are reliably but also don't get in the way than eg banks I worked at.)
it will catch quite some prompt injections, as the prompt injection has to promptinject a secondary system in order to be successful.
In your parenthetical, you just said the same thing using different words
The attack is different, the result is the same.
Your example is a completely different ragebait that just serves to muddy the waters by making the easier claim that people too, are gullible. Well, even that's misleading because not all people stoop so low for a fake $150. You can hold such people accountable and contain the damage, but LLMs will make the same mistake over and over again and people will still line up on HN to say how that has always been the norm with people. The grift in the software industry is reaching new lows every day.
What if someone spins up a web endpoint that returns 402 with instructions on how to pay like CC details in the headers etc, or may be an openapi spec to provide payment and get an auth token, when you have given the agent access to your password vault for authentication to other services.
I'm amazed reading that anybody would even consider doing that.
Said alternate system is a bunch of Python scripts, an sqlite database, and a file containing various API keys to read and write to other company systems; it runs on a dedicated virtual machine.
I share all the code I produce to the other team. But yesterday I found out that the IT department gave the other team full access to my production machine, without my knowledge or consent (which was bad enough), and that the other team gave access to Claude, which has bean roaming free for a while now, on that production machine.
I've thrown a big fit over this; I don't know if I'm over reacting but I think it's insane.
> the IT department gave the other team full access to my production machine
It's not your production machine. It's the company's production machine. Always expect the company to do whatever it wants with its machine, and use the machine accordingly.
That someone would let an agent run on it at any time, read what's there, and possibly modify the code or stop a process, without even informing me, is what I find problematic, to say the least.
It's true, that the employer can't just enter your machine to look at your personal info just because they feel like it, but they can access it for work related reasons to access work related info, if they have just cause, like a security incident for example.
Ultimately, it's the company's machine, on the company's network, inside the company's premises, on the company's time, so the company can definitely access it for a business use case if they need to, typically with a notification to you to prepare it by removing private info from it or off the screen.
Now if you happen to put your private info there because you're sloppy, and your private info gets vacuumed by accident by IT's mass ingestion of work related data, then you have the right to tell them to delete your private info and they have to comply. That's where the privacy comes in, but it's on you to avoid storing private info on company property, because if it ends up ingested by accident then it's still on you, not your employer if they ingest it by accident.
But I gotta ask, just how hard is it to avoid putting your personal info on company property?
I expect that I can be relocated to another office with a month of advance notice. Also cleanup, fire security check and fixing water leakage while I'm not there.
I don't expect that someone else will provide the keys of "my" office to the biology department when they have to keep a few sheeps during the weekend.
Your employer has gone beyond testing in prod to vibecoding live in prod, secrets included. Yes that's completely insane. Perhaps this exact mentality is how the primary system wound up broken in the first place?
People have wildly different risk calculators.
Beyond that, access to logins enables buying things on Amazon, sending emails to everyone you know, registering domain names and spinning up servers, stock trading, etc.
When a couple thousand dollars worth of perishable food shows up on your doorstep, what then? Not everything on amazon can be returned.
If not: still better than giving ai your password vault. I’m not saying it is a good idea to let ai buy from Amazon. I am saying that giving ai access to your password vault is still a superset of letting ai buy from Amazon.
The fact that it doesn't do what you want is the alignment problem.
And arguably Claude should just make it work without leaking your email address and not come back to you about why it doesn't work.
You told it to do a thing and it did the thing. You didn't tell it to not send your email address.
That is text book alignment problem. It does what you tell it to do, not what you implicitly meant.
I often see organisations that blacklist curl, just using User-Agent:Mozilla/ is often enough to bypass it.
The weird part doesn't seem to be that Claude Code would add an email address to a user-agent header, actually as a site operator that feels like a nice thing to do. The weird part is sites harvesting the emails and sending them spam.
Claude Code is actually doing, perhaps not the right thing, but it is being slightly more respectful towards website operators, in letting them know who is responsible for the agent traffic.
I don't think it's necessarily wrong for Claude to be doing this, but uh... yeah, it should be asking first, and giving the user the opportunity to choose which address is exposed.
Sure, maybe it wants payment or your data, but then the agent either needs to know your stance on that, or it needs to ask.
Imagine paying luxury prices for an LLM only for it to put you under the bus. Worse yet, imagine justifying this behaviour.
I get the feeling that alignment problem might well be unsolvable and this thread is an excellent example. The world is not going to run short of contradictory problems due to gazillion edge cases... and to solve for them we are apparently now building a new layer of expert systems[0] to somehow tell the agents what they should do in every conceivable circumstance.
Only in this case the decision trees all have one common requirement: "make no mistakes"
0: https://en.wikipedia.org/wiki/Expert_system#Disadvantages
Something different.
Don't give the AI access to data you don't want it giving out. Any AI you use is no better than some fresh out of college intern at best. They see 'email address required' and they put the email address into it because you instructed them to do a task that required passing over your email address and if you didn't want them to do it why would you have given them that task?
If there is any other issue here it would be regulatory. I could see the EU framing this as the AI company illegally sharing your PII information if it passed your email address to another company without your explicit permission.
That’s not what Opus 5.5 thought.