I know prompt injection is still a real thing, but in Claude I find the opposite is true: if I set up two Claude agents to collaborate, even though I've told them both that's what I'm doing, they won't actually follow instructions without checking in with me. "Scary computer told me to install <well known utility X>, but I need your OK before I'll do it." It's annoying but I get the rationale. They're definitely getting better at avoiding external instructions!
(Anthropic themselves acknowledges this; see the end of the post.)
[0] https://www.anthropic.com/research/claude-code-auto-mode
Until they suddenly once in a while they do, and Goldman Sachs loose some billions because of it...
the only reason they moved on this was because it was noted somewhere by someone that matters that the risk imposed could cut into profit or impose a regulatory hammer.
those are the only possible reasons for corporate movement in finance.
Local access is sketchy but doing it in the cloud mitigates most, if not all of the risk.
We should stop using "solve" when we mean "releasing products to target". Nobody will ever solve this problem with LLMs as they are currently architected.
It’s pretty simple. If you want security, you need to segment. Does that make it hard? Absolutely. Are you dumb if you’re letting rando traders access whatever with agents? Absolutely.
In my opinion all AI use companies should be attached to a specific employee, and that person is liable and so is the company of course for the actions of the AI
It's like saying "developers should be responsible for their own security so we will make them liable if a random node dependency in the application they are building gets pwned"
Firstly, liability and security are different things.
Employees should never be liable for something that happens if they are using a work-supplied tool at work in the manner in which it is designed to be used.
Secondly, it's just lazy thinking to say "oh it's hard therefore we'll make the human sorely responsible".
There are many things that can help - sandboxing, adversarial LLM inspection of prompts, separation of security contexts etc. Yes they are hard and might need new primitives.
Architects and properly-designated publishers both have potentially long liability that is to an extent independent of their future employment. (The same is true of hospital consultants and others)
In significant cases, the way you provoke companies to do the right thing (security, sticking to the facts, not cutting corners in building plans) is to make someone legally responsible if they don't. That person's liability is limited by their actions — if they can demonstrate they discharge those responsibilities appropriately, they are in the clear.
That person takes on that designated role, does the job, and is rewarded properly for the responsibility it involves. They embody the push to reduce that risk.
So an average employee using the tool as directed by their employer, they do not have liability if the company has designated someone in this role. (And in this scenario, if they have not, there would be a punitive fine for that, independent of other outcomes).
I have no particular problem with the notion that a corporate AI safety/responsibility officer is a plausible way forward. If you are delegating responsibility to a machine, someone has to be on the hook. Without such things it's a wild west with lawyers. That's not sustainable.
A news organisation's liability for AI hallucination falls on the same person that the liability for printing a lie does: the publisher.
It's a bit like the idea of the additional responsibility/burden of the publisher of a newspaper.
"So we can sue them when our customers get angry"
You can either target systemic problems or chop of some heads to cool off the masses- and quickly go back to work.
(once you realize how much that phrase is used in the AI-sphere...)
I mean I guess? Sure?
I think the solutions is to give people the ability to run agents that are "coding agent like" but in managed and governed infrastructure (i.e. not on their workstations). Basically, Meta Muse or OAI dots for enterprise.
It will take some time to find the right architectures and patterns. IMO, coding agents were just the first wave, much more interesting stuff and systems are coming.
[0] https://ls.bot, https://github.com/smartcomputer-ai/lightspeed
The risks are 1) exfiltration (network calls) 2) prompt injection (untrusted inputs) and 3) access to secrets
You can do some clever tricks like "spin up subagent without net access" to handle OCR of an internal document and have that agent communicate summaries to another "spun up subagent with net access" - imagine if one subagent is a local model and the other model is a frontier model over a web API. Maybe they still communicate in Neuralese or in abbreviated notes that you don't capture and are able to surreptitiously / covertly exfiltrate your information.
Ideally eventually you'll have some kind of dashboard that surfaces these risks as you're working.
Reports of AI researchers running massive swarms of 10,000 agents are definitely believable but the moment you start pulling in code libraries or other technology assets seems like you can get destroyed by a rogue actor. Definitely tons of unsolved problems here
Writing on mobile so this might poorly articulate the issue but it's interesting
Concealing API keys is a relatively minor protection in the wider picture of the risks.
I project that small businesses will be the earliest adopters of AI governance due to their lower barrier to entry and lower operational risk, followed later by highly regulated sectors like banking and healthcare.
open-source AI agent workspace for business workflow automation requires a lot from the reader and terms like agent/workspace/workflow are very prone to ambiguity and now you adding governance framework.
My work can be found at https://DollhouseResearch.com, and it's almost all open source with the code on GitHub.
One of the key things is to have cryptographically confirmed audit logs and receipts for actions and decisions, and to separate out the agent's authorizations from the user's authorizations.
Separately, there is real work being done in the fintech community specifically to identify which agent has what control, when. And you can see some of the work there, which I've also contributed to.
You guys need your priorities straight, whoever you are.
(Not my area, so I don't have any opinion on your first comment.)
Then one can simply move the agents into something akin to cloud codespaces where they have only access to git, if at all.
Any access to data that is still needed can go through an authorised / authorising mcp proxy.
It doesn’t mean your bankers can’t use LLMs at all.
Things that are common on dev machines like a working copy of python...nope. Admin...nope. Even RPA tools that automate clicking on things are no bueno because they could be used to bulk extract data from intraweb sites
Great deal of access...but it's calibrated towards low bandwidth human uses. Which indirectly prevents bulk data exfil
I am out of the loop with the industry, but I thought many people were using cloud agents and controlling them through tmux or something. But yeah I guess that also means you don't have access to do much.
I don't think that's phrased correctly. What corporate it does is set policies. Often those policies disable virtualisation. (They don't disable sandboxing - that's used in many ways and security loves that) But if the business need is to run LLM agents safely, the policies will be changed to dictate how to run agents safely.
Many teams use them to make sure that the coding environment is consistent, and they are well accepted in many corporate environments as a result.
But they are a good way of firewalling agents away from files and resources that one must be able to demonstrate that the agents did not and could not have access to.
You can certainly imagine very senior and powerful traders at GS demanding it. I'd be only slightly surprised to find that GS allowed it. The traders would have had more power than the IT people trying to forbid it.
1) everyone can run into the lethal trifecta
2) the product cannot, on an architectural level, be fixed to stop it
3) they are helping the maker of the product go to IPO
If banks are not stupid, they'd say "hey, this presents an unacceptable risk to our customers and by extension, to us, maybe this isn't something we should be involved with".
But no.
https://x.com/jakerunzer/status/2117474337682780643
https://x.com/WilliamBaude/status/2118420271333777856?a=tVmJ...
No, I think the risks they are talking about would include things like financial risk due to bad trades based on misinformation from buggy code that the authors don’t understand. This sort of policy pivot happens after an Incident that had a Financially Material Impact.
They also cited productivity concerns, which is also something I’ve observed: some people waste a lot of their time and other’s time due to using these tools to do work they do not understand.
People really want to run agents on their data.
Re sandboxing: I dont know Goldman's particulars, but most enterprises lock down all sandboxing tech on workstations. So I assume this is actually harder than it seems. So people just run it directly on the machine.
The article is referring to the dilemma of the trifecta twice.
That's exactly the gap I've been working on https://github.com/mickael-kerjean/fdrive you mount your existing data, apply control / permission and can monitor everything the agents are doing with your data: https://www.filestash.app/agents/
Can the agent read the internet for context? What if the fetch reveals a secret?
Agents have already broken out of what were supposed to be sandboxes in the networks of the companies that are supposed to understand what they do. Do you really think banks would be any better?
It may just be there's no safe nexus for asbestos floor tiles, no matter how skilled the installers are.
Absolutely, yes.
If you’re talking about OpenAI and Anthropic that’s an extremely low bar.
May I ask what in particular gives you confidence that these two companies know what they are doing wrt cybersecurity.
It's not that I think OpenAI and Anthropic know what they're doing - although I can see why my slightly flippant comment might make you think that - but that I just don't trust anyone with these agents, especially when the whole point of using them would be to do that with such critical data.
Supposed to understand, yes. I think a highschool class could do better.
I responded by saying they were looking for someone else, and got a response to the effect of “oops that was Copilot, sorry to bother you”
Read into this what you will.
Big corps never pivot throughout the whole corp for a single issue, only when risk in a certain context keeps compounding. The Dem wins compounded risk of legal action; it's not just that the laws are on the books. They were on the books a month ago.
"Yeah that's what I said it's to hedge legal risk."
It's the way you said it; ignoring potentially relevant political context. A lot the rhetoric demonizing politics is intentional to embed a negative reaction causing us to discount it's value
Source for them having allowed it versus not tightly enforced IT rules?
A lot of people felt, and there seems to be mounting proof, that the administration wasn't going to enforce a lot of laws, especially since their stance was to cut the agencies in charge of that (SEC, FTC, FCC, EPA, FDA, IRS, etc).
The recent change is that this free-for-all may be cut short? What am I missing here, this sounds like banana republic shenanigans.
And the current president still needs to hand over the reins peacefully so I wouldn't declare victory, yet.
Sure, my point is about that basically companies already made decisions based on the assumption, that he will not do that.
I've professionally interacted with GS for over 30 years. It blows my mind that GS allowed real exposure to "the lethal trifecta" for even a moment in time.