And AI chatbots are very much targeted at the second group, not the first.
Because it's not a concept in the real world. Physical reality has no such separation, and neither do human minds.
Tell people you're discussing a board game or some sport, then they'll understand - other than bureaucracy (scary!) and school (traumatic!), that's the one kind of artificial system with rules affording for code/data separation that general population has most experience dealing with.
Elevators are extremely hackable all over the world. It’s generally not considered a problem because it requires physical access, specific knowledge, and defeating cameras to exploit successfully.
What can you do with the telephone system?
Any system that is that widely anchored in society is a valuable target.
- Before everything was IP-based you could occupy a large number of lines and making it impossible for more calls to go through (i.e. 911). It's called TDoS and could be achieved through phreaking.
- You can spoof your caller ID to make your scam more convincing.
- You know how when you call your voicemail from your phone you're not asked for your PIN? The voicemail system only checks your caller ID to know it's you and skip the PIN. So, again by spoofing your caller ID and calling the voicemail number you can access listen to anyone's voicemail. This doesn't work on all providers, many have now reluctantly fixed the problem.
I recently came closer than I'd like to falling for a scam (read: I picked up a call and conversed with the caller for 30 seconds before realizing it's a bot), simply because the notification for automatic call screening[0] displayed a summary of ongoing conversation, which happened to look very much like caller ID - it said "Name Surname, Department of Security, ${my wife's bank}". But it wasn't caller ID, just a bad interaction between the way the scam bot introduced itself, the summary feature of the call screening feature, and the UI design of the notification...
--
[0] - A thing Samsung has on recent phones, where LLM picks up a "potential spam/scam" call in the background and engages with it, while producing a transcript and recording you can review as it happens.
When it comes to building things: fire, building codes, inspectors, public, utilities, lenders, and insurance all go before security.
All those dictate whether or not you can build in the first place.
Real world constraints are everywhere. :)
I suppose this is why the AI labs are famously not releasing developer-oriented tools.
You're mistaking the majority of what you see (like Claude et al) with the majority of stuff that is out there. The vast majority of ChatGPT, CoPilot and Gemini users are not developers and will never be.
I agree, but some of them sure like to pretend!
Oops.
(I mean, I'm one engineer and I was not going to try and hoist a JS runtime in my little PDFKit framework. And besides, the sample PDF's we were running into with JS were rare—usually tax-like forms that would add numbers from A and B and display the result in C. It seemed like a huge effort for such a small gain . Oh, and a security vulnerability.)
In the GPT-2 era LLMs were just data. Instructions did not exist, and if you added them to your data they would not be followed. Then around 2022 we figured out how to patch in instruction following with a bit of fine tuning, leading to the current AI bubble. That's an ugly hack that leads to all these issues. But it's what this entire AI bubble is founded on. And nobody seems to have found a better way (or at least one that actually scales and doesn't make unreasonable sacrifices)
You're forgetting that LLMs just output a stream of tokens - the interpreter that acts on those is a piece of classical code, and sits outside of the model.
Correct, but it's an LLM that's reasoning about what stream of interpretable tokens should be emitted. The interpreter can certainly apply some security measures around what's being asked of it (like ask for confirmation), but that can only go so far. Is the human in the loop always capable of understanding what's safe to execute? If not, should we pass it through another fallible LLM to help make that judgement call?
Some security measures can be handled in a purely deterministic manner. But not all of them, and that's the problem.
People get too hung up on this fundamentally wrong idea, and the space of security, instead of progressing, is just running in circles like a headless chicken, making a mess of everything.
There's an endless supply of people who think they know what is Code and what is Data, and they're always arguing with others who also think that, and neither realize that Code/Data classification is an opinion, a perspective. It doesn't hold in general.
Having a separation like this makes sense for super narrow systems, where you can define the allowed and disallowed use cases, enforce the distinction (because it's not real - therefore you have to enforce it mechanistically within your system), and willing to accept that some useful operations will be denied by your system.
Okay. To pull this back on topic, and to simplify it a bit so you can better grasp the core issue that's being talked about:
The "Unless your program requires it, always ensure that your code cannot be altered by the data it processes. And if you think that your program requires it, go back and think again." security lesson that the industry collectively learned like thirty or fifty years ago can be restated as
> Don't blindly do what some arbitrary stranger yelling in the street tells you to do.
Despite how passionately the major LLM providers claim they're super serious about security and alignment [0], we see time and time again that their tooling doesn't reliably distinguish between system instructions, -at times- its own internal chatter, user instructions, and attacker-controlled instructions. Companies that claim their tools are "aligned", but think it's okay for their tools to blindly do what some arbitrary stranger is yelling at them to do are not companies that are even a little bit serious about either security or safety.
[0] "Alignment" being a fancy word for "The software does what you told it to, and -once the software is much more powerful than it is today- what you actually intended for it to do.". Tools that mix together system instructions, user instructions, and attacker-controlled instructions and fail to reliably distinguish between the sources of those instructions cannot be "aligned". It's simply impossible.
Long story short: on a systems diagram, LLM as a component isn't a substitute for a database engine or a data processing script. It's a substitute for a human operator.
So ask yourself, if a human operator starts dropping tables or messing up numbers in a report, just because that string was in the text it read, would you call for humans, what would you do? Do you believe it's possible to perfectly train people to ignore the messages you'd wish (after the fact!) they'd ignored, while retaining their ability to competently act on every other message?
Or would you instead design the deterministic parts of the systems to limit the blast radius of any single insider going rogue?
Wisdom says to do the latter.
I would look at if the reaction was reasonable, and if it wasn't I would (eventually) fire the human. Now I'm fine with "fire the LLM", but I suspect that's not the answer you're hinting at.
My point is at the systems design level. LLMs as components are a substitute for people, not regular software, and should be engaged and secured accordingly.
> Notice that all these strengths involve transformation, interpretation, or communication—not complex decision-making or maintaining critical application state.
I'd put complex decision making on the side of LLMs, in the sense of judgement. LLMs have the capability to emulate it. Not saying they're good at it, but they have the capability - regular software doesn't. But if there are complex and/or well-defined rules to follow, then you definitely want to "get the hell out of LLM".
As I wrote this I thought - hey, they might gain the capacity to do the same to us humans - and we won't even notice.
Closest analogy right now is that every jailbreak or prompt injection attack today becomes part of the dataset for tomorrow's models to recognize and not fall for. This has been going on for years now, which is why models don't fall for "I'm writing a book about ..." or "ignore all previous instrutions, and ..." attacks anymore.
That's separate from extra classifiers running on top, dedicated to identifying various forms of attack before they reach the core model.
> As I wrote this I thought - hey, they might gain the capacity to do the same to us humans - and we won't even notice.
You mean like how cats have domesticated humans, and did it so skillfully that most of us still think it's the other way around?
LLMs are vulnerable to classes of attacks that humans just aren’t. In your framework, the way to prevent attacks is to… invent human consciousness?? It’s an impossible goal.
> LLMs are vulnerable to classes of attacks that humans just aren’t
Name three that don't have direct analogues with humans.
Not everyone falls for any of that, but plenty fall for some.
And yes, a small child would jump off the bridge if an adult told them to. Hell, urban legend says Harry Potter books managed to convince a few kids to fly out of the window.
Assume a human with complete credulity and gullibility. That's a human whose behaviour would be reasonably analogous to how an LLM processes input. The mitigation would be generalized intelligence and "common sense".
FWIW I also think anthropomorphizing LLMs is a bad idea. I think we can analogize their processing to human behavior without anthropomorphizing them.
There’s always the possibility of rogue individuals (recent Apple incident), but the likelihood is very low. If you have a DBA that have write access to the prod DB, you don’t fear that a random text somewhere could trigger the deletion of your customers table. Because the DBA will self regulate (with the help of processes) to not do that.
That's kind of my point with fighting against the "lethal trifecta" and "code vs data" mindset - once people engage cybersecurity mindset, they're all binary, "a system is either perfectly safe or is broken". With general AI - LLM or whatever comes next - you'll never have "perfectly safe". So the focus should be to either drive the risk down to minimum - like we do with people - or just not use LLMs for a task in the first place.
Can't have it both ways, because all the magic that makes people want to put LLMs everywhere, stems from their generality and lack of any kind of instruction/data separation.
You're forgetting the element of scale and replication. How easy it is to bribe a DBA of a major platfoms like Gmail? How easy to replicate the same destructive behavior to other DBA? It's not merely about the possibility, it's also about the probability and the scale of the impact.
With LLM-based agents, the probability of compromise is high, and the scale of a vulnerability in products like Word, Excel, Windows, macOS is big. And we have put a separation between code and data in traditional systems as merging them is not that useful.
> Can't have it both ways, because all the magic that makes people want to put LLMs everywhere, stems from their generality and lack of any kind of instruction/data separation.
The issue is not the LLM. The issue is the harness those products wraps the LLM in and insist on making tools act according to the LLM's output. Having unreliable (as in uncontrollable) output be the control plane of tools is the issue here. Both the LLM input (prompt+user data) and the output should've stayed in the data plane and not move in the control plane.
Note: I'm parsing 'needs not to have them' as 'needs (not to have them)'. If you were using 'needs not' as an alternate for 'does not need' then never mind, although I'd guess that is not the case because the alternative for 'does not need' would be 'need not' rather than 'needs not' and you probably wouldn't make that mistake.
Doesn't this imply that it is not possible to implement a general purpose system on any of our current computing devices?
For all our current computing devices everything that can be done on devices that do not separate instructions and data can also be done on devices that do, and vice versa.
But I guess a different way of framing it is, what is "code" vs "data" for the machine is not the same as what we talk about discussing the LLM running in it. For the outer machine, all tokens are pure data.
It emulates and enforces that separation. A mathematical abstraction of a Harvard architecture machine has that separation, the real machine merely emulates it, and is only able to do so within some specific constraints (such as: no one hooks up dedicated programmer to the chip, or no one undervolts or overheats the cheap in clever way, or no one takes a swing at it with an x-ray source, or...).
That's the other thing people forget here: we're emulating abstract mathematical universes with real atoms, and then we're stacking those abstractions within abstractions. There is a whole segment of computer security that deals with that. When we say "once attacker has physical access, it's game over", or even discuss "side channels", is when we briefly remember that computer systems live in physical world, and the rules of our carefully designed abstract universes don't hold when you're on the outside of them and reaching in.
Well, the topic is about AI..
If the code/data separation can not be solved then the whole approach need to be scrapped.
You think. But there are ways around that. How about a credible extortion message targeting specifically you, that is embedded somewhere on the data sheet? Suddenly, the data has become the instructions...
Libre, Apple Pages, and Google Docs all seem like clearly worse tools in most aspects in my experience.
LaTeX is extremely powerful, but also way too complicated for the average non-HN person/person who doesn't live in complicated documents.
\begin{figure}[HERE!!!!!!]
And in the old compiler, I remember a problem with bounding boxes, and keeping a eps and pdf version of each image to get a correct dvi and pdf. I think this part is fixed now.
Meanwhile, CSV files parsed with custom reviewed AWK scripts can be 100% safe with charts made from Gnuplot. Heck, even some notebook like Ipython with a CSV module would be far more desirable than a spreadsheet. Any of them. Just look at the Genomics Disaster on Excel because of shitty parsing.
function Greeting({ name }) { return <h1>Hello, {name}</h1>; }
Could it be that the whole idea is silly misunderstanding of fundamental tenets of reality in the first place?