undefined

points

[-]

The overwhelming majority of web app security issues, even the really complicated and sexy sounding ones, at their core are just “script kiddie style hacking”. Injection attacks and RCEs like this Jinja injection may seem “script kiddie” but they are a dime-a-dozen.

It is folly to assume that most companies don’t have similar weaknesses somewhere in their stacks. In the past, they were just sufficiently buried that they would get noticed and fixed before an attacker found them. With AI able to probe at thousands of these potential weaknesses per second… that’s not so true anymore.

by spiresofagartha2 hours ago|

parent|

[-]

This hasn't been my experience both searching and reading vulnerabilities. Hackerone is public for some companies and vulnerabilities and you can see how contrived exploits can get. Even before AI, Burp Suite Pro was enough to get the so-called script kiddie style hacking.

by nickpsecurity2 hours ago|

parent|

prev|

[-]

The commenter may nit be assuming that. Instead, they may be saying rich companies should implement proven, security methods that block script kiddie-style attacks. HughingFace apparently isn't doing that.

That's how I read the comment.

by cmiles81 hours ago|

parent|

[-]

Correct. Basic security practices would have stopped most of this. Thus what is most insightful here is less that the attack was sophisticated but more that Hugging Face was running a setup that, by standards that are the norm elsewhere, would appear amateurish.

by pyth01 hours ago|

parent|

[-]

You might be able to say that HF made some amateurish mistakes, but this attack certainly was fairly sophisticated. It relied a number of separate exploits or vulnerabilities that it combined and used over the course of multiple days. Calling it otherwise seems to be downplaying it.

by 2 hours ago|

parent|

prev|

[-]

deleted

by 0xDEAFBEAD4 hours ago|

prev|

[-]

Didn't the model initially obtain internet access by discovering a zero-day vulnerability?

In any case, I would guess that a lot of unicorn startups like HuggingFace could be hacked by a sufficiently determined script kiddie working at 100x speed. The practical implications of a coming AI hacking wave could be large, even if agents are just doing grunt work really fast. Most organizations suck at security.

Seems to me that the most likely scenario is: Black hats are currently tuning the recent Kimi release for this type of work, and we'll see a flood of similar attacks within the next few months. (Why would this not happen?) Note that regulation is useless here, because black hats don't give a crap about regulators!

by mr_mitm2 hours ago|

parent|

[-]

It's a zero day in some caching proxy who may not have had that threat model in mind. I guess its primary purpose is caching packages, not restricting internet access.

That's a far cry from finding a zero day in a hypervisor or even Docker, which OpenAI conveniently left out in their first statement by simply calling it a "sandbox".

by 0xDEAFBEAD2 hours ago|

parent|

[-]

I can't say for sure, but intuitively this type of vulnerability seems like it could be handy for an attacker trying to move through a target network? In any case, JFrog seems to be taking the situation quite seriously

https://jfrog.com/blog/jfrog-and-openai-collaboration-on-zer...

by justincormack1 hours ago|

parent|

prev|

[-]

That is the threat model, that is supposed to be safe isolated access in a vpc that isolated applications can access but it has external access.

by cmiles81 hours ago|

parent|

prev|

[-]

The deeper question though is why is someone running a model that’s supposedly so dangerous in an environment that can even get access to the internet. That just appears reckless.

There are ways to test supposedly dangerous things. What was on display here looks more amateur hour than serious testing.

by nickpsecurity2 hours ago|

parent|

prev|

[-]

I wonder if whatever had the zero day was written in a memory-safe language with strong authentication and a secure parser. Such were the recommendations to stop many 0-days before GPT-2 was invented.

If it had poor security, the attack would be both evidence of poor security and proof that the agents can compromise poor security which might still be amazing.

by justincormack1 hours ago|

parent|

[-]

Its likely written in Java, so memory safe.

by PunchyHamster3 hours ago|

parent|

prev|

[-]

In OpenAI sandbox. which was probably vibe coded. That is to say it is probably far easier to achieve than escaping something more battle tested.

> Note that regulation is useless here, because black hats don't give a crap about regulators!

I'd argue more than useless and actively harmful, as you get denied access to tools that black hats use against you.

by 0xDEAFBEAD3 hours ago|

parent|

[-]

"To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor."

https://openai.com/index/hugging-face-model-evaluation-secur...

Based on some quick Googling, Artifactory was created by a software supply chain company called JFrog, which is publicly traded and valued at around $9 billion.

>I'd argue more than useless and actively harmful, as you get denied access to tools that black hats use against you.

Indeed. Still, perhaps the US and China can coordinate to stop releasing new models with even more advanced capabilities. Such coordination could be bad for big AI companies in the US, since they've been making money by offering more powerful models than the open-weight stuff which comes out of China. Any halt to the AI race will create equalization and erode their competitive edge. Speaking for myself, I personally wouldn't mind seeing US AI companies take a financial hit from such an an international agreement.

by plandis1 hours ago|

prev|

[-]

I wonder how current models would handle something like the xz backdoor? Could they perform the social engineering and technical aspects to make something like that happen?

by TeMPOraL1 hours ago|

prev|

[-]

The elephant in the room is still that this was done by an LLM and not by a human. Script kiddie or not, in this reality, in the year of our lord 2026, people still call SOTA LLMs "useless", "scams", and "stochastic parrots".

by AlienRobot37 minutes ago|

prev|

[-]

It's only script kiddie hacking if you get the exploit from someone else. If you program it yourself, it's normal hacking.