- Impacted versions (v1.82.7, v1.82.8) have been deleted from PyPI - All maintainer accounts have been changed - All keys for github, docker, circle ci, pip have been deleted
We are still scanning our project to see if there's any more gaps.
If you're a security expert and want to help, email me - krrish@berri.ai
What about the compromised accounts(as in your main account)? Are they completely unrecoverable?
And
> Dropped you a mail from [email]
I don't think there is any indication of a compromise, they are just offering help.
The compromised PyPI packages were litellm==1.82.7 and litellm==1.82.8. Those packages have now been removed from PyPI. We have confirmed that the compromise originated from the Trivy dependency used in our CI/CD security scanning workflow. All maintainer accounts have been rotated. The new maintainer accounts are @krrish-berri-2 and @ishaan-berri. Customers running the official LiteLLM Proxy Docker image were not impacted. That deployment path pins dependencies in requirements.txt and does not rely on the compromised PyPI packages. We are pausing new LiteLLM releases until we complete a broader supply-chain review and confirm the release path is safe.
From a customer exposure standpoint, the key distinction is deployment path. Customers running the standard LiteLLM Proxy Docker deployment path were not impacted by the compromised PyPI packages.
The primary risk is to any environment that installed the LiteLLM Python package directly from PyPI during the affected window, particularly versions 1.82.7 or 1.82.8. Any customer with an internal workflow that performs a direct or unpinned pip install litellm should review that path immediately.
We are actively investigating full scope and blast radius. Our immediate next steps include:
reviewing all BerriAI repositories for impact, scanning CircleCI builds to understand blast radius and mitigate it, hardening release and publishing controls, including maintainership and credential governance, and strengthening our incident communication process for enterprise customers.
We have also engaged Google’s Mandiant security team and are actively working with them on the investigation and remediation.
It would be great if Linux was able to do simple chroot jails and run tests inside of them before releasing software. In this case, it looks like the whole build process would need to be done in the jail. Tools like lxroot might do enough of what chroot on BSD does.
It seems like software tests need to have a class of test that checks whether any of the components of an application have been compromised in some way. This in itself may be somewhat complex...
We are in a world where we can't assume secure operation of components anymore. This is kinda sad, but here we are....
It is so much better than, you know... "We regret any inconvenience and remain committed to recognising the importance of maintaining trust with our valued community and following the duration of the ongoing transient issue we will continue to drive alignment on a comprehensive remediation framework going forward."
Kudos to you. Stressful times, but I hope it helps to know that people are reading this appreciating the response.
Personally, I am a heavy user of Firejail and bwrap. We need defense in depth. If someone in the supply chain gets compromised, damage should be limited. It's easy to patch the security model of Linux with userspaces, and even easier with eBPF, but the community is somehow stuck.
But, sadly, there's no x-platform way to do this, and sandboxing APIs are incredibly bad still and often require privileges.
> It's easy to patch the security model of Linux with userspaces, and even easier with eBPF, but the community is somehow stuck.
Neither of these is easy tbh. Entering a Linux namespace requires root, so if you want your users to be safe then you have to first ask them to run your service as root. eBPF is a very hard boundary to maintain, requiring you to know every system call that your program can make - updates to libc, upgrades to any library, can break this.
Sandboxing tooling is really bad.
(And no, most linux namespace stuff does not require root, the few things that do can be provided in more-controlled ways. For examples, look at podman, not docker.)
That's true, sort of. I mean, that isn't the whole point of sandboxing because the threat model for sandboxing is pretty broad. You could have a process sandbox just one library, or sandbox itself in case of a vulnerability, or it could have a separate policy / manifest the way browser extensions do (that prompts users if it broadens), etc. There's still benefit to isolating whole processes though in case the process is malicious.
> (And no, most linux namespace stuff does not require root, the few things that do can be provided in more-controlled ways. For examples, look at podman, not docker.)
The only linux namespace that doesn't require root is user namespace, which basically requires root in practice. https://www.man7.org/linux/man-pages/man2/clone.2.html
Podman uses unprivileged user namespaces, which are disabled on the most popular distros because it's a big security hole.
Not really.
Let's say I am running `~/src/project1 $ litellm`
Why does this need access to anything outside of `~/src/project1`?
Even if it does, you should expose exactly those particular directories (e.g. ~/.config) and nothing else.
> Even if it does, you should expose exactly those particular directories (e.g. ~/.config) and nothing else.
Yes, but now you are in charge of knowing every potential file access, network access, or possibly even system call, for a program that you do not maintain.
Not really. I try to capture the most common ones for caching [1], but if I miss it, then it is just inefficient, as it is equivalent to a cache miss.
I'll emphasize again, "no linter/scanner/formatter (e.g., trivy) should need full disk access".
1 - https://github.com/ashishb/amazing-sandbox/blob/fddf04a90408...
Have you solved for publishing? You'll need your token to enter the container or you'll need an authorizing proxy. Are cache volumes shared? In that case, every container is compromised if one is. All of these problems and many more go away if the project is built around them from the start.
It's perfectly nice to wrap things up in docker but there's simply no argument here - developers can write sandboxes for their software more effectively because they can architect around the sandbox, you have to wrap the entire thing generically to support its maximum possible privileges.
Inside the sandbox but not on my machine. Show me how it can access an unmounted directory.
> Have you solved for publishing? You'll need your token to enter the container or you'll need an authorizing proxy.
Amazing-sandbox does not solve for that. The current risk is contamination; if you are running `trivy`, it should not need access to tokens in a different env/directory.
> All of these problems and many more go away if the project is built around them from the start.
Please elaborate on your approach that will all me to run markdown/JS/Python/Go/Rust linters and security scanners. Remember that `trivy` which caused `litellm` compromise is a security scanner itself.
> developers can write sandboxes for their software more effectively because they can architect around the sandbox,
Yeah, let's ask 100+ linter providers to write sandboxes for you. I can't even get maintainers to respond to legitimate & trivial PRs many a time.
So it says right on the tin of my favorite distro: 'Warning: Beware that the docker group membership is effectively equivalent to being root! Consider using rootless mode below.' So # docker run super-evil-oci-container with a bind mount or two and your would-be attacker doesn't need to guess your sudo password.
1. That super evil OCI container still needs to find a vulnerability in Docker
2. You can run Docker in rootless mode e.g. Orbstack runs without rootRootless mode requires unprivileged user namespaces, disabled on almost any distribution because it's a huge security hole in and of itself.
My dad was confident he could figure it out based on his perplexity Pro account. He attacked the problem from several angles and used it for help with what to do, how to do it, what to ask for when visiting offices, how to press them to move forward, and tons of other things.
Got the problem resolved.
So it definitely can reduce hiring lawyers even.
Same logic for software developers.
Write a detailed postmortem, share it publicly, continue taking responsibility, and you will come out of this having earned an immense amount respect.
Were you not aware of this in the short time frame that it happened in? How come credentials were not rotated to mitigate the trivy compromise?
It's very much not production grade. It might miss sneaky ways to install litellm, but it does a decent job of scanning all my conda, .venv, uv and system enviornments without invoking a python interpreter or touching anything scary. Let me know if it misses something that matters.
Obviously read it before running it etc.
the developer has made a new github account and linked their new github account to hackernews and linked their hackernews about me to their github account to verify the github account being legitimate after my suggestion
Worth following this thread as they mention that: "I will be updating this thread, as we have more to share." https://github.com/BerriAI/litellm/issues/24518
also how are we sure that docker images aren't affected?
That being said this hack was a direct upload to PyPI in the last few days, so very unlikely those images are affected.
EDIT: no, it's compromised, see proxy/proxy_server.py.
Was your account completely compromised? (Judging from the commit made by TeamPCP on your accounts)
Are you in contacts with all the projects which use litellm downstream and if they are safe or not (I am assuming not)
I am unable to understand how it compromised your account itself from the exploit at trivvy being used in CI/CD as well.
We have deleted all our pypi publishing tokens.
Our accounts had 2fa, so it's a bad token here.
We're reviewing our accounts, to see how we can make it more secure (trusted publishing via jwt tokens, move to a different pypi account, etc.).
Trivvy hacked (https://www.aquasec.com/blog/trivy-supply-chain-attack-what-...) -> all circleci credentials leaked -> included pypi publish token + github pat -> | WE DISCOVER ISSUE | -> pypi token deleted, github pat deleted + account removed from org access, trivvy pinned to last known safe version (v0.69.3)
What we're doing now:
Block all releases, until we have completed our scans
Working with Google's mandiant.security team to understand scope of impact
Reviewing / rotating any leaked credentials
https://socket.dev/blog/trivy-under-attack-again-github-acti...
https://github.com/BerriAI/litellm/issues/24518#issuecomment...
In CI they could easily have moved `trivy` to its own dedicated worker that had no access to the PYPI secret, which should be isolated to the publish command and only the publish command.
Privilege escalation on macOS is very hard indeed. Apple have been improving security for a long time, it is far, far ahead of Linux or Windows in this regard. The default experience in Xcode is that a release-mode app you make will be sandboxed, undebuggable, have protected keychain entries other apps can't read, have a protected file space other apps can't read, and its own code will also be read-only to other apps. So apps can't interfere with each other or escalate to each other's privileges even when running as the same UNIX user. And that's the default, you don't have to do anything to get that level of protection.
App Store apps are the exception, which is great, but presumably we're not talking about that? If we are, then yeah, app stores solve these problems by making things actually sandboxed.
Token in CI could've been way too broad.
From a security standpoint, you would rather pull in a library that is compromised and run a credential stealer? It seems like this is the exact intended and best behavior.
More strangely (to me), this is often addressed by adding loads of fallible/partial caching (in e.g. CICD or deployment infrastructure) for package managers rather than building and publishing temporary/per-user/per-feature ephemeral packages for dev/testing to an internal registry. Since the latter's usually less complex and more reliable, it's odd that it's so rarely practiced.
No one initially knows how much is compromised
That said, I'm sorry this is being downvoted: it's unhappily observing facts, not arguing for a different security response. I know that's toeing the rules line, but I think it's important to observe.
i'd much rather see a million open PRs than a single malicious PR sneak through due to lack of thorough review.