Of course, this is already miles better than the current state of affairs where an old but popular package could become an infection vector at any time.
I guess the use case is that you might want to build a wheel for an older release for a newer version of Python?
(This will hopefully change pretty soon, with the “upload 2.0” work.)
1) Upload all files in a staging state. Can be done asynchronously via multiple build hosts. Files in this state are referenced via their cryptographic checksums (e.g. SHA, Blake, etc)
2) Make visible with a single call by providing a manifest with checksums for all source dists/wheels contained in the release. All artifacts are made visible atomically and the release is immutable.
That allows authors to prepare uploads over however many days they need to coordinate hardware, but doesn’t allow for users to discover a release in a partial state.
The time limit is needed because a release can contain different binary wheels for different architectures.
Consider the simplest case: your releases go out via GitHub Actions and separate wheels are built on the Windows, Linux, and macOS runners.
Those won't all end at exactly the same time, so you need a release window during which they can finish and upload their generated files.
That window used to be unlimited, now it's 14 days.
That might seem like a long time, but it means more manual release processes still have time to coordinate, or release processes that need access to less common hardware that might require queuing for a while.
I understand wanting to get things out ASAP. but two weeks is not the end of the world to wait as a consumer of professional software packages.
2. There are hashes in the HTML served by PyPI. These are updated at the whim of both the index and the publisher. Even though they are checked by pip during install, they are worthless.
3. There are many ways to install packages that work around (2). Custom index server doesn't have to provide hashes, and pip will happily install that. You can install from sources, from a package you've downloaded somewhere, form VCS, you can build it during install, all without even prompting the user to confirm the very scary choices.
NB. I have no idea how do you make the leap from "adding files to release" to "not modifying the release". To me, adding file to release is sure as hell modifying it. Here's a very simple malicious example:
I release package "innocent" with an empty "scripts" section. Then, in the subsequent modification to this release, I add the "scripts" section with a script named "notebook". Now, whenever my user wants to run Jupyter notebook, they will call my "notebook" program, not the one from Jupyter package.
In concept why can’t the full set of files in a release be a single, one-way hash value, with both adding or releasing changing the hash value?
(Also: we’d need to determine what it even means to hash a set of files. Do we order by canonicalized filename, by upload time, etc. Each of these has surprising implications!)
Edit: to be clear, it’s not ideal or fully intentional that uploads to PyPI are not atomic. But it’s the status quo and any change will take years.
If it weren’t for AI, Python would have died as a language ecosystem due to their perpetual neglect of this area. The python community clearly had no ability to solve this problem themselves. I got sick and left and started using better tools.
It’s only now that they’re the cornerstone of the world economy that other people are coming in and cleaning up their messes for them. It’s pitiful. Python failed and needed to be saved from itself.
You lock to the distribution instead of the release so your build doesn't have to download distributions for platforms you're not using.
PyPi just decided to do what VCSes already did, worse
We also figured out 20 years ago that SHA1 was not quite as strong as initially estimated, and not quite 10 years ago that generating two colliding documents was merely a matter of some serious computing power. A few projects went ahead and changed the name of their master branch, but SHA256 preference remains elusive.
The release consists of one sdist and zero or more wheels. Until now you were able to upload additional wheels at later time.
(This doesn’t change your observations at all! Just as a demonstration of how Python packaging’s data model can be unintuitive.)
Something that I wish was included in PEP-694 is the notion of a "tombstone" invalid/empty sdist artifact to deal with that case, to make it easier for package managers to interpret and surface those different failure classes.
Like, sure, this is a package manager UX problem, but it's been happening for long enough that I feel like it might be time for PyPI to help managers do better.
While this may quantify how disruptive the change would be to those projects that are able to and do upload additional binaries to PyPI later, it fails to quantify how many projects already completely circumvent this block before it is even introduced.
e.g. If you tell pip to install from source.. the result may already be that you install a binary that PyPI never saw. A common hack for dealing with NVidia internals, which can explode into a large CUDA major version x GPU arch x platform x implementation x python_version cartesian product. The "extras" mechanism is not quite sufficient to model such combinations.
sample code: https://github.com/Dao-AILab/causal-conv1d/blob/4f6ae4e26ae5... https://pypi.org/project/causal-conv1d/
Every one I’m aware of works either as a one-shot (you have to submit everything in one push) or lets you keep adding new assets forever (other, obviously, than PyPI with the addition of this 14 day wall).
Again, can you think of any packages managers that have a finalize step like you’re describing?
All the package managers I’m aware of do one of two things:
1. You push once with everything baked in.
2. You push as many things as you want forever.
Python releases can sometimes have many different package files (for example, because each platform can have its own wheel), which makes the first option pretty challenging.
You’ll never get served a partially uploaded wheel for amd64 Linux, but somebody could come back a year later and add a 2nd wheel to that release for a different architecture.
The way it's done with rubygems, if you messed something up with the gem (pushed secrets, etc.), you "yank" (remove) the release and push a new one (different version). You can't work with the files in a release once it's been pushed.
This is unintuitive, but the TL;DR is that files will never change on PyPI, but (previously) a user could upload a new file to a release years after their last upload to that release. This has some legitimate use cases (like allowing people to support new Python versions without bumping a package’s version), but also makes introduces challenges around locking and release security that are elaborated in the thread linked by the blog post.
I agree this could probably be ratcheted down from 14 days over time, though.
You can’t go back later and add “evil.py” to a bunch of existing release files, but you could previously go find a bunch of releases that didn’t have arm64 files, publish malicious ones, and use that to catch people using those versions on arm64 systems
...why Python is just breaking compatibility so bad with new version it needs that ?
If you ever used Maven, NPM or... I can't think about any other tool that doesn't automatically check checksums and signatures. Any Linux package manager ever used... Python's Wheel format has provisions for checksums and signatures! But they aren't checked.
Instead Python gets absurdly ineffective workarounds that will probably inconvenience a few developers and will do zilch for users.
The tool fragmentation is insane, the demand to create "source distributions" was maybe funny in 2002 but just a hindrance now.
Packages no longer build since distutils was ripped out and upstream replaced it with meson etc.
Since building from source no longer works, which is profitable for third party vendors like Conda, "wheels" are uploaded. And they cannot be built on the server since the whole "scientific" ecosystem is perpetually broken. And they are separate artifacts, leading to the above problem.
Shipping checksummed tar archives is of course it not possible, that would hurt the income streams of the package profiteers.