The same issue would occur if glibc adds a new version of an existing symbol and then the GPU driver is recompiled. (Or, for that matter, if a GPU driver adds a dependency on a symbol which glibc has always supported but which isn’t in the subset that you reimplemented, though in theory that could be solved if you reimplemented 100% of the symbols.)
It's also not just new symbols, the loader semantics also aren't static and new enough libraries may not support older semantics - e.g. the loader used to use DT_HASH entries for symbol resolution but now they are no longer present on all distributions.
I wish it would that simple for practical use cases.
I ship professional software for colorists for Hollywood studios and they absolutely love to never upgrade. We have to ship for RockyLinux 8. Sad.
>Rocky Linux 8 is supported by the Rocky Linux project until May 2029.
Just the other day I tried running an older binary and it failed with a glibc error, despite it being linked to a glibc version that's barely 5 releases behind the one on my system. So maybe glibc isn't backwards compatible after all...
I'm not offering a silver bullet, but the approach I've implemented is much better than what the industry currently offers.
Advocates of static linking keep forgetting once upon a time UNIX only had static linking, then we had overlays, and eventually dynamic linking came to be.
And I also remember very well that dynamic linking appeared ONLY because we were catastrophically short on memory; everything else was added much later.
Now we have plenty of memory, and we can very well return to our blessed roots!
Ah, you have lots of memory, we're very wealthy. /s
Yes, it's not trivial, but I hope that over time everything will settle down.
> if I understood your implementation correctly you're hooking this into an already running musl which could cause backwards compatibility issues if musl changes under you
No, I hook this to musl, which is statically linked into my binary, and I have complete control over it.
From what I remember, GPU access on Linux 'works' by accessing specific FDs under /dev, which are vendor specific - this is what these libs do under the hood.
The libraries don't have any magic powers - if the FD is inaccessible, you won't be able to do anything.
So there's some vendor specific access needed in containers anyway (or a blanket allow, which is a BAD idea).
Also not sure why dynamic linking isn't good enough for this - the issue lies with the permissions, not how you load/link libraries.
> GPU: Vulkan and OpenGL drivers are supplied by the host as shared objects, usually built against glibc, and a fully static musl binary cannot normally dlopen() them.
Why? Have people managed to break the ancient concept of shared libraries, and this is a fix for that?
Shared libraries have always been broken in Linux. Unfortunately many things like GPU drivers, graphics libraries and NSS need shared libraries to dynamically load certain runtimes (because you don't want to load all possible GPU drivers in existence to your RAM). So an ecosystem has been developed on top of terrible ABI and architecture GNU/glibc provided.
I've become obsessed with getting rid of it, especially after I realized that contributing to GNU itself was a dead end. Freestanding Linux programming turned out to be much more fun anyway.
All libraries out there should adopt the SQLite design: programmers provide it with all the necessary functions. Instead of libraries hard depending on glibc, we get to inject the libc-ish subset it needs. Then we can use whatever we want under the hood. I'm working on porting SQLite to freestanding Linux system calls so it can run with zero dependencies. Wish I could say the same for software like mesa, I'd need a lot of help for this one...
Can't be done. The libc is legacy, it can't be changed without breaking everything. It's also mandatory on every operating system other than Linux.
A change in paradigm is necessary. Freestanding C, not hosted C. This completely gets rid of the libc and is a surprisingly clean language. Linux only, because it's the only kernel with a stable binary interface. Every other OS forces a C runtime.
I once worked on a liblinux project that embodied this... Stopped because Linux itself has a nolibc thing in the kernel tree and I didn't want to compete with it. Now I'm working on the Rust version.
> what is necessary for users of the C language to actually do stuff
Surprisingly little. I wrote an entire lisp interpreter in freestanding C with Linux system calls. It managed to survive for a rather long time without any memory allocation at all.
The system layer is refreshingly tiny. It consists of a memory allocator and extremely basic functions like memmove and strlen. I successfully got rid of total nonsense like thread local errno, locales, implicit buffering, cached global state, possibly more. All that stuff is gone! Exactly one global survived: the stack canary generated by GCC and clang. Every other symbol in the ELF is controlled by me.
Wasn't able to get rid of the NUL terminator. Linux itself needs it. To get rid of that little billion dollar mistake requires an entirely new kernel with zero UNIX/POSIX influence. I had to make my peace with that one. All my buffers maintain an extra NUL byte at the end.
Great choice for small programs, but what if I want hardware accelerated 3d?
With Linux system calls alone it should be possible to set up kernel mode setting without depending on any toolkit at all. This should be enough to get a framebuffer for software rendering.
For hardware acceleration though, one must give this graphics context to an OpenGL ES implementation. That's where it gets ugly. There is no way to divorce that from the libc short of literally rewriting it.
Maybe Vulkan will enable it? I can't say for sure at my current knowledge level.
I'm sure those OSes make efforts to make said runtime binary compatible between executables.
That's when you run into the Darth Vader of binary interfaces.
> I have altered the ABI. Pray I do not alter it further. -- De Raadt
Windows explicitly does not want you to link the system libc. You are expected to bring your own, and doing so means your process has multiple libc's loaded into its address space.
And if you choose to build a binary that doesn't need a libc, you won't be bringing one.
This is categorically false; UCRT[1] is a thing. The 'U' stands for universal. Unlike Linux, Windows allows developers to choose their ABI boundary and also ship that boundary if they desire, or use the 'system' one and ask older platforms to install redistributables or Windows update packages. There's the old and new C runtimes in MSVCRT.DLL and UCRTBASE.DLL, the C++ runtime in VCRUNTIME140.DLL, Win32 in KERNEL32.DLL, USER32.DLL and more, and then the stable-ish kernel interfaces in NTDLL.DLL, in order of 'closeness to the kernel'.
And also, 'libc' is a UNIXism; on Windows the term is CRT, for 'C runtime'.
[1]: https://learn.microsoft.com/en-gb/cpp/porting/upgrade-your-c...
> At some point, the decision was made to just give up and declare it an operating system DLL, to be used only by operating system components.
https://devblogs.microsoft.com/oldnewthing/20140411-00/?p=12...
That only massively compounds the problem.
> And if you choose to build a binary that doesn't need a libc, you won't be bringing one.
NT system calls are not stable. You still need to link against ntdll.dll at the very least, like a forced Linux vDSO.
The Windows ecosystem, that manages to deliver built binaries easily & widely, regardless of whether the author has a 1 year old OS or a 15 year old OS, suggests that it's not as big a problem as you believe.
SQLite already does this with its VFS layer. You hand it the OS functions it needs instead of it grabbing them, etc.
You could look at going straight to syscalls, mesa would be a rewrite nightmare
I went zero-dependency on a production CMS. Rust, no framework, no external crates beyond argon2 for password hashing. Running live on client sites. The hard part wasn't building it, it was accepting that everything you reach for is pulling glibc or similar assumptions back in through the side door. SQLite's VFS model is the right pattern. More things should work that way.
That's currently the real core of the problem.
The loader (and libdl) need to be decoupled from the glibc itself under Linux.
Without that, any attempt to ship static binaries (or any binary with a different Libc) will be a source of perpetual pain.
nss plugins and its associated pain (sssd and avahi) are an other examples of that.
C++ abi should not be included in this. It is independent from the other pieces and historically a source of incompatibility on its own.
Saying "C/C++ abi" as if they are the same is looney tunes, the former is very simple and stable and the latter is very complex.
How libstdc++ initializes global variables absolutely depends on glibc and ld-linux.so. That is part of C++ ABI.
Somehow most of my portability issues seem to be caused by glibc, its symbol versioning and close ties to the dynamic loader. Minor versions aren't compatible, no two Linux distros ship the same version and you can't just provide your own without also patching in your own dynamic loader.
At least as far as the defaults on Linux go I consider C the root of all evil.
In which case as long as you're using the documented public API and compile your program with the oldest version of glibc you want to support (some Ubuntu from 4-5 years ago should cover pretty much every current desktop) you should be fine. And with something like Docker this is trivial to do.
Sure it is annoying that you cannot use your current distro (especially if you use some rolling distro) to make binaries for everyone, but it takes very little effort to work around that. The only issue i can think of is if you absolutely want to compile using the latest version of your compiler and you cannot build the compiler from source to work in the Docker (or whatever) contain to work against the older glibc.
In complex cases, it turns out that the old version of glibc also pulls in other libraries and the compiler, and you're stuck with a very ancient sysroot. You may often find that you can't compile new library versions in such a sysroot and link them statically.
So, it looks good on paper, but forget about the ravines.
All solutions to this problem are hacky, complex and controversal and highly fragmented, where this should be BASIC functionality
Or do the MS thing, and ship multiple versions like msvcrt
C++ global/static variable initialization depends on the specific version of glibc (they don't usually break compat, but they can and they did in the past) which also provides ld-linux.so that loads those global variable placeholders in the correct manner such that glibc and libstdc++ can initialize them correctly.
This is just one example. Thread local variables and behavior of things like pthreads with signal, fork etc all depend on glibc.
And that's the correct approach and also one that many have taken. We just need someone willing to maintain that as an easy mode SDK for everyone.
Who said that? This approach has many problems that have already been discussed here, not to mention the fact that it leaves Alpine and Bionic-based systems out in the cold.
Let me remind you that Bionic is the most widespread libc in the Linux world, and Alpine is the most popular Docker layer.
The world doesn't end with glibc. And it doesn't begin with it.
Until you define a thread local variable (C11) or use atomics (also C11) or define a global with an initial value. Then it happily generates code that depends on "whatever my target glibc + ld-linux.so needs".
The ABI is strongly dependent on explicit libc implementation in current Linux systems. There is no libc independent ABI on Linux.
When you compile libc, you also get a binary loader ld-linux.so with it. They are not two independent components of a system.
Basically all .so files compiled with glibc require the ld-linux.so that's also generated by that glibc (or a later version, if they didn't break the binary compatibility).
There are a lot of stuff that's executed by ld-linux.so and glibc that are not explicitly documented but they are absolutely necessary for your program to start and correctly initialize things like global variables or signal handling or loading other dynamic libraries. Some of that functionality sits in ld-linux.so and some of that in glibc. They have circular dependencies to each other. glibc expects ld-linux.so to put things in certain order but ld-linux.so also must load glibc first to have access to certain APIs. They are not part of System V ABI. They are not documented.
Musl maybe can implement this but it is simply reverse engineering what glibc did and then playing a game of cat and mouse. There is no independent ABI standard.
Then, what is the exact reason a library compiled against glibc must be loaded by a specific ld-linux? I could see that this is true for C++ perhaps, or when you use very special features, but I do not see this for C.
I often compiled programs against one version of glibc and run it against a different version, so I know there is not a tight coupling. So please be specific in explaining in what scenarios this would break.
Solo is basically pg83's answer to the architecture you just described. If there is no libc independent ABI, build your own loader and shim the boundary.
I can understand Linus's obsession with taste and the areas it was overlooked or traded.
Not really, though. glibc uses symbol versions that are forward but not backward compatible. If you got an error that said "this program was built for a newer version of <distro>" would you say the same thing?
Note this is the same (if not worse) on MacOS, and on windows you used to distribute the CRT with your application just to deal with the same problem.
Yes glibc has some backwards compat but you cannot load a binary compiled with a newer version of glibc using an older ld-linux.so. That's because the interdependency. Nor you can load binaries that depend on different libc.so files with glibc systems
I cannot comment on macOS, I have never used it. However this is not a problem with Windows. You can ship a newer CRT or you can install it as a system component using Microsoft's MSI. The dependency is one way on Windows. CRT purely depends on Win32. Moreover the loader is completely independent and DLLs are loaded into their own unique scoped namespace unlike Linux that loads them in global symbol namespace. That's why you can mix and match DLLs compiled for different CRT versions.
On Windows you don't need to ship a new binary loader. I can just ship Windows 10 UCRT DLL (which is the new libc of Windows) to Vista and my binaries will work. The binary loader isn't interlinked with the libc.
One of the ways Windows manages to support multiple libc's is by being careful not to mix allocators; if a system API you call allocates on your behalf, your libc can't free it, the system API will offer a function to free it.
Windows loader is certainly available to user programs though; LoadLibrary has been around longer than many developers.
As for older Windows systems not being able to load new DLLs, they can; the format hasn't changed in a very long time. I've had experience with installing some DLLs on Windows NT 3.51 and running a modern Firefox, which is about 20 years behind the times.
MS haven't made this work arbitrarily far back, I believe they deprecated targeting Windows XP in one of the more recent toolchains, but that was purely a "not worth supporting" situation.
EDIT: mixed up talking about older & newer
Additionally many other OSes have had both approaches since their early days, Xerox PARC ones.
For some strange reason they assume to know better than all those researchers.
What you can't do is build something statically with musl and then reliably dlopen shared libraries built with glibc.
If you break or remove a shared lib here, you may no longer be able to compile something from source. I had that happen in the past before I started to use more statically compiled programs (and busybox too).
Assuming everything works as-is via shared libraries at all times, makes little sense for ALL linux systems. For instance, some people upgrade glibc manually. Then you need a working base system to resume compilation. I do that for my customized gobolinux system, so I can use any program version as well as any glibc version (assuming I can still compile the program; many older programs no longer compile).
That is when I realized the true horror of (g)libc. It wants to inject itself at the root of the library/program and everything from threading to dlopen/dlsym is impacted. I tried a lot of workarounds including trying to implement a loader myself, but the complexity (and fragility) grew so much that I felt it was not worth it.
Finally, I retreated into the safe world of a freestanding runtime + syscalls. FFI, if it has to happen, will occur via IPC of some kind. A second process linked against glibc that will manage calls on behalf of the clean first one.
Rude.
It only gets better from there - https://github.com/pg83/solo/blob/main/CONTRIBUTING.md!
Telling me, a coder you never met or interacted with before, that I would introduce more bugs than The Product(tm) is pretty rude and prejudiced.
> The project author believes that, with capable human direction, modern LLMs write code faster than people and introduce fewer bugs.
Obviously, this is an offense to me.
> Telling me, a coder you never met or interacted with before, that I would introduce more bugs than The Product(tm) is pretty rude and prejudiced.
Don't exaggerate. I didn't say that you personally introduce more bugs (as you rightly pointed out, I don't know you and don't know how often you introduce bugs), I said that the average developer introduces more bugs than an SOTA LLM.
But why? You didn't write it. You said so yourself, you wrote the initial text but it was all reworded by claude.
I could choose to believe in a social fiction and you can't stop me.
If we look at the issue at its core, we're still a statically linked program in a hostile environment, forced to dynamically load device drivers from the system.
It's similar to Golang; on MacOS, it has to use libSystem, even though otherwise, these are the statically linked Go binaries we're used to and love.
Let me add a little more detail: if I use vdso with gettimeofday in a statically linked program on Linux, am I still a statically linked program, or not? :)
Actually, scratch that, because Windows and macOS have historically struggled with ABI compatibility as well (macOS less so, due to not caring about backward compatibility in the first place).
How did we get to the point where people feel they need to go to the length of embedding an ELF loader in their binary (!!) rather than just linking with glibc?
Most Linux distros have been built around the ability to compile their software together in a large repository, from source, so this was rarely ever an issue. Proprietary distribution or executing binaries from the internet like on Windows just wasn't really a common issue.
The problem arises when you start combining distros (glibc and MUSL for instance) or if you try to do the Windows model of sharing software. Historically, projects just compiled different versions for different distros.
When doing static compilation, just targetting an old version of glibc (which is generally forward compatible) also works.
You can hack your way into using software like this (or rather, have an LLM hack its way in) but I don't think any real distro actually cares. This issue exists in a quite small space where people are trying to use proprietary software built for glibc in MUSL environments for whatever reason, and the usual compatibility tricks don't work.
It's a niche use case for most Linux distros. It's not a "complete failure" of the GNU/Linux userland, it's the result of a couple of proprietary components not having MUSL builds available, or MUSL-based distributions not including libraries people want.
I'd like glibc to change so that these hacks aren't necessary for these use cases anymore, but it's not really a problem in practice for the vast majority of Linux use cases.
Yes, on the one hand, each specific distribution doesn't have this problem because it can pick up everything it needs.
But for us, independent developers of small programs, the problem is truly stark: we can't afford to build our programs for every distribution. And we can't afford to waste time navigating all the idiosyncrasies of various package repositories, both technical and political (not all repositories allow easy access).
And we can't count on someone else packaging our work until we become incredibly popular.
Most people use the major distributions like Ubuntu, Debian, Fedora, Arch, alpine linux, void linux,… A build machine with VMs can easily take care of those.
> And we can't afford to waste time navigating all the idiosyncrasies of various package repositories, both technical and political (not all repositories allow easy access).
Most (if not all) package managers allow for custom repositories. No need to get access to the official ones.
This is possible because Zig allows for targeting most glibc versions out of the box with its cross-compilation support.
1) Why should I limit myself to the available APIs?
2) Not just glibc. For example, if I build against the latest libstdc++, it will automatically support the more recent glibc. And pinning the old libstdc++ -well, that's just not a good idea.
I think the disconnect is mostly people that can't decide if they want a stable distro or a rolling release distro. Most everyone uses a stable distro because it's stable, but then the want some up-to-date software that isn't ore-built for their (crusty old) stable distro and they get annoyed. My solution was to finally give in and embrace a rolling release distro (I use arch, btw). If there isn't a package for something I want, it's not hard to build something myself because all my build tools, kernel, and libs are up to date.
Other reasons to want static linking is to distribute proprietary software with no source code available. Linux certainly does not cater to that scenario and I suppose some might call that a complete failure ¯ \ _ ( ツ ) _ / ¯
There are also much less well-known "little things" that regularly pop up here and there.
> If your CI uses an older glibc you should be fine AFAIK.
In any case, my binaries work not only under glibc, but also under Alpine, and (work in progress) under android/bionic.
Not sure how the comments about alpine or bionic relate either to my claim that cross distro glibc is fine.
Overall, both of our points of view on compatibility were discussed well in that thread; we probably shouldn't repeat ourselves. :)
By that definition, any change of any kind will break ABI, Hyrum’s law and spacebar heating and all that.
- Linus Torvalds
That bug report was a good read.
You can also compile using old distro and old Glibc to get similar effect. Though you would miss the advances of the newer compilers.
https://docs.appimage.org/reference/best-practices.html
I hear you about WINE though.
Due to FUD, mostly.
Sane people do just link with an old enough version of glibc.
valid point. I am usually okay with LLM generated code since even if it might not be architecturally sound It is usually well commented and has tests and documentation for helping another agent/human debug any issues.
But, just the painful experience of debugging any dlopen related crashes and/or intermittent bugs; and the sheer amount of tokens burnt by an LLM chasing tangents when shown a stack trace; I wouldn’t touch this at least as a packager/consumer of certain apps for personal usage on older distros. So far, AnyLinux-Appimages seem to be a mature solution with great support from the developers, in case anyone lands here for packaging applications to run on older distros.
I don't think anybody believes this, and interjecting it into every thread is not really convincing anyone.
The Linux ecosystem already settled on containers to solve the problem. Namely Docker, Flatpak, Podman ec.
It may not be clear to a non-english first language person, but when I read Claudes documentation, my brain immediately picks up claude-speak. Therefore, I expect the code to be generally correct, maybe, depending on how specific I was during my prompting. In no way shape or form do I really trust it, at least until i dig into the code and validate my mental model. And query the review for edge cases etc...
By writing your documentation with Claude, my brain immediately associates the quality of the project with the quality of unreviewed Claude output.
To a degree this is unfair, as it is like judging the quality of someone's work based on their accent.
But Claudes accent, has a high correlation with Claude, so unlike with people, where an accent has no bearing on technical ability, Claude being Claude does.
I would much rather read a typo ridden sentence than claudism, even just as a forward, explaining what you did vs the ai, and telling the users how much we should trust it.
Also, If you really insist on using AI to write, have another model rewrite docs/comments into regular English (opus 4.6 for example is much better than 4.7, 4.8, or 5.
It is open source so you can do whatever you like, but people (especially native English speakers) will discount your work, because Claude, especially opus 5 writes very very badly.
The people who say they would rather see a typo infested mess of a readme are serious. Or even just write in your native language and then have Claude translate.
Both of those are better indications of proof of effort than a Claude readme.
That's exactly what I did.
I'm sorry, but I agree with the author: if a certain writing style makes you associate the work with low-quality, then that's your problem. The author shouldn't have to rewrite the readme just to avoid triggering your automatic unfounded associations. If you look at the substance of the work, including the test cases, then this is clearly not easy work that can be vibe coded in a single pass.
It's just like emdash. Everybody digs on how it's a signifier of LLM text, but I've used emdash for years because it's gramatically correct. I shouldn't have to stop using emdash just to avoid kneejerk reactions.
statistically, it only makes sense for your expectations to immediately be low when you encounter this writing style because there are just so much slop out there
the author is free to keep that writing style but they should be aware that this will—at least on the surface level—make their project look exactly like the metric ton of slop we see posted everyday everywhere
Yacks
[1] https://www.linker-aliens.org/blogs/rie/entry/direct_binding...
[2] https://web.archive.org/web/20011004090044/http://developer....
Purely academically, this is a very cool piece of code! Just hoping that it gets a thorough vetting before used by privileged/security-critical software :)
In any case, we take testing very seriously—every glibc shim we've written is covered with tests, and we run our loader against 1000 of the most popular Debian packages. The project has 100% code coverage. Perhaps, if I have the time, I'll also do some fuzzing on this thing.