The new APT 3.0 solver

Can anyone explain why libc versions are so often a problem when installing software? I understand that all libraries in an executable need to use the same version of malloc. But otherwise I don't understand the reason for these clashes. Even malloc can be swapped out for another function with the same functionality. And I also don't understand why libc needs to be updated so frequently, anyway.

They are a problem because gcc automatically links to the latest version of glibc.

As to why they don't add an option to specify an older version? I don't know either and it is rather annoying to have to use docker images of older OSes to target older glibc versions. It's just one of many things that prevents linux from being as popular as windows for desktop users.

I've made a library named "glibc-compatibility": https://github.com/ClickHouse/ClickHouse/tree/master/base/gl...

When linking with this library before glibc, the resulting binary will not depend on the new symbol versions. It will run on glibc 2.4 and on systems as old as Ubuntu 8.04 and CentOS 5 even when built on the most modern system.

An easier option is to build against musl, remove the DEPENDS with patchelf, and use that. There are a few incompatibilities with glibc (more on aarch64 than am64), but it's manageable (you might need to patch a few headers in musl and provide missing/incompatible functions). And you'll have a binary that works against musl and glibc both.

Here is another interesting tool, this one for patching executables

https://github.com/corsix/polyfill-glibc

Thank you! This has solved multiple problems I've had for years.

Thanks to corsix, who is also a user here

How does this work?

In case parent didn’t see your q: https://www.lightofdawn.org/wiki/wiki.cgi/NewAppsOnOldGlibc

You are a hero.

Hmm. The MSVCRT.DLL/MSVCRTD.DLL not being binary compatible between releases of Visual Studio is the same thing, except of course you can't even combine some modules compiled for debug with modules compiled without debug in the same executable. The Windows problem has always been so so much worse that pretty much all developers simple resorted to shipping the OS system runtime with every package and it's just expected nowadays. It's where the phrase "DLL hell" originated, after all.

Not to say the ABI problem isn't real if you want to combine binary packages from different Linux-based OSes. Plenty of solutions for that have cropped up as well: containers, flatpacks, snaps, the list goes on.

> The Windows problem has always been so so much worse

Hard, hard disagree. The problems are somewhat comparable. But if any platform is more painful it's Linux. Although they're similar if you exclude glibc pain. At least in my personal experience of writing lots of code that needs to run on win/mac/linux/android.

> pretty much all developers simple resorted to shipping the OS system runtime with every package

Meanwhile Linux developers have resorted to shipping an entire OS via docker to run every program. Because managing Linux environment dependencies is so painful you have to package the whole system.

Needing to docker to simply launch a program is so embarrassing.

> except of course you can't even combine some modules compiled for debug with modules compiled without debug in the same executable

That's not any different on Linux. That has more to do with C++.

> Meanwhile Linux developers have resorted to shipping an entire OS via docker to run every program. > Needing to docker to simply launch a program is so embarrassing.

I have never needed docker "just to launch a program". Docker makes it easy to provide multiple containerised copies of an identical environment. Containers are a light alternative to VM images.

I assume you find the existence of Windows containers just as embarrassing? https://learn.microsoft.com/en-us/virtualization/windowscont...

> Docker makes it easy to provide multiple containerised copies of an identical environment.

Correct. The Linux architecture around a global of dependencies is, imho, bad and wrong. The thesis is it's good because you can deploy a security fix to libfoo.so just once for the whole system. However we now live in a world where you actually need to deploy the updated libfoo.so to all your various hierarchical Docker images. sad trombone

> Containers are a light alternative to VM images.

A light alternative to Docker is simply deploy your dependencies and not rely on a fragile, complicated global environment.

> I assume you find the existence of Windows containers just as embarrassing?

Yes.

I know my opinion is deeply unpopular. But I stand by it! Running a program should be as simple as downloading a zip, extracting, and running the executable. It's not hard!

I wish you all had experienced the golden age when running a program was as simple as apt-get install program and run it.

I find it hard to discuss the merits of Linux va windows with regard to deploying software without addressing the elephant in the room which is the replacement of a collectively maintained system that ensure software cohabitation by the modern compartmentalised collection of independent programs.

Flatpak it's fine. For serious reprocibility and unmatched software instalations, guix.

> Correct. The Linux architecture around a global of dependencies is, imho, bad and wrong. The thesis is it's good because you can deploy a security fix to libfoo.so just once for the whole system. However we now live in a world where you actually need to deploy the updated libfoo.so to all your various hierarchical Docker images. sad trombone

Only if you choose to use docker. As other people have pointed out most things on Linux can be deployed using a package manager.

> A light alternative to Docker is simply deploy your dependencies and not rely on a fragile, complicated global environment.

So like flatpak?

> I know my opinion is deeply unpopular. But I stand by it! Running a program should be as simple as downloading a zip, extracting, and running the executable. It's not hard!

How is that better than apt install? That (or equivalent on other distros) is how I install everything other than custom code on servers (that I git pull or similar).

I guess your software is not libre, right?

In that case, I think you should stick to snap and flatpak.

> Running a program should be as simple as downloading a zip, extracting, and running the executable. It's not hard!

You can do that; just most choose not to and use packaging infrastructure instead. It's not mandatory.

Instead of Docker, static linking is a much more elegant solution if library/dep management is painful.

Switching to static linking using a sane libc (not glibc) can be a pain initially but you end up with way less overhead IMO.

Static linking is a good way to avoid the problem, but I’d hardly call it “elegant” to replicate the same runtime library in every executable. It’s very wasteful - we’re just fortunate to have enough storage these days to get away with it.

> I’d hardly call it “elegant” to replicate the same runtime library in every executable. It’s very wasteful - we’re just fortunate to have enough storage these days to get away with it.

I think you need to quantify "very wasteful". Quite frankly it's actually just fine. Totally fine. Especially when the alternative has turned out to be massive Docker images! So the alternative isn't actually any better. Womp womp.

An actually elegant solution would be a copy-on-write filesystem that can deduplicate. It'd be the best of both worlds.

> Switching to static linking using a sane libc (not glibc) can be a pain initially but you end up with way less overhead IMO.

how does that work when you app needs access to the gpu drivers ?

Containers are a relatively recent phenomenon. Outside of distro packages, distributing statically linked binaries, dynamic binaries wrapped in LD_LIBRARY_PATH scripts or binaries using $ORIGIN based rpath are all options I have seen.

This comment is full of inaccuracies and mistakes, and is a terrible travesty of the Windows situation.

> except of course you can't even combine some modules compiled for debug with modules compiled without debug in the same executable.

There's a good reason for this, which IMO Unix-like compilers and system libraries should start adopting, too. Debug and Release binaries like the standard C and C++ runtimes and libraries cannot be inter-mixed because they have have different ABIs. They have different ABIs because the former set of binaries have different type layouts, many of which come with debug-specific assertions and tests like bounds-checking, exception try-catch, null-dereference tests, etc.

> The Windows problem has always been so so much worse that pretty much all developers simple resorted to shipping the OS system runtime with every package and it's just expected nowadays.

This is not true at all. There are several layers to the counterargument.

Firstly, UCRT has supplanted MSVCRT since Visual Studio 2015, which is a decade old this year. Additionally, UCRT can be statically linked: use `/MT` instead of `/MD`. And linking back to the previous quote, to statically link a debug CRT, use `/MTd`. Set this up in MSBuild or CMake using release/debug build configurations. UCRT is available for install (and maintained with Windows Update) in older versions of Windows going back to Vista.

Next, Windows by default comes with several versions of C++ redistributables going back to Visual Studio 2005. All of these redistributables are also regularly maintained with Windows Update.

Finally, Windows SDK versions targeting various versions of Windows are available for all supported Visual Studio developer environments. The oldest currently available in Visual Studio 2022 is Windows XP SP3[1].

These all serve to thoroughly solve both combinations of backward-forward compatibility, where (a) the runtime environment is newer than the developer environment, and (b), the runtime environment is older than the developer environment.

It is perfectly possible to compile a single `.exe` on Visual Studio 2022 in Windows 11, and expect it to run on Windows XP SP3, and the vice versa: compile a single `.exe` on Visual Studio 6, and expect it to run on Windows 11. No dynamic libraries, no DLLs, nothing; just a naked `.exe`. Download from website, double-click to run. That's it. No git clone, no GNU Autotools, no configure, no make, no make install (and then f*cking with rpaths), nothing. Prioritising binary-only software distribution means Windows has prioritised the end-user experience.

> It's where the phrase "DLL hell" originated, after all.

This is also incorrect. 'DLL hell' is a pre-NT problem that originated when packagers decided it was a good idea to overwrite system binaries by installing their own versions into system directories[2]. Sure, the versioning problem was there too, but this is itself a result of the aforementioned DLL stomping.

[1]: https://learn.microsoft.com/en-us/cpp/build/configuring-prog... [2]: https://en.wikipedia.org/wiki/DLL_Hell#DLL_stomping

I fully daresay writing C and C++ for Windows is an easier matter than targeting any Unix-like. For context, see what video game developers and Valve have done to check off the 'works on Linux' checkbox: glibc updates are so ridiculously painful that Valve resorted to simply patching WINE and releasing it as Proton, and WINE is the ABI target for video games on Linux.

That Windows is generally better at handling compatibility is one thing, but I'm curious about the following part:

> There's a good reason for this, which IMO Unix-like compilers and system libraries should start adopting, too.

Why? I'm understanding from your comment that you can't mix debug and release objects on Windows because the ABI is different. That's not a feature, that's a limitation. If it works on Linux to mix debug-enabled objects with "release", what use would it have to make it not work anymore?

IIUC debug symbols can be totally separated from the object code, such that you can debug the release if you download the debug symbols. A well configured GDB on distros that offer this feature is able to do it automatically for you. It seems very useful and elegant. Why can't Windows do something like this and how is it an advantage?

(Genuine question, I have a remote idea on how ELF works (wrote a toy linker), not much how DWARF works, and not the slightest idea on how all this stuff works on Windows)

> If it works on Linux to mix debug-enabled objects with "release", what use would it have to make it not work anymore?

There is no difference between Linux and Windows here. The debug/release issue is ultimately up to the API developer.

C++ has has the standard template library (STL). libstdc++, libc++, and MSVC STL are three different implementations. STL defines various iterators. A common choice is for a release-mode iterator to be a raw pointer, just 8 bytes on 64-bit. But the debug-mode iterator is a struct with some extra information for runtime validation, so it's 24 bytes!

The end result is that if you pass an iterator to a function that iterator is effectively two completely different types with different memory layouts on debug and release. This is a common issue with C++. Less so with C. But it's not a platform choice per se.

> IUC debug symbols can be totally separated from the object code, such that you can debug the release if you download the debug symbols. A well configured GDB on distros that offer this feature is able to do it automatically for you. It seems very useful and elegant. Why can't Windows do something like this and how is it an advantage?

MSVC always generates separate .pdb files for debug symbols. Windows tooling has spectacular tooling support for symbol servers (download symbols) and source indexing (download source code). It's great.

The difference is that on Linux, "compile my program in debug mode" does not enable libstdc++'s expensive (and incompatible) mode.

> The difference is that on Linux, "compile my program in debug mode"

"Linux" does not have a "compile my program in debug mode" magic toggle (or Release or whatever for what it's worth). Different IDEs and toolchains may have different defaults and expectations. "g++ -g" is not debug mode, it's debug symbols.

My point is the real difference is the implementation of the three libraries. It's not a fundamental platform difference.

The expectation that debug libraries are are compiled with ABI changing flags is, while not fundamental, an significant platform difference.

Historically this might be because MSVC didn't even preserve the standard library ABI across versions (although it has done so for the last 10 years at least), so there were little ABI stability expectations.

> If it works on Linux to mix debug-enabled objects with "release"

it definitely does not. MSVC's debug mode is akin to for instance using libstdc++ with -D_GLIBCXX_DEBUG which does change the ABI. Just passing "-g" which enable debug symbols is very different from what Microsoft calls Debug mode, which adds very extensive checks at all levels of the standard library (for instance, iterators become fat objects which track provenance, algorithms check preconditions such as "the input data is sorted", etc.)

Note tough that libstdc++ has an ABI compatible debug mode that still adds a significant amount debug checks (and it is meant for production deployment).

Yes, I wonder that too. The comment says that debug and release builds have ABIs because they have different type layouts. But why do they have different type layouts? Bounds checking and assertions shouldn’t change the type layout. It seems to me that debug flags should generally only modify code generation & asserts. This is usually the case on Linux, and it’s extremely convenient.

If windows is going to insist on different libraries in debug and release mode, I wish the development version of the library bundled debug and release builds together so I could just say “link with library X” and the compiler, linker and runtime would just figure it out. (Like framework bundles on the Mac). Windows could start by having a standard for library file naming - foo.obj/dll for release and foo-debug.obj/dll for debug builds or something. Then make the compiler smart enough to pick the right file automatically.

Seriously. It’s 2024. We know how to make good compiler tooling (look at go, Swift, rust, etc). There’s no sane reason that C++ has to be so unbelievably complex and horrible to work with.

Windows debug builds add extra sanity checks for which it needs extra members in types. For instance, a vector<T>::iterator is just a T* in a regular build, but in a debug build it also keeps a pointer to the vector so it can check bounds on every access.

But yes, C++ punts a lot of things to the build system, partly because the standard has to work on embedded systems where shared libraries don’t exist. A better build system could fix most of these things, but every build system that tries ends up massively complicated and slow, like Bazel.

But fixing it in Windows doesn’t mean they also need to fix it in embedded systems. Microsoft is in control of their whole ecosystem from the kernel to userland to visual studio. Microsoft could make C++ on windows sane without anyone else’s permission. Their failure to do that is on them and them alone.

I think browser vendors have the right idea when it comes to evolving standards. Vendors experiment using their own products and then come together and try and standardise their work at committee. I think that would be a much better idea than either doing nothing or, as you say, trying to boil the ocean.

> Microsoft could make C++ on windows sane without anyone else’s permission.

C++ on Windows is perfectly OK, especially if you're building on Windows for Windows using MSVC or Clang-cl.

> Bounds checking and assertions shouldn’t change the type layout.

Any bounds checks and assertions that rely on storing additional data such as valid iterator ranges or mutation counters would need to change the type layout, wouldn't they?

Even if the STL were purely a header-only library (and influenced only by code generation changes for debug builds), there's still the problem of ABI compatibility across different translation units--or different libraries--which might be built with different options.

EDIT: One of your sibling comments goes into greater detail!

> I wish the development version of the library bundled debug and release builds together

Almost all do. Look for binary library releases; they almost always supply Debug and Release binaries together.

> Windows could start by having a standard for library file naming - foo.obj/dll for release and foo-debug.obj/dll for debug builds or something.

Funnily enough, it does. foo.lib for Release; food.lib for Debug.

There's a bit of a conflation here; partially my fault. Allow me to clarify...

To generate debug symbols for a given binary (whether executable or library) on Windows and MSVC's cl.exe (and Clang on Windows), compile with `/DEBUG`[1] and one of `/Z7`, `/Zi`, or `/ZI`[2]. This is equivalent to `-g` on Linux gcc/clang. In particular, `/Z7` generates separate `.pdb` files, which contain debug symbols for the binary in question.

The options that the parent commenter and I were discussing, i.e. `/MD`, `/MDd`, /MT`, and `/MTd`[3] have to do with the C and C++ runtime link configuration. These correspond to multithreaded dynamic, multithreaded dynamic debug, multithreaded static, and multithreaded static debug respectively. Therefore, the small `d` refers to debug versions of the C and C++ runtimes. The differences between the debug and release versions of the C and C++ runtimes are listed in the following links[4][5][6][7][8]. The last link in particular demonstrates the debug CRT's functionality.

Conventionally on Windows, debug binaries are linked to the debug versions of the C and C++ runtimes; ergo the requirement that 'Release and Debug binaries on Windows cannot be combined'. This convention is respected by all maintainers who release binary libraries on Windows.

There is no equivalent on Unix-likes: it'd be like having 'debug' versions of libc.so.6/libstdc++.so/libc++.so/libpthread.so with different ABIs. If you wanted to change between release/debug here, you would have to at least re-link (if not re-compile) everything. Imagine having `-cstdlib=libc-debug` and `stdlib=libc++-debug` options.

Both sets of options (debug symbol options and C runtime link options) are orthogonal, and may be freely combined. Hence, it is perfectly possible to link the debug versions of the C and C++ runtimes to a 'release' executable, although it would be pretty weird. For instance, `/O2 /LTCG /arch:AVX2 /MTd`. Equivalent imaginary GNU-style command: `-O3 -flto=thin -march=x86-64-v3 -cstdlib=libc-debug stdlib=libc++-debug -static`. You can see what I mean, I hope.

[1]: https://learn.microsoft.com/en-gb/cpp/build/reference/debug-...

[2]: https://learn.microsoft.com/en-gb/cpp/build/reference/z7-zi-...

[3]: https://learn.microsoft.com/en-gb/cpp/build/reference/md-mt-...

[4]: https://learn.microsoft.com/en-gb/cpp/c-runtime-library/c-ru...

[5]: https://learn.microsoft.com/en-gb/cpp/c-runtime-library/crt-...

[6]: https://learn.microsoft.com/en-gb/cpp/c-runtime-library/debu...

[7]: https://learn.microsoft.com/en-gb/cpp/c-runtime-library/run-...

[8]: https://learn.microsoft.com/en-gb/cpp/c-runtime-library/crt-...

> ...having 'debug' versions of libc.so.6/libstdc++.so/libc++.so/libpthread.so with different ABIs

would be actually cool

I think the main reason they don't offer a `--make-my-binary-compatible-with-the-ancient-linux-versions-users-often-have` is that GCC/glibc is a GNU project and the are philosophically against distributing software as binaries.

I don't think there's any technical reason why it couldn't be done.

To be fair to them though, Mac has the same problem. I worked at a company where we had to keep old Mac machines to produce compatible binaries, and Apple makes it hard to even download old versions of MacOS and Xcode.

I guess the difference is MacOS is easy to upgrade so you don't have to support versions from 13 years ago or whatever like you do with glibc.

I used to think that binary compatibility benefits proprietary applications, but I'm not so sure anymore. From a commercial perspective, when we break binary compatibility (not that we want to), it's an opportunity for selling more stuff.

Many distributions do periodic mass rebuilds anyway and do not need that much long-term ABI compatibility. Binary compatibility seems mostly for people who compile their own software, but have not automated that and therefore couldn't keep up with updates if there wasn't ABI compatibility.

I agree. It's annoying for closed source apps but they generally have the resources to deal with it anyway. E.g. with Questa I can just unzip it and run it. No trouble.

It's disproportionately annoying for open source projects who don't want to waste their time dealing with this.

> I think the main reason they don't offer a `--make-my-binary-compatible-with-the-ancient-linux-versions-users-often-have` is that GCC/glibc is a GNU project and the are philosophically against distributing software as binaries.

You don't have to statically compile glibc, gcc just needs an option to tell the compiler to target say, version 2.14 instead of the latest one.

The newest glibc has all the older versions in it. That's why you can compile on say ubuntu 14 and have it run on ubuntu 24.

No like, the point is that the only reason you (and I: I do this all the time, including with my open source software... like: no judgment) want to target some old version of glibc is so you can distribute that binary to people without caring as much about what version of the OS they have; but that would be unnecessary if you just gave them the source code and have them compile their own copy for their system targeting the exact libraries they have.

Unfortunately most people don't want to bother compiling, myself included. I tried gentoo one time and it took 1 hour to compile 5 minutes worth of apt-get on ubuntu.

Only the dynamically linked bits, the statically linked startup code and libc_nonshared.a are missing from newer versions. Most programs don't need them (who needs working ELF constructors in the main program?). The libc_nonshared.a bits can be reimplemented from scratch easily enough (but we should switch them over to header-only implementations eventually).

Yes exactly. That's what the flag would do. But it doesn't exist.

> They are a problem because gcc automatically links to the latest version of glibc. As to why they don't add an option to specify an older version?

Because glibc and ld/lld are badly designed. glibc is stuck in the 80s with awful and unnecessary automagic configure steps. ld/lld expect a full and complete shared library to exist when compiling even though it expects a different shared library to exist in the future.

Zig solves the glibc linking issue. You can trivially target any old version for any supported target platform. The only thing you actually need are headers and a thin, implementation free lib that contains stub functions. Unfortunately glibc is not architected to make this trivial. But this is just because glibc is stuck with decades of historic cruft, not because it's actually a hard problem.

> awful and unnecessary automagic configure steps

Steps taken when? When building glibc? And - what steps?

> ld/lld expect a full and complete shared library to exist when compiling ...

But ld and lld are linkers...

> Zig solves the glibc linking issue.

But Zig is a language. Do you mean the Zig standard library? The Zig compiler?

> The only thing you actually need are headers and a thin, implementation free lib that contains stub functions.

Why do you need stub functions at all, if you're not actually using them?

The zig compiler can compile C and C++, using llvm, and it also packages various libc implementations, including glibc, musl, mingw, and msvc, and more for other OSes. Some people use it as a more convenient golang-like cross-compiler. And this whole combination is a smaller download and install than most other toolchains out there. It just took some inspiration and grunt work, to dissect the necessary (processed) headers and other bits of each libc ... hats of to the zig devs.

Random blog post I just found about it: https://ruoyusun.com/2022/02/27/zig-cc.html

Linking to the latest version of glibc is, in itself, not a problem -- glibc hasn't bumped its soname in ages, it is using symbol versioning instead. So you only get a problem if you use a symbol that doesn't exist in older glibc (i.e., some specific interface that you are using changed).

As for using an older version of glibc, _linking_ isn't the problem -- swapping out the header files would be. You can probably install an old version of the header files somewhere else and just -I that directory, but I've never tried. libstdc++ would probably be harder, if you're in C++ land.

Recent libstdc++ has a _dl_find_object@GLIBC_2.35 dependency, so it's not exactly trivial anymore to link a C++ program against a older, side-installed glibc version because it won't have that symbol. It's possible to work around that (link against a stub that has _dl_find_object@GLIBC_2.35 as a compat symbol, so that libstdc++ isn't rejected), but linking statically is much more difficult because libstc++.a (actually, libgcc_eh.a) does not have the old code anymore that _dl_find_object replaces (once GCC is built against a glibc version that has _dl_find_object).

This applies to other libraries as well because there are new(ish) math functions, strlcpy, posix_spawn extensions etc. that seem to be quite widely used already.

I find it's best to treat this as a case of "cross-compilation to old glibc version". That is, you don't want to just link against an old glibc version, you want a full cross-compilation toolchain where the compiler does not pick up headers from `/usr/include` by default (but instead has its own copy of the system headers), and where libstdc++ is also linked against that old glibc version.

We used https://crosstool-ng.github.io/ to create such a "cross-compiler". Now it doesn't matter which distribution our developers use, the same source code will always turn into the same binary (reproducible builds, yay!) and those binaries will work on older distributions than the developers are using. This allows us to ship dynamically linked linux executables; our linux customers can just unzip + run, same as our windows customers, no need to mess with docker containers.

The downside is that we can't just use a library by `apt install`ing it, everything needs to be built with the cross compilation toolchain.

If you're building Rust, check out cargo zigbuild (yes, zigbuild) which lets you target old glibc.

chroot has existed for many years.

The problem is actually: nobody adhering to semantic versioning.

Hence the reason why there is a separate API/ABI level embedded into a shared object file, which is pretty much always completely different from the library's package version because they broke compatibility so often.

Additionally there's lots of libraries that break their symbols and hash tables on every subminor update. (Usually when they use LLVM, but I'm not saying it's LLVM's fault)

For example, harfbuzz and libicu are a regular offender of this, meaning they fuck up every downstream project because the method signatures contain a randomized chunk which changes on every single subminor version.

libicu is deliberately doing this because one of its purpose is to offer Unicode ordering. If consumers try the new ordering on non-rebuild objects, it could lead to data loss. That's what happened with glibc 2.28. [0]

If postgres prefers explicit library versions to prevent data corruption, then icu is doing things correctly and making sure that downstream consumers don't shoot themselves in the foot.

[0]: https://wiki.postgresql.org/wiki/Locale_data_changes

Honestly, the more I was researching about what happened in that glibc version update, the more I thought "self inflicted".

If you're using NULL terminated strings everywhere, and not UTF8/16, it's your own fault for doing so. Especially if you are building a database which should always contain a length of bytes or at least a CRC check before each entry's contents in its serialized file format to prevent exactly this from happening.

(Let alone the question why is there no schema version that should have prevented this from happening in the first place)

You don't have to update, it's just that developers seem to like to run the latest stuff, and many appear to build production binaries on their laptops (rather than in a controlled build environment, where dependencies are tightly managed, and you could deliberately stick to an older distribution easily enough).

The dependency errors indicate real issues because of the way most distributions handle backwards compatibility: you have to build on the oldest version you want to support. Those errors happen if this rule is violated. For glibc-based systems, the effect is amplified because package managers have become quite good at modeling glibc run-time requirements in the package-level dependencies, and mismatches result in install-time dependency errors. Admittedly, I'm biased, but I strongly suspect that if we magically waved away the glibc dependency issues, most applications still wouldn't work because they depend on other distribution components, something that's just not visible today.

Because:

* Developers like to be on the latest-and-greatest distro, and rarely perform builds in a chroot. Sometimes this is sheer apathy; other times it is because backporting library dependencies is annoying.

* End-users can't even be assumed to be on the latest LTS.

Related, almost nobody understands `rpath` and why it should always be used instead of static linking (assuming normal dynamic linking doesn't work of course). There's some common FUD going around but unless you're a setuid or similar program it's not actually true (and even then, it's only an issue with some uses of rpath).

> Solver3 is a fully backtracking dependency solving algorithm that defers choices to as late as possible. It starts with an empty set of packages, then adds the manually installed packages, and then installs packages automatically as necessary to satisfy the dependencies. [...]

> If you have studied SAT solver design, you’ll find that essentially this is a DPLL solver without pure literal elimination

DPLL algorithm: https://en.wikipedia.org/wiki/DPLL_algorithm

prefix-dev/pixi is a successor to Mamba and Conda which wraps libsolv like dnf. https://github.com/prefix-dev/pixi

OpenSUSE/libsolv: https://github.com/openSUSE/libsolv

OpenSUSE/zypper may have wrapped or may still wrap libsolv?

TIL libsolv also supports arch .apk and .deb; there's a deb2solv utility to create libsolv .solv files: https://github.com/openSUSE/libsolv/blob/master/tools/deb2so...

"[mamba-org/rattler:] A new memory-safe SAT solver for package management in Rust (port of libsolv)" (2024) https://prefix.dev/blog/the_new_rattler_resolver https://news.ycombinator.com/item?id=37101862

Pixi is integrating uv: https://github.com/astral-sh/uv/issues/1572#issuecomment-194...

astral-sh/uv [1] solves versions with PubGrub [2], which was written for Dart: [1] https://github.com/astral-sh/uv [2] https://github.com/pubgrub-rs/pubgrub

uv's benchmarks are run with hyperfine, a CLI benchmarking tool: https://github.com/sharkdp/hyperfine

astral-sh/rye was mitsuhiko/rye, and rye wraps uv with "fallback to unearth [resolvelib] and pip-tools": https://github.com/astral-sh/rye https://lucumr.pocoo.org/2024/2/15/rye-grows-with-uv/

So, TIL it looks like rye for python and pixi for conda (python, r, rust, go,) are the latest tools for python packaging, not apt packaging.

fpm is one way to create deb packages from python virtualenvs for install with the new APT solver, which resulted in this research.

From the guy that brought you the keepassxc downgrade.

https://github.com/keepassxreboot/keepassxc/issues/10725

Wait so this guy decided to fork the project, and seemingly abused his position to supplant the previous version with his own opinionated fork? All this while disregarding the opinion of upstream devs themselves and being arrogant and stubborn in his replies.

What an spectacular way to break things for end users.

If there's one thing to learn and apply from Linus, IMHO, is his attitude about NEVER breaking userspace in the Kernel. This lesson can be adapted to most software, and we really should strive to more of it, not less (obviously adjusting to each case; here, replace "userspace" with "user setups")

First, this is not in Debian stable (yet). Issues like this is one reason why Debian testing/unstable exist. Secondly, it is common and expected for distributions to select the feature flags they deem appropriate. It is not a fork. The mistake here was not to provide a compatible upgrade option in addition to the new default.

When I found out that it was just in unstable my position flipped on it. The guy has no business being as crass as he is in the comments section of the GH issue but lack of tact from an opinionated software dev in internet discourse is nothing new. And the point behind the petty of switching to package/package-full, which is pretty standard in the Debian world, is perfectly reasonable so long as it happens on a major version bump.

I don't disagree that it's annoying to the upstream devs but c'est la vie when you have a bunch of 3rd parties repackaging your code. They won't be the first to have a "uninstall your distro's package and install the upstream version before reporting a bug" in their issue template.

I never thought I would say this but: I am a pretty recent flatpak convert. This thing just made me more convinced about how it is a good thing to let the Devs deliver the apps themselves and let the distro people do the distro stuff.

After accepting flatpak I started using fedora silverblue and then switched to opensuse aeon and I have been very happy. The only pain point was getting Emacs working properly.

What downgrade are you referring to?

https://github.com/keepassxreboot/keepassxc/issues/10725

Thanks. I love opensource drama.

Seems like another Pottering in making - my way or highway developer.

After reading so many GPT news item, I was confused what APT is, no explanation on the page, no link ... took me a while to realise it is about debian linux' packaging tool apt.

That’s like saying you are confused because an article said Linux and it didn’t explain what it was…

I think install time dependency resolution was a bad idea and needs to be removed entirely. Packages for package managers that do it are untestable by design, because their runtime behavior cannot be known at build time.

"Functional package management", as implemented in nix and guix, is a much saner approach.

The thing is, Debian's solvers are always deterministic. Given from a known state, it always ends int the same state. This allows getting/setting selection lists with "dselect", and using apt for system automation feasible.

In my last decade (or 15 years to be exact) with Debian, I never suffered from non-deterministic behavior or met with surprises from apt.

The solver might be deterministic if you take some sufficiently large global state into account, but the process of

    apt install bar

My point is that a package "bar v3.2.1" that depends on "libfoo v1.2.3" needs to be considered a different package than "bar v3.2.1" that depends on "libfoo v1.2.4" or even "libfoo v1.2.3 compiled with -O3 instead of -O2", because the resulting package can behave differently due to the variation.

But as soon as you have that there is no need for install time dependency resolution anymore, since it is all done at build time.

It might get better now. Packages often have dependency loops, so that "A depends on B depends on C depends on A". Autodependencies in a loop like that would never get uninstalled.

One of the more obscure outcomes of that is if you later installed something else that had one of those autodepends as a less favoured alternative, it would be used regardless - ie instead of what the package you were installing said it preferred.

I'm guessing this fixes that particular wart. By starting from a blank slate every time it effectively ignores those autodepend loops, so they will get uninstalled.

Hallelujah.

I wish Debian/APT had first-class support for having multiple versions of the same library.

On a large system it's inevitable that packages will be upgrading at different rates, and a bit of duplication can be a way out of complex system-wide dependency chains held by the most outdated package.

Despite lack of tooling support, Debian still does not completely avoid duplicated packages. They merely manually move the major version number to the package name (foo v10.0 and foo11 v11.0), causing package names to vary between distros and Debian versions. This seems like the worst solution, since APT can't automatically do it where it would be useful, and renamed packages complicate use of tools like pkg-config, ansible, and providing install instructions to users.

That is an orthogonal issue. Guix has first-class support for versions (where "version" means the version string of the source code), while nixpkgs uses the same naming-based approach as Debian.

The important part is being able to install two packages in isolation, irrespective of if they represent the same software in different versions or if they are completely unrelated. Apt couldn't install two unrelated packages if they contain files that have the same path, to my knowledge.

As soon as one foregos putting everything into a single filesystem structure (like in FHS, which most distros follow to some extent) the logical conclusion will be to prefix each packages directory with some identifier that meaningfully describes its content (which could be considered an "extended version" of this package). With nix this is a hash of all the inputs that went into the package (its own source code, compiler flags, build settings, dependencies, etc. and recursively the same for all of its dependencies). This means that if the compiler flags of e.g. a five levels deep transitive dependency are changed, then the package that depends on it changes as well, which is only reasonable since its behavior could change.

At that point all dependency resolution has happened at build time and none is required at runtime. Being able to install different versions of the same package is merely a by-product of this approach.

One server = one feature = one library version

That's the golden rule for easy upgrades, sane dependencies managements (both technicals and organisationals)

There is no good reason to not spawn hundreds or thousands of VMs : just do it, make your life easier and stop bothering with a whole bunch of issues

You may "booh" me. But in the end, you know I'm right : who won, the 80' era supercomputer, or the k8s/serverless world ? Yeah

> The most striking difference to the classic APT solver is that solver3 always keeps manually installed packages around, it never offers to remove them.

Y'know, when you put it like that it does seem rather obvious, doesn't it? Principle of least surprise and all that.

Edit: To be clear, not knocking anyone for it; hindsight is 20/20 and I've never written a dependency solving algorithm so I'm in no position to judge.

It’s not that clearcut, because sometimes you forget all the packages you installed manually over the years and aren’t using anymore, and it can make sense to be offered to remove one when it helps resolving a conflict with a newer package that you actually want to use now.

The healthy choice is to rebuild rather than upgrade any long-lived OS installation (e.g. your desktop) on distro release.

Corollary: make it easy on yourself: mount /home separately (and maybe /var too), and keep /etc in scm.

I’ve been upgrading Debian in-place for almost two decades now and prefer that approach. Security updates are automated (daily), major-version upgrades have almost no downtime, and only very rarely is there a critical configuration that needs to be adjusted.

The installed packages can be listed with `dpkg --get-selections`, and that list can be replayed should it be necessary to recreate the installation, plus a backup of /etc/. But I never had to do this, Debian just works.

I was previously a maintainer of certain Debian packages, and of similar vintage, so this advice comes with the extra salt of having seen how the sausage is made. I shudder to think how many abandoned files, orphan packages, and obsolete/bad-practice configurations might be lurking in a system that has only been release-upgraded for decades. Yes, no doubt it functions. By the same token, people can live in their own filth. Should they? I choose not to.

That said, I may do a speculative dist-upgrade on a snapshot to reveal & prepare for conflicted conffiles in advance, but I'll throw that away, I won't rely on the merged result across a release upgrade.

But when there's a configuration change dpkg-configure asks you if you want the upstream one, no?

Debian releases are somewhat slower than Ubuntu or similar - given infinite time, esoteric configurations will break on update due to some edge case 4 dist-upgrades ago.

> The healthy choice is to rebuild rather than upgrade any long-lived OS installation (e.g. your desktop) on distro release.

I wish there was more rigor (and testing) with this sort of thing. Generally systems should have the invariant that “install old” + “upgrade” == “install new”. This property can be fuzz tested with a bit of work - just make an automated distribution installer, install random sets of packages (or all of them), upgrade and see if the files all match.

/etc makes this harder, since you’d want to migrate old configuration files rather than resetting configuration to the defaults. But I feel like this should be way more reliable in general. And people want that reliability - if the popularity of docker and nix are anything to go by.

Alas that many packages will not upgrade conffiles gracefully when their structure/semantics change, and many are beyond the ability of any package manager's built-in diffing tools to apprehend. That is why I place /etc under scm for long-lived hosts, because what happens next instead looks an awful lot like doing a conflicted three-way code merge following a breaking-change vendor release. The greatest OMFGFFS in this realm, by far, came with the big lurch over to systemd.

related: https://etckeeper.branchable.com/

> The greatest OMFGFFS in this realm, by far, came with the big lurch over to systemd.

I think by the time I started with it, NixOS was already on systems, but reportedly that transition went off without a hitch. Before the transition, NixOS configs were in charge of generating OpenRC scripts or whatever, then afterwards the same configs generated systemd units instead. When the system has total control over the config files like that you get to bypass certain challenges!

You might notice that the old announcement for that feature includes no special care other than the requisite reboot to change init systems: https://web.archive.org/web/20200423143059/https://releases....

This is a very contentious topic, you can see my initial take on this in

https://blog.jak-linux.org/2023/10/10/a-case-for-different-u...

The problem is that's not really true. i.e. consider you get postfixed pulled in by postfix | mail-transport-agent and the new distro release changes it to exim4 | mail-transport-agent as we picked a new default MTA.

Does this mean we should, by default, replace your MTA and require you to have to configure a different one?

You will be able to do this (pick the same automatically installed packages given the same set of manually installed ones; installing missing Recommends, etc, by using --fix-policy and some more options perhaps).

As for configuration this is worse, files managed by ucf have 3-way merging but it's not automatic and most configuration isn't managed by ucf but by dpkg.

In theory yes, but this hits with the fact that defaults change.

For example a new system won't have pulseaudio, but it won't be removed and replaced automatically because that would be potentially disruptive to existing users.

The even healthier choice is to store all your system state in a config, you can then just occasionally clean up the config. This also lets you put extensive comments on why a package is needed or even create a submodule for a specific project.

Shame that the only bistro that lets you do this is extremely arcane and difficult to learn if you do not have a PhD in mathematics.

Is the mapping from config -> state reproducible? Traditional distros drift over time with lots of upgrades because the state is not just a pure function of the set of packages to be installed (the "config"). It could depend on the system's history, such what packages were installed or removed in the past.

The so-called "atomic" distros don't attempt to make the (config -> state) transformation reproducible but rather sidestep the problem by snapshotting and replicating the output of the transformation -- the files on disk after various installs, upgrades, and config changes.

> Shame that the only distro that lets you do this is extremely arcane and difficult to learn if you do not have a PhD in mathematics.

Hey, there are two! Remember GuixSD

Guix is not that difficult.

I don't think NixOS is either, but I figured I'd let a shitposter have their hyperbole. :)

I don't begrudge any Guix contributors protesting that the GuixSD docs are undersold by such hyperbole, though! If that's you, carry on. :D

Flakey Windows taught me this doctrine, and it has carried over to any OS that I use (mostly Linux). A directory with 'portable apps' was essential in Windows, fortunately apt makes this a whole lot simpler :)

1. Find ways to automate setting up my env. Keep that instruction current. apt and chocolatey/ninite have made this far simpler than it used to be. 2. Keep data in directories that are auto-backupped. Concretely I have everything in 'shares', and these shares have survived various changes in tooling, buut their location and principle have stayed the same. Also helps organizing your files :) 3. Have short but precise instructions on how to setup new envs, wherever automation isn't possible or impractical.

Has worked for me for decades now, and it means I'm good to go within an hour of (re)installing a fresh OS.

What do you use to "keep /etc in scm"?

etckeeper was the standard tool last time I really ran the kind of distro that benefits from it

Thanks for the recommendation! :)

This is one of my favourite things about Fedora Silverblue and rpm-ostree. If I run `rpm-ostree status`, I get a list of all the packages I’ve installed or removed compared to the base image. Makes it really easy to keep track of what’s different in my system compared to the current default.

I don’t know if other distributions do this, but I quite like gentoo’s “world” file. Its a text file listing all the manually installed packages. Everything in the list is pinned in the dependency tree, and kept up to date when you “emerge update”. I constantly install random stuff for random projects then forget about it. The world list lets me easily do spring cleaning. I’ll scroll through the list, delete everything I don’t recognise and let the package manager auto remove unused stuff.

I think nix has something similar. I wish Debian/ubuntu had something like that - maybe it does - but I’ve never figured it out.

I think OSes really should take inspiration from the newest generation of programming languages here. Rust and Julia (as well as a few others) do a lot better than the OS. Features like environments (e.g. for installing per user packages), and a clean separation of a project and an environment would be very nice.

I hesitate to say it because it comes up in approximately every single conversation about package management, but... NixOS does all of those things. Whole list of packages in configuration.nix, whole of /etc encoded into configuration.nix, flakes let you pin everything, (therefore) it's trivial to check the whole OS config (packages, versions, configurations) into version control, home manager lets you do all this per-user (or you can do it in the system-wide configuration.nix if you really want), and you can drop a flake.nix+flake.lock in any project directory and scope things that way.

Debian has debfoster.

Alpine has a similar /etc/apk/world file. Further, you can edit this file and then `apk fix` will update the system to reflect any changes.

apt-mark showmanual shows you all manually installed packages and apt-mark auto marks stuff as automatically installed.

The list of manually installed packages is effectively the "world".

The new solver takes inspiration from apk in that it marks the world/manually installed packages the root of the dependency problem (hence why they can't be removed by the solver itself, only manually).

On Debian-derived systems, the list can be found with:

  aptitude search '?installed ?not(?automatic)'

There's probably a way without aptitude, but using Debian without aptitude is like riding a bicycle without gears.

Yeah, I've hit that exact situation on one of my machines running Void Linux (so obviously completely different package manager). I personally think it's best to tell the user that it can't make the given constraints work and suggest which package(s) are the problem, and then if the user is willing to part with them then they can `apt remove foo && apt upgrade` or w/e (which is more or less my experience of how Void's xbps does it).

I love when this happens when I find a new product, I think to myself would be even better if they did this really simple tweak. Then a week or two later they do the same exact change.

Apt probably moves a little slower on “big” changes, since theres many implications. I cant even imagine the number of obscure scripts that something like this breaks because the output and behavior is slightly off somehow.