Managing Linux GLIBC6 version dependencies

Releasing binaries on Linux can be a fraught process; library version dependencies can make it difficult to predict when a system will be able to run your software.

The problem arises primarily from GLIBC6 - the GNU C Library that underpins all programs compiled with GCC or G++. In theory it was once possible to statically link this library, but that hasn't been officially supported for a long time - and even if you try, with a complex enough program (such as one using X11 graphical components), other statically linked libraries will pull in parts of GLIBC6 dynamically anyway, almost guaranteeing that your program will break on any system with a different version of the library.

GLIBC6 provides internal symbol versioning, which sounds like a great idea in theory - if your program uses older symbols, any user's system that has a newer GLIBC6 will still provide those older symbols for it to access. The problem is that this depends on the user's system being more up to date than the system the binary was built on. With our games, this is not often the case - many user systems use "stable" distributions with much older versions of GLIBC6 than our cutting edge build machines. It's easy for our programs to end up depending on symbols simply not available in these older versions of GLIBC6. So if we do nothing to control our symbol version dependencies, we find our games just don't run on many stable Linux distributions.

Common solutions

So how do most developers work around this? There are a few common ways of avoiding the problem, but as we'll soon see, none is particularly satisfactory.

Older GCC & G++

GLIBC6 symbols are chosen by the compiler. The simplest solution is to use an older version of GCC that won't include any symbols from newer versions of GLIBC6. This is generally pretty fool-proof, and why you still see a lot of software being built with GCC 4.9 in the year 2017, which had already been superceded two years ago.

The problem with this is obvious; you're forced to use an older compiler which lacks many modern features added in the past several years; support for the latest C++14 and C++17 standards, support for std::experimental features, new optimisation flags, etc, are all missing or severely impaired in comparison to the latest compiler versions. Your code may be less optimised, and you're forced to limit yourself to avoiding the latest language features and improvements. That's no good.

Avoid new features

Many programs will not cause GLIBC6 version problems simply because they don't use newer library features. Simply avoiding some library calls can reduce the required version to something acceptable on the target system. Of course, you still have to analyse the program to see which GLIBC6 features create a dependency on a newer version than you want to require.

One of the problems with this is that you're still limiting yourself to entirely avoiding certain features - sometimes arbitrarily chosen ones, such as quick_exit, for which older GLIBC6 versions do exist. And other dependencies may be brought in by compiler optimisations automatically...

Turn off optimisation

One example of GLIBC6 dependencies brought in automatically by GCC / G++ optimisations is vectorisation using OpenMP and LibMVEC. These manifest as symbols requiring GLIBC6 2.22 looking like _ZGVbN4vv_powf, _ZGVdN8v_logf, _ZGVeN16v_expf etc. Turning off all optimisations can get rid of dependencies for these symbols, but of course your game will then run a lot slower - not an acceptable solution!

Dedicated build environment

Some developers like to use a full virtual machine or a dedicated Docker instance for building release versions on Linux. This has the advantage of allowing you to control all library versions the build environment has, regardless of who is deploying it and where, and ensuring you don't bring in any dependencies newer than you require without having to micro-manage the library versions.

The simplicity of the process is appealing - if you want to guarantee your game runs everywhere that has Ubuntu 14.04, for example, you just create a virtual machine with this exact version, don't bring in any newer packages, and build your game on that.

However, you have to use the older GCC this specific old distro provides, often older even than you really need to limit yourself to the GLIBC6 versions you want to support. This has all the disadvantages of the "older GCC version" solution - and of course you have all the hassle of building in a virtual machine, on top of that. Your builds are slower and take longer, you can no longer integrate your build environment as tightly with your IDE for rapid iteration, and the end result is still limited by an older compiler. Fine if simplicity is all you want, but the cost in developer performance (due to loss of modern features and slower iteration time) can be too great.

Our solution

So how do we manage to build games using the latest GCC and G++ (6.3.0 at the time of writing this post) and still let our games run on all the stable distributions, which only ship with GCC 4.9? The trick is to take an intelligent, multi-faceted approach.

Decide what version to support

At the time of writing, all Debian-based stable distributions and other major distributions we are aware of all support at least GCC 4.9. This corresponds to GLIBC6 2.19 - so this is the version we're choosing to target. We now know that any binary that requires symbols from versions newer than this won't work on all our target platforms, so we have a specific number to aim at.

Detect which symbols need which version

Before you can do anything to alter version dependencies, you need to find out exactly which symbols require what versions. It's possible to search for "Version References" in the output of objdump -pC my_binary, and filter for GLIBC_ - you immediately see what symbols in your program require which version.

You can wrap this in a nice tidy script to iterate through all such symbols, find the ones which require a version higher than your target version of 2.19 (or whatever you decided on earlier), and neatly output the results. It's even possible to then inspect the linkable object files individually for use of each symbol from a newer version than you want to support, and report in a user-friendly way exactly which source files bring in those symbols, and inform the user:

Explicit symbol versioning

As our script above advises, once we know exactly which symbols are causing a problem, it's then possible to mitigate this requirement by explicitly specifying the version of the symbol we want to use. The trick is to add __asm__(".symver symbol_name,symbol_name@GLIBC_VERSION"); just prior to the call that brings in the symbol, where symbol_name is the GLIBC6 symbol causing you problems, and VERSION is the version of the symbol you want to use rather than the default latest:

How do we know what version to request? That's easy; use objdump -TC /lib/x86_64-linux-gnu/ or whatever the path is of your GLIBC6 dynamic library, and find the symbol you're interested in - you'll see all available versions of that symbol. Simply specify the latest version that's lower or equal to your target version of 2.19 (or whatever you chose to support). You'll notice the script above does this automatically and suggests exactly the snippet of code that the user needs to paste in. Once this is done, this is the result - we've dropped the highest GLIBC6 dependency in our binary to 2.17, without having to switch to an older compiler or avoid using any language or library features!

Disable just the optimisations you can't support

This works very well for symbols called directly from your code, but what about dependencies the compiler brings in automatically due to optimisations? Again we can see our dependencies for things like the vectorisation case mentioned above when -fopenmp is used:

With these you can see exactly which symbols are brought in with the current optimisation level, and rather than blindly disabling all optimisations, you can disable just the vectorisation optimisation you know your target GLIBC6 version can't support. This lets us keep all the important optimisations and language features, and doesn't limit us to using an older compiler.

Compile C libraries with custom GLIBC6

Even when you've done all of the above, you may find that your binary depends on some GLIBC6 symbols newer than your target supports, and that there is no earlier symbol version for these. Again one common cause of this is LibMVEC being brought in for C libraries you're linking to statically; two such libraries that bring in those dependencies by default with the latest GCC are GLFW3 and LibVorbis.

The best solution is to link to a custom build of these libraries, still using your latest GCC, but with a custom GLIBC6 that has LibMVEC support disabled. This is a bit more work, but provides the ideal solution - letting you use the latest libraries built with the latest compiler, without limiting our use of language features, libraries, or optimisations other than the automatic vectorisation that's causing us symbol versioning problems.

Build a custom GLIBC6

The first step is to compile and test a custom GLIBC6. Download the target glibc6 version you want to support, 2.19 in our case, and build it in its directory. You'll need to use an older version of GCC for this portion (4.9 in this case), but it doesn't prevent you from using a newer version of GCC for either your program or the libraries you're building.

mkdir build; cd build  
../configure "--prefix=/usr/src/glibc-nomvec" --enable-addons --disable-mathvec --disable-werror
make -j $(nproc) CC=gcc-4.9 && make install  

Here we've chosen to install it in /usr/src/glibc-nomvec which we'll reference later. Of course you can use any path you like.

Build your C libraries with the new GLIBC6

For a normal configure and Makefile based build system, such as that in LibVorbis, simply configure as normal but first set your compiler flags and linker flags to use the new GLIBC6 you've just built:

CFLAGS='-isystem /usr/src/glibc-nomvec/include' LDFLAGS='-L/usr/src/glibc-nomvec/lib' make -j $(nproc)  

For a library using CMake (such as GLFW3), configure as usual, but make the following modifications to the CMakeCache.txt after the initial cmake .. call before you build:

CMAKE_C_FLAGS:STRING=-isystem /usr/src/glibc-nomvec/include  

If you've built the library before, don't forget to make clean before you make with the new configuration. The result will be a C library that does not depend on LibMVEC, and hence will not require the newer GLIBC6, even though it's built with your latest version of GCC!

Bringing it all together

The three techniques above enable you to use the latest version of GCC and G++, all the latest C++11 / C++14 / C++17 language features, and almost all optimisations except LibMVEC vectorisation, and give you static libraries you can link with that won't drag in any unwanted version dependencies either; the result is a program built with the very latest cutting edge compiler and language features, that will run on every single stable Linux distribution out there.