7

With all the discussion about the xz-tools supply chain attack on the Linux distros, what confuses me:

As stated here or on the infographic here, the attackers worked their way to becoming trusted maintainers of the project. They used this to alter the source code by including some manipulated binary "test files".

However, they also had to tamper with the build process of the package. This they achieved by changing the source code inside the released tarball. These changes were never made in the source code hosted on GitHub, thus hiding them.

I find two things puzzling here:

  • This means the code inside the tarballs differed from the repository code, which seems to be not unusual as confirmed here:

    The release tarballs upstream publishes don't have the same code that GitHub has. This is common in C projects so that downstream consumers don't need to remember how to run autotools and autoconf. The version of build-to-host.m4 in the release tarballs differs wildly from the upstream on GitHub.

    How can this be "common", especially for potentially system critical C projects? Doesn't this essentially defeat the whole "security by transparency" effect that Open Source software supposedly has?

  • If all it needs is to have full access to a "trusted" upstream repository to freely manipulate the released tarballs that will find their way into the Linux distros, why go through all the extra effort described in the above articles of multiple stage loaders etc.?

2
  • 1
    Aha, this reminds of a CTF challenge I once had. It was a web challenge, for which there was an exploit available, but the releases would rick roll if you ran them instead of compiling them yourself (even the green checkmark was there!), hence a very good researched and overall good question. Commented Apr 5 at 18:27
  • according to that ARS article it was a source code tarball, so they'd still want to obfuscate things in case someone noticed something before building. As to why that's different than github's source code... that's a good question. I'm not so sure it was... maybe it was just a different branch? release vs development? Those can have different upstreams methinks... Commented Apr 5 at 19:45

1 Answer 1

5

This means the code inside the tarballs differed from the repository code, which seems to be not unusual as confirmed here:

Indeed, it's common to make release archives that have additional files compared to the repository. The quote you found explains why. There are many software projects where to get usable code, you need two steps:

  1. Use fiddly third-party tools to generate platform-independent files. The tools can be fiddly for a number of reasons: because you need a specific version (e.g. autotools), because they only run in a very specific environment (long dependency chains which practically require their own environment, so someone made a Docker file), because they require a lot of computation time (common for some kinds of data assets), etc.
  2. Use commonplace third-party tools to generate platform-dependent ready-to-use files. Typically, at this point, all you need is a compiler and associated tooling.

People who want the sources, as in the preferred form of modification, grab them from version control. People who want ready-to-use packages grab them after step 2: binary releases, or packages in a package manager or app store. People who want to build the program for their machine grab them from step 1. Step 1 is what release archives are used for.

Doesn't this essentially defeat the whole "security by transparency" effect that Open Source software supposedly has?

Uh, no? If you want the most transparency, you download the sources directly in their primary form, which is version control. If you download a derived product, such as a source release or a binary release, then you need to look at the production chain from the actual source (i.e. the version control) to what you're downloading.

Git/GitHub release archives are not great for transparency for other reasons anyway. They aren't fully reproducible: you can't just run git archive on your machine and be guaranteed to get the same archive binary. All that Git guarantees is that you get the same file contents and modification times. It doesn't guarantee an unambiguous representation in the archive format, and changes in the compression library can affect compressed archives. So while you can audit them, it's not as straightforward as keeping a reference hash for the archive file. Thus GitHub release archives aren't much better for transparency than manually generated archives. (But if the manually generated archives have additional content, then of course you need to audit that additional content. That's due to the additional content, not to the method of production.)


If all it needs is to have full access to a "trusted" upstream repository to freely manipulate the released tarballs that will find their way into the linux distros, why go through all the extra effort described in the above articles of multiple stage loaders etc.?

This has nothing to do with release tarballs. If you have full access to the repository, you can commit malicious code. That code can affect all users, not just people who use a release tarball.

The fact that the xz backdoor was only present when building from a release tarball (and only when building those under certain conditions) was a deliberate choice from the author of the backdoor. The source code is visible to everyone, so if they had just committed the backdoor itself, it would have been detected and rejected immediately. So they took several steps to make the backdoor harder to detect. Based on the initial write-up by Andreas Freund who discovered the backdoor, and with a bit of speculation since I can't know the backdoor author's thought process:

  • The directly malicious code is only contained in a test data file. Since it's a data file that's supposed to be a test case, people aren't expected to pay much attention to its content, so it had a high chance of getting undetected.
  • The code that injects the malicious code into the build is part of a build script, which the backdoor author edited. This script is written in a macro language that relies on multiple parsing, which means that the presence of an eval statement is not in itself suspicious, thus the malicious part doesn't stand out.
  • This script is not present in the Git repository. As far as I can see, it is maintained locally by the author of the backdoor, who is the person who prepared the release. Presumably the backdoor author didn't commit this critical part of the backdoor to Git to reduce scrutiny.
  • The backdoor is deliberately designed to affect only major Linux distributions, not all users. In particular, it would not be present in what security audits would typically consume. For example, OSS-Fuzz was fuzzing a non-backdoored build.

In terms of transparency, the major missing piece in xz seems to have been release preparation. Releases should be prepared by running a script that is itself reviewed like the rest of the source code. Ideally the release preparation script should be reproducible, meaning that you can run it at home and get the same result. If it isn't (e.g. due to timestamps or to compression library differences), then either the script should be run in an environment that is itself auditable (e.g. a Docker container running on a CI system), or the output of the release script should be reviewed. This applies to all release artifacts, whether they contain ready-to-run binaries or intermediate files that still need to be compiled.

As far as I can tell, the xz project relied on manual auditing of release artifacts. And such review was probably cursory, given that the releases were prepared by one of the two active maintainers. The backdoor seems to have been engineered specifically to be hard to detect at this stage, but in the specific case of xz, the biggest problem seems that it's critical piece of infrastructure that was maintained by a single overworked unpaid person until the malicious actor came along to ostensibly help.

2
  • Thanks for the detailed answer! Do you know whether the affected releases were source or binary archives?
    – Niclas
    Commented Apr 7 at 23:31
  • @Niclas From xz, only source archives. Does the xz project even have binary archives? The affected binaries that I'm aware of are the ones in distributions targeted by the backdoor. Commented Apr 8 at 10:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .