The xz-utils backdoor: The supply chain RCE that got caught

What you need to know

 

• The xz-utils package in versions 5.6.0 and 5.6.1 includes a malicious backdoor that could, in specific circumstances and configurations, allow remote access to SSH sessions for remote code execution (RCE) on selected Linux systems.
• As a precaution, all Linux users are advised to ensure their xz-utils version is earlier than 5.6.0 and downgrade if necessary, especially if running public sshd. While only a small percentage of systems worldwide could be directly vulnerable, this may change with further analysis.
• All signs point to a multi-year, carefully planned supply chain compromise operation by an advanced threat actor that may have also tampered with other open-source packages.

On March 29, 2024, software engineer Andres Freund reported finding a backdoor in the liblzma library, part of the xz-utils package. What started with investigating a drop in OpenSSH performance on a pre-release Debian Linux system turned into a global security scare that is still unfolding. Luckily, the backdoor was discovered before the compromised library version became more widely used, so relatively few systems could be immediately affected. The bigger story is how the backdoor was created, hidden, and distributed—and how it could have compromised the security of millions of systems if it went into widespread use.

How xz-utils got backdoored

Open-source software is commonly downloaded in packages called tarballs that are compressed using one of several popular compression utilities—most often Gzip (making .tar.gz files), but XZ is also used (resulting in .tar.xz files). XZ compression is also used internally by some programs, making the xz-utils package a necessary part of any Linux system.

The xz-utils project was created and maintained by Lasse Collin until a helpful and very insistent contributor going by the name of Jia Tan recently succeeded in fully taking over the project on GitHub. Among Jia’s latest commits were alleged compression performance improvements to the liblzma library, published in versions 5.6.0 and 5.6.1 of xz-utils. These are the versions that included the backdoor, but the compression utility was only a stepping stone to a much bigger prize.

One piece of software that depends on the liblzma library is OpenSSH, though only in some system configurations, specifically where it’s been patched to play nicely with system notifications from the systemd process manager (notably in Debian Linux). In that setup, any running SSH server depends on liblzma—and getting control of those remote shell sessions was the ultimate goal.

The payload: Malicious code? What malicious code?

The backdoor was reported by Red Hat as CVE-2024-3094 as “malicious code” in the package. What makes it different from most software vulnerabilities is that the source code itself is clean and secure. The backdoor is hidden in separate “test” files and only reassembled and inserted into the library during compilation. What follows is a hugely simplified overview of what is known about the backdoor, especially considering that every step is obfuscated and performed with fiendishly clever tricks using innocent text-processing utilities.

Before source code written in a language like C or C++ can be executed, it needs to be compiled from a text file into a binary file. This is a complicated process, so most open-source projects also include ready compilation scripts (makefiles) alongside the source code and any additional files. For convenience, the whole thing can be downloaded as a single tarball package—and this is where Jia Tan put the malicious code.

To avoid detection by scanners, the malware binary was, in effect, cut up into several pieces, and the gaps filled up with junk. For additional stealth, it is only included in the packaged tarball, so it’s not there if anyone examines the individual files in the repository. But if the package from an infected tarball is compiled on a system that meets specific configuration requirements, the build scripts reassemble the malicious code and attach it to the liblzma library, where it waits for a specific function call from a remote secure shell (SSH) session.

If all the conditions are met, a malicious actor can activate the backdoor by connecting to a compromised system over SSH and sending their encrypted access key. When successful, this could allow them to bypass the entire authentication process and gain unauthenticated remote access to the system.

Now imagine what would happen if this wasn’t caught and the backdoored unstable versions became stable versions that were gradually incorporated into all major Linux distributions during the next few years, spanning thousands if not millions of Linux servers and workstations worldwide… No wonder this CVE scored 10 out of 10 for severity.

The helpful contributor who took over and then vanished

If the maintainer of a long-standing and widely used open-source project putting a backdoor in that project sounds unthinkable, that’s because it is. As noted, the malicious code was introduced by the mysterious Jia Tan, aka JiaT75, who only became the maintainer shortly before. When the story broke, people started piecing together the online activity and history of this Jia—and discovered someone who seemingly only popped into existence in October 2021.

Around that time, JiaT75 started making small contributions to various open-source projects, most likely to build credibility rather than engage in malicious activity. (Although having a curious preference for projects that somehow touched SSH.) Getting involved in xz-utils, Jia gradually became more and more active, eventually gently persuading the founder to relinquish control of the venerable project in the name of innovation (with the aid of several other suspiciously eager contributors). With that, Jia was finally ready to upload the backdoored bits and pull off what Michał Zalewski has called “one of the most daring infosec capers” he has ever seen.

While the “Jia Tan” moniker was clearly intended to look Chinese and nearly all of Jia’s logged activity is from a Far East time zone, researchers have pointed out several oddities that don’t fit the “Chinese software enthusiast” cover story. Notably, Jia’s active hours correspond very closely to 9 am to 5 pm in Central Europe. The user was also active during some major Chinese holidays but inactive during some European holidays. Finally, a handful of login timestamps include the CET time zone rather than the usual one, as if someone forgot to change the system time before logging on.

One theory is that the JiaT75 account is not an individual but an advanced threat actor group, with many pointing to APT29 (aka Cozy Bear) as a group with similarly stealthy operational patterns and sufficiently advanced tech skills. You may remember them from the SolarWinds Orion hack—also a supply chain attack, as it happens. Whatever the case, Jia (unsurprisingly) vanished into thin air when the backdoor was reported and has not been seen since.

A new era for exploiting the reliance on open-source software

Compared to the devastation of something like the MOVEit Transfer data breaches, this whole story might seem like a non-issue: nobody was hacked (that we know of), nothing was lost, and the compromise attempt was foiled. On top of that, only a narrow subset of systems could currently have been targeted, and only in specific circumstances. While that’s all true, the details of this incident should be ringing the loudest software supply chain security alarm bells since that SolarWinds Orion incident.

The technical innovation of the attack was to hide malicious code not in the source but in innocent-looking additional files packaged with it. The sophistication, stealth, and multi-year patience of Jia Tan points to an advanced threat actor group with the resources and motivation to gamble on a long game where the prize could be persistent RCE on thousands of systems. Yes, the xz-utils backdoor was found, but mostly by coincidence and sheer luck, as Andres Freund himself is quick to point out. Though an experienced software engineer, Freund is not a security researcher, nor was he even investigating that specific package. It was a very lucky find for everyone.

It’s pretty clear there’s a high risk that a similar future attempt may succeed. Given the scale of the operation, it seems unlikely that a global threat actor would invest all that time and effort into compromising only one niche package, targeting (at least initially) a very narrow group of systems. Which begs the question: How many other open-source packages have already been backdoored by extremely helpful contributors with no prior history?

“While the audacity of the whole operation is striking, it’s not surprising that someone managed to hide a backdoor in plain sight, given how much developers have to rely on third-party components and libraries that often come with their own dependencies,” notes Sven Morgenroth, Staff Security Engineer at Invicti. “It’s like with Node.js projects, where you might have relatively few direct dependencies but get a node_modules folder full of additional ones. This is a problem for security because even small coding mistakes (not to mention deliberate backdoors) can quickly propagate from dependencies to your otherwise secure application.”

The open-source ecosystem was built on mutual trust and support. As both erode and the maintainers of crucial software components are left to their own devices, it looks like Jia Tan and friends are actively stepping in to backdoor and wire-tap the very foundations of the information age. The xz-utils incident merely serves as a reminder and proof point that supply chain attacks are indeed the #1 global software security threat. “Given the sheer amount of third-party code powering our applications and the lack of volunteers to audit these components, it’s close to impossible to assess the security of an application without using some form of automation,” concludes Morgenroth.

In the meantime, we’re keeping an eye on this story and will update here as new details emerge.