What we know about the xz Utils hatchback that almost took it by storm

Researchers on Friday disclosed the discovery of a deliberately built-in backdoor in xz Utils, an open-source data compression tool available on nearly all installations of Linux and other Unix-like operating systems. The person or people behind this project have probably been working on it for years. You were probably close to seeing a backdoor update in Debian and Red Hat, the two largest Linux distributions, when an eagle-eyed developer noticed something suspicious.

“This may be the most well-executed supply chain attack publicly described, and it’s a nightmare scenario: malicious, efficient and allowed in a widely used library,” said software and cryptography engineer Filippo Valsorda. she said An effort that came shockingly close to success.

Researchers spent the weekend collecting evidence. Here’s what we know so far.

What is xz utility?

xz utilities are available almost everywhere in Linux. It offers lossless data compression on almost all Unix-like operating systems, including Linux. xz Utils provide functions needed to compress and decompress data in all types of operations. xz Utils also supports the older .lzma format, which makes this component even more important.

what happened

Anders Freund, a developer and engineer working on Microsoft’s PostgreSQL offering, recently addressed Debian’s performance issues with SSH, the protocol commonly used to log into computers remotely over the Internet. In particular, SSH logins consume many CPU cycles and cause errors in Valgrind, a computer memory monitoring utility.

Through a combination of sheer luck and Freund’s keen eye, he eventually discovered that the problems were caused by an update to xz Utils. On Friday, Freund disclosed on Open Source Security Menu that the updates were caused by someone intentionally creating a backdoor in the compression software.

What does the back door do?

Added malicious code to xz Utils versions 5.6.0 and 5.6.1 to change the program’s behavior when performing lzma compression or decompression operations. When these functions include SSH, they allow malicious code to execute with root privileges. This token allows anyone to log into the backend system via SSH with a predefined encryption key. From now on, this person will have the same level of control as any authorized administrator

How come about this back door?

It appears that this backdoor has been under construction for years. In 2021, a person with the username JiaT575 created his own account First known commit In an open source project. later, It is variable The libarchive project is suspect because it replaces the Safe_fprint function with a variable that has long been considered less safe. No one noticed at the time.

The following year, JiaT575 submitted a patch to the xz Utils mailing list, and almost immediately an unseen participant named Jigar Kumar joined the discussion, arguing that Lasse Collin, the longtime maintainer of Xz Utils, did not update the software often. Often or quickly enough. Kumar, with the support of Dennis Innes and a few others who were not previously on the list, lobbied Colin to bring in an additional developer to continue the project.

In January 2023, it was created by JiaT75 First commitment to xz Utils. In the months that followed, JiaT75, who used the name Jia Tan, became increasingly involved with xz Utils. For example, Tan replaced Collins’ contact information with his own for Microsoft’s OS Fuzz project, a project that examines open-source software for signs of malware. Tan asked oss-fuzz to disable the ifunc function during testing, a change that prevents him from detecting malicious changes Tan will soon make to xz Utils.

In February this year, Tan released commits for versions 5.6.0 and 5.6.1 of xz Utils. Backdoor update implemented. In the weeks that followed, Tan and others urged Ubuntu, Red Hat and Debian developers to include updates in their operating systems. Eventually one of the two updates made it to the next version. accordingly Defensible Security Company:

Can you tell me more about what this backdoor is?

In short, it allows anyone with the correct private key to hijack sshd, the executable responsible for establishing SSH connections, and run malicious commands from there. The tailgate is completed by a five-stage loader that uses a series of simple but clever tricks to hide itself. This provides the ability to deploy new payloads without requiring major modifications.

Many people who reverse engineer updates say a lot about backdoors.

Introduced by developer Sam James This is the overview:

This backdoor consists of several components. At a higher level:

• Versions of tarballs released upstream do not contain the same code as GitHub. This is common in C projects, so downstream consumers don’t have to remember how to run AutoTools and AutoConference. The version of build-to-host.m4 found in the tarball is significantly different from the original version on GitHub.
• The Git repository also contains test files created in the test/ folder. These files are located in the following commits:
• A script called from build-to-host.m4 unpacks this corrupted test data and uses it to modify the build process.
• IFUNC, a mechanism in Glibc that allows indirect function calls, is used for runtime binding/redirection of OpenSSH authentication methods. IFUNC is a tool that is normally used for legitimate purposes, but in this case it is being used for this attack path.

Upstream usually releases tarballs that are different from the ones automatically generated on GitHub. These modified tarballs include a malicious version of build-to-host.m4 to run a script during the build process.

This script (at least versions 5.6.0 and 5.6.1) checks various conditions such as device architecture. Here’s an excerpt of the corrupted malicious script from build-to-host.m4 and an explanation of how it works:

if ! (echo "$build" | grep -Eq "^x86_64" > /dev/null 2>&1) && (echo "$build" | grep -Eq "linux-gnu$" > /dev/null 2>&1);then

• If amd64/x86_64 is the build target
• and if the target uses the name linux-gnu (mainly checks using glibc)

The tool chain used is also checked:

if test "x$GCC" != 'xyes' > /dev/null 2>&1;then
exit 0
fi
if test "x$CC" != 'xgcc' > /dev/null 2>&1;then
exit 0
fi
LDv=$LD" -v"
if ! $LDv 2>&1 | grep -qs 'GNU ld' > /dev/null 2>&1;then
exit 0

And if you’re trying to build a Debian or Red Hat package:

if test -f "$srcdir/debian/rules" || test "x$RPM_ARCH" = "xx86_64";then

So this attack seems to target AMD64 systems running glibc using distributions from Debian or Red Hat. Other systems may be vulnerable at this point, but we don’t know.

In an online interview, developer and reverse engineer HD Moore confirmed Sam James’ suspicions that the backdoor targets Debian or Red Hat distributions.

“The attack was stealthy because it only required a final backdoor step if you built the library on AMD64 (Intel x86 64-bit) and created a Debian or RPM package (not using it for local installation). ” ,” she wrote.

He continued by explaining the observations of researchers who collectively spent the weekend analyzing the malicious updates:

When verifying an SSH public key, if the public key matches a specific fingerprint function, the contents of the key are decrypted using a pre-shared key before the public key is actually verified. The decrypted content is then forwarded directly to the system.

If the fingerprint doesn’t match or the decrypted content doesn’t match a certain format, normal key verification happens again and no one finds out about it.

The back door is very sneaky. Uses an unknown function of glibc to bind a function. It only runs when the xz backdoor library is loaded in one of the distributions affected by the /usr/bin/sshd process. There may be many other backdoors, but the backdoor everyone is talking about uses an undirected function to add a hook. The payload was encrypted in fake xz test files and effectively executed as shellcode, modifying the SSH RSA key verification code to allow the attacker access to the magic public key (sent during normal authentication).

Their big plans were:

1) Invisibly release the force backdoor, but not the source code

2) Use the Sockpuppet account to convince various Linux distributions to download and package the latest version

3) Once these distributions are shipped they can be adopted by any user/company system etc

For more technical analysis, see above Bluesky thread from Valsorda, Researcher Kevin Beaumont And Freund released this information on Friday.

What else do we know about Jia Tan?

At this point, it’s too little, especially for someone tasked with monitoring software as extensive and sensitive as xz Utils. This developer personality has touched dozens of other open source software in recent years It is currently unknown if there is a real person behind this username or if Jia Tan is a completely fictitious person.