Table of Contents
On Friday, a lone Microsoft developer rocked the world when he created a back door was intentionally placed in XZ Utils, an open source data compression utility available on almost all installations of Linux and other Unix-like operating systems. The person or persons behind this project have probably been working on it for years. They were probably very close to seeing the backdoor update merge into Debian and Red Hat, Linux’s two largest distributions, when an eagle-eyed software developer spotted something suspicious.
“This may be the best-executed supply chain attack we’ve ever publicly described, and it’s a nightmare scenario: malicious, skilled, authorized upstream in a widely used library,” said software and cryptography engineer Filippo Valsorda. said of the attempt, which came frighteningly close to success.
Investigators spent the weekend gathering clues. Here’s what we know so far.
What is XZ Utils?
XZ Utils is almost ubiquitous in Linux. It provides lossless data compression on virtually all Unix-like operating systems, including Linux. XZ Utils provides crucial features for compressing and decompressing data during all kinds of operations. XZ Utils also supports the older .lzma format, making this part even more important.
What happened?
Andres Freund, a developer and engineer working on Microsoft’s PostgreSQL offering, was recently troubleshooting performance issues that a Debian system was experiencing with SSH, the most commonly used protocol for logging into devices remotely over the Internet. In particular, SSH logins consumed too many CPU cycles and generated errors fall gravela utility for monitoring computer memory.
Through sheer luck and Freund’s careful eye, he eventually discovered that the problems were due to updates made to XZ Utils. On Friday, Freund took to the Open Source Security List to reveal that the updates were the result of someone deliberately putting a backdoor in the compression software.
What does the back door do?
Malicious code added to XZ Utils versions 5.6.0 and 5.6.1 has changed the way the software functions when performing operations related to .lzma compression or decompression. When these functions involved SSH, malicious code could be executed with root privileges. This code allowed someone in possession of a predetermined encryption key to log into the backdoor system via SSH. From then on, that person would have the same level of control as any authorized administrator.
How did this backdoor come about?
It seems like this backdoor has been in the works for years. In 2021, someone with the username JiaT75 has his/her first known union to an open source project. In retrospect, the change to the libarchive project is suspect, because it replaced the safe_fprint function with a variant that has long been considered less secure. No one noticed at the time.
The following year, JiaT75 submitted a patch via the updating the software often or quickly enough. Kumar, with the support of Dennis Ens and several other people who had never been on the list, pressured Collin to bring in an additional developer to service the project.
In January 2023, JiaT75 made their commit first to XZ Utils. In the months that followed, JiaT75, who used the name Jia Tan, became increasingly involved in XZ Utils business. For example, Tan replaced Collins’ contact information with their own on oss-fuzz, a project that scans open source software for vulnerabilities that can be exploited. Tan also requested that oss-fuzz disable the ifunc feature during testing, a change that prevented it from detecting the malicious changes Tan would soon make to XZ Utils.
In February this year, Tan released commits for versions 5.6.0 and 5.6.1 of XZ Utils. The updates implemented the backdoor. In the following weeks, Tan and others called on developers from Ubuntu, Red Hat and Debian to merge the updates into their operating systems. Ultimately, one of the two updates made its way to various releases, according to security company Tenable. There’s more about Tan and the timeline here.
Can you say more about what this backdoor does?
In a nutshell, it allows someone with the right private key to hijack sshd, the executable file responsible for making SSH connections, and execute malicious commands from there. The back door is implemented via a five-stage loader that uses a series of simple but clever techniques to conceal itself. It also offers the opportunity to deliver new loads without the need for major changes.
Several people who reverse engineered the updates have much more to say about the backdoor. Developer Sam James has provided an overview here.