L56 · Case study: SolarWinds (2020) and xz-utils (2024)

Module 8: Secure SDLC and Supply Chain · ~20 min · Lesson 56 of 71

🎯 What you'll learn

• How attackers hid a backdoor inside SolarWinds' own build, so signed, official updates delivered it
• How "Jia Tan" spent more than two years earning a maintainer's trust in xz-utils, then backdoored sshd through it
• Why code signing, code review and SCA missed both, and what could have caught them
• How to map the failures to earlier lessons and to your own builds, images and vendor tools

🔥 Why this matters

On 13 December 2020, FireEye revealed that attackers had shipped a backdoor inside routine updates of SolarWinds Orion, network monitoring software used by governments and large companies. On 29 March 2024, Microsoft engineer Andres Freund posted that the compression library xz-utils had been backdoored to attack SSH servers. In both cases the code arrived from the real source, through the real channel, with the real signature. Everything L55 taught you to check would have said "fine".

🏢 The companies and the setup

SolarWinds sells IT management software. Its Orion Platform monitors servers and network devices, so an Orion server typically holds admin credentials for much of the network, and often can reach the internet for updates. SolarWinds signed each release with its code-signing certificate (L17), and customers' systems checked that signature.

xz-utils is an open-source compression tool and library (liblzma) installed on almost every Linux system. For years it was maintained by one volunteer, Lasse Collin. OpenSSH doesn't use liblzma itself. But Debian, Fedora and others patch sshd to notify systemd when it starts, and libsystemd was linked to liblzma, so liblzma's code was loaded into every sshd process.

A note on sources. SolarWinds facts come from SolarWinds' filings and blog, FireEye/Mandiant, CrowdStrike and US government statements. xz facts come from Freund's oss-security post and public timelines (such as Russ Cox's). Who "Jia Tan" is remains unknown; any attribution is speculation.

⏱ Attack timeline

SolarWinds: inside the build

 ATTACKER (US government: Russia's SVR; Russia denies)
   │ 1. Sep 2019: inside SolarWinds' network; the
   │    way in was never publicly confirmed
   ▼
 ORION BUILD SERVER
   │ 2. autumn 2019: harmless test code in a release,
   │    a dry run of the technique
   │ 3. Feb 2020: SUNSPOT implant watches for
   │    MsBuild.exe building Orion, swaps one source
   │    file for a backdoored copy, restores it after
   ▼
 SIGNED UPDATES (Mar-Jun 2020)
   │ 4. backdoored DLL, signed with SolarWinds' real
   │    certificate: Orion 2019.4 HF5, 2020.2,
   │    2020.2 HF1
   ▼
 UP TO ~18,000 CUSTOMERS install it
   │ 5. SUNBURST sleeps ~2 weeks, checks for security
   │    tools, then beacons over DNS
   ▼
 HAND-PICKED TARGETS (~9 US agencies, ~100 companies)
     6. more malware, stolen credentials, forged SAML
        tokens (L26) to read cloud email

SUNSPOT checked that the original source file matched what it expected, so a mistake wouldn't cause build errors and get noticed. Nothing malicious ever appeared in Orion's source repository.

xz-utils: inside the maintainer team

 "JIA TAN" (account JiaT75; real identity unknown)
   │ 1. Oct 2021: first small, helpful patches
   ▼
 PRESSURE ON THE MAINTAINER
   │ 2. 2022: new accounts (widely believed to be
   │    sock puppets) complain about slow progress
   │    and push for a new maintainer: Jia Tan
   ▼
 CO-MAINTAINER (by early 2023)
   │ 3. commit rights, makes releases; lays
   │    groundwork in small, plausible commits
   ▼
 RELEASE TARBALLS 5.6.0 (24 Feb) + 5.6.1 (9 Mar 2024)
   │ 4. payload hidden in binary "test files"; a
   │    modified build script (m4) exists ONLY in
   │    the tarball, not in git
   ▼
 DISTRO BUILDS (x86-64, glibc, .deb/.rpm only)
   │ 5. build script splices payload into liblzma
   ▼
 sshd ──(distro patch)──▶ libsystemd ──▶ liblzma
     6. hooks RSA_public_decrypt: a command signed
        with the attacker's private key runs as root,
        before any login (CVE-2024-3094, CVSS 10)

The dates

 2019 Sep      SolarWinds network accessed
 2020 Feb 20   SUNBURST injected into the Orion build
      Mar-Jun  trojanized updates downloaded
      Dec 13   FireEye publishes SUNBURST; CISA orders
               agencies to disconnect Orion
      Dec 14   SolarWinds: "fewer than 18,000" customers
 2021 Jan      CrowdStrike publishes SUNSPOT analysis
      Apr 15   US attributes the campaign to Russia's SVR
      Oct      xz: Jia Tan's first patch
 2023 Oct 30   SEC sues SolarWinds and its CISO
 2024 Feb 24   xz 5.6.0 released (backdoored)
      Mar 29   Freund's post; CVE-2024-3094
      Jul 18   judge dismisses most of the SEC's claims
 2025 Nov 20   SEC drops the rest (settled in principle
               in July), with prejudice

What was (and wasn't) hit

SolarWinds. "Fewer than 18,000" is an upper bound of customers who installed an affected version, from SolarWinds' SEC filing. The attackers followed up at far fewer: US officials said about nine federal agencies and about 100 companies, including Microsoft (source code viewed) and FireEye (red-team tools stolen). The goal was espionage, mostly email.

xz. It was caught weeks after release, before stable distributions (RHEL, Debian stable, Ubuntu LTS) shipped it. It reached rolling and pre-release ones: Debian testing and unstable, Fedora Rawhide and 41, openSUSE Tumbleweed, Kali. No exploitation in the wild has been publicly confirmed.

How each was found. FireEye was investigating its own breach. xz was found because Freund noticed SSH logins using unusual CPU (about half a second each in his tests) and valgrind errors.

🔍 What went wrong

🛡 What would have stopped it

• Independent, reproducible builds. Build the same commit twice, on separate isolated builders, and compare hashes (first deep dive). SUNSPOT changed one builder; the hashes would differ.
• Build from the repository, not the tarball. Regenerating xz's build files from git would have left out the malicious m4 script; the payload "test files" were in git, but inert without it.
• Egress allow-lists for vendor servers. An Orion server that may only reach known hosts can't beacon to an attacker's domain (L12's deep dive). Watching its DNS helps too (second deep dive).
• Less linked code in sensitive processes. sshd didn't need liblzma. systemd had already changed to load compression libraries only when needed (dlopen), which some analysts think made the attacker hurry; that motive is speculation.
• Support maintainers. One unpaid maintainer is a single point of failure.

🧪 Worked example: "Are we running the bad xz?"

The question on 29 March 2024: which of our images and hosts have liblzma 5.6.0 or 5.6.1? On a host or inside an image you own:

xz --version
# xz (XZ Utils) 5.4.5
# liblzma 5.4.5
dpkg -s liblzma5 2>/dev/null | grep '^Version' || rpm -q xz-libs

Across stored SBOMs (L55), it's one query:

jq -r '.metadata.component.name as $app | .components[]
  | select(.name | test("^(liblzma5|xz-libs|xz-utils|xz)$"))
  | select(.version | test("^5\\.6\\.[01]"))
  | "\($app)  \(.name) \(.version)"' sboms/*.json
# ledger-batch    liblzma5  5.6.0-0.2          ← affected
# statements-svc  liblzma5  5.6.1+really5.4.5-1 ← false alarm

The second hit is Debian's fixed package: it rolled back to 5.4.5 code with a version string that still sorts above 5.6.1. Version strings lie in both directions, so check the distro advisory. Later clean releases (5.6.2 onward) exist too, so "5.6.x" alone doesn't mean affected. AWS's bulletin said Amazon Linux wasn't affected, so typical EKS nodes were fine; base images inside pods were a separate question.

🧰 Lessons for your own work

1. Your build agents are production. Whoever controls the runner controls every artifact (L57).
2. Give vendor agents least privilege and an egress allow-list. Monitoring, backup and security tools hold broad credentials, which makes them prime targets.
3. Keep an SBOM per image. "Are we affected?" should take minutes.
4. Deploy what you reviewed. Build from tagged commits, pin by digest (L14), verify provenance (L55).
5. Chase small anomalies. A p99 latency jump after a base-image bump deserves five minutes.

🛠 Try it (optional, ~5 min)

On your laptop (macOS or Linux), check whether xz is installed and from where (Homebrew briefly shipped 5.6.x in March 2024; the backdoor targeted Linux, and Homebrew rolled back anyway):

which -a xz; xz --version
brew list --versions xz 2>/dev/null

If you have Docker, compare a stable base image you use: docker run --rm debian:stable-slim dpkg -s liblzma5 | grep Version.

🧠 Key takeaways

• A signature proves where code came from, not that the build or the author was honest.
• SolarWinds' source was clean; the build was not. Compare independent builds.
• xz's git history was clean; the tarball was not. Build from what you reviewed.
• Patient social engineering of an overworked maintainer is a supply chain attack too.
• Both were found by people who chased anomalies; detection is part of supply chain defense.

📖 Words to know

⏭ Next time

L57 locks down the pipeline itself: secrets in CI, OIDC from GitHub to AWS instead of stored keys, branch protection, signed commits, and poisoned pipeline attacks.

SUNSPOT worked because nobody could tell that the shipped DLL didn't match the reviewed source. A reproducible build produces bit-for-bit identical output from the same source, so anyone can rebuild and compare. A mismatch means something in one build environment is different, or lying.

Two builders, one hash

                 tagged commit a1b2c3
             ┌───────────┴────────────┐
             ▼                        ▼
   ┌──────────────────┐     ┌──────────────────┐
   │ builder A        │     │ builder B        │
   │ (release CI)     │     │ (separate AWS    │
   │                  │     │  account, creds) │
   └────────┬─────────┘     └────────┬─────────┘
            │ sha256 9f74...         │ sha256 9f74...
            └──────────┬─────────────┘
                       ▼
           equal → sign and publish
           differ → stop: one builder is not clean

An attacker now has to compromise both builders the same way. SolarWinds says it moved to building in parallel environments and comparing the results after the attack. Debian, Arch and NixOS run public reproducibility checks on their packages.

Why builds differ, and the fixes

Worked example: is your jar reproducible?

In a Maven project of your own, add one property (supported by recent Maven plugins; older plugin versions ignore it):

<properties>
  <project.build.outputTimestamp>2026-01-01T00:00:00Z</project.build.outputTimestamp>
</properties>

mvn -q clean package && shasum -a 256 target/ledger-utils-1.4.2.jar
mvn -q clean package && shasum -a 256 target/ledger-utils-1.4.2.jar
# 3f1c...e9  target/ledger-utils-1.4.2.jar
# 3f1c...e9  target/ledger-utils-1.4.2.jar    ← identical

Without the property, the two hashes differ because of the timestamp in each file entry. If they still differ, diffoscope (open source) shows exactly which bytes changed and where.

The tarball gap

Autotools projects ship tarballs with generated files (configure, m4 macros) that aren't in git. That's normal, which is why it was a good hiding place: a reviewer reading git never sees them. The check is to regenerate them from the tagged commit (autoreconf -fi) and diff against the tarball. In xz, build-to-host.m4 was a real gnulib file with a few malicious lines added; a diff against a regenerated copy shows those lines. Since 2024, more distributions build from git tags directly.

🧠 Extra takeaways

• Reproducible builds turn "trust the build server" into "check two builders agree".
• Fix timestamps, ordering and paths first; they cause most differences.
• Release tarballs can contain code that's not in git; build from tags or diff regenerated files.
• A mismatch is a lead to investigate, not proof of attack; usually it's a timestamp.

SUNBURST ran on thousands of networks for months. Its authors put a lot of work into blending in, and knowing how helps you see where even careful attackers leave traces.

How it hid

• Patience. It waited about 12 to 14 days after installation before doing anything.
• Checks. It stayed quiet on machines not joined to a domain, on SolarWinds-looking test domains, and when it found a long list of security tools running.
• Camouflage. Its HTTP traffic imitated Orion's real telemetry, and its DNS lookups were subdomains of avsvmcloud[.]com styled like AWS AppSync names, e.g. <encoded>.appsync-api.us-east-1.avsvmcloud[.]com. The encoded part carried the victim's internal domain name.
• Selection. The DNS answer told each infected server whether to go dormant or connect to a second-stage server. Most victims never got past the beacon.

The signals that still showed

 monitoring server (should talk to: devices, vendor update host)
    │
    ├─▶ DNS: 40-character random label + new domain  ← signal 1
    │
    ├─▶ HTTPS to a host it has never contacted         ← signal 2
    │
    └─▶ later: new MFA device, unusual SAML tokens,
        mail read by an app                            ← signal 3

FireEye's own detection reportedly started with signal 3: an alert about a new device registered for an employee's MFA. Signals 1 and 2 need a baseline: knowing what a server normally talks to.

Worked example: long DNS names in Route 53 Resolver query logs

Route 53 Resolver query logging records every DNS query from a VPC. Here's a made-up line, saved to a local file:

{"query_timestamp":"2026-03-02T10:14:07Z",
 "query_name":"r1q6arhpujcf6jb6qqqb0trmuhd1r0ee.appsync-api.example.net.",
 "query_type":"A","rcode":"NOERROR","srcaddr":"10.20.3.41",
 "srcids":{"instance":"i-0123456789abcdef0"}}

Find first labels longer than 25 characters, grouped by source:

jq -r 'select((.query_name | split(".")[0] | length) > 25)
  | "\(.srcaddr)  \(.query_name)"' resolver-sample.json \
  | sort | uniq -c | sort -rn | head
#   1 10.20.3.41  r1q6arhpujcf6jb6qqqb0trmuhd1r0ee.appsync-api...

Expect false positives (CDNs and some AWS services use long labels), so allow-list known domains first. GuardDuty also analyzes DNS for known bad domains and exfiltration patterns (L52). The stronger control is still the egress allow-list from L12: then the beacon simply fails.

🧠 Extra takeaways

• Careful implants delay, check their environment, and imitate legitimate traffic.
• They still need to talk out; DNS and egress logs from servers with a narrow job are the place to look.
• Identity anomalies (new MFA devices, odd SAML tokens) often surface what network logs miss.
• A baseline of "what this server normally talks to" is what makes any of these signals visible.