L06 · Case study: Equifax (2017)

Module 1: Security Foundations · ~20 min · Lesson 6 of 71

🎯 What you'll learn

• How one unpatched Java library led to one of the largest data breaches in US history
• Why an expired certificate meant nobody could see the attack for 76 days
• How to map each Equifax failure to the ideas from L01–L05
• Five checks for Java services, API gateways such as Kong, and databases

🔥 Why this matters

On 9 March 2017, Equifax's security team emailed about 430 people: a critical Apache Struts flaw is being exploited, patch within 48 hours. Two months later, attackers walked in through exactly that flaw, stayed 76 days, and took personal data on about 147–148 million people. If you've ever thought "we sent the patch notice, so we're covered", this lesson is for you.

🏢 The company and the setup

Equifax is one of the three big US credit bureaus. Credit bureaus hold data on people who never signed up and can't opt out: names, Social Security numbers, birth dates, addresses, and credit histories. That makes them a very high-value target.

The weak point was ACIS (Automated Consumer Interview System), an internet-facing portal where consumers dispute errors on their credit report. According to the House Oversight Committee report, it was custom-built in the 1970s and ran on legacy systems in a data center in Alpharetta, Georgia. Its web layer used Apache Struts, a popular open-source Java framework for building web apps.

Two more pieces matter:

• A TLS inspection device (an "SSL Visibility" appliance) sat in front of ACIS. It decrypted HTTPS traffic so the intrusion detection systems behind it could inspect it. To decrypt, it needed a valid certificate.
• Behind ACIS were databases. ACIS needed only three of them to do its job.

⏱ Attack timeline

Facts below come from the GAO report (GAO-18-559, August 2018) and the House Oversight Committee staff report (December 2018), which drew on Mandiant's forensics.

The attack path

 ATTACKER (internet)
   │
   │ 1. HTTPS request with a malicious Content-Type header
   │    exploits CVE-2017-5638 in Apache Struts (unpatched)
   ▼
 ┌─ TLS inspection box: certificate EXPIRED ───────────────┐
 │  default setting: pass traffic on UNINSPECTED           │
 └───────────────────────────┬─────────────────────────────┘
                             ▼
 ACIS dispute portal (2 web servers + 2 app servers)
   │ 2. uploads ~30 web shells (hidden remote-control pages)
   │
   │ 3. reads a config file on a mounted file share:
   │    database usernames + passwords, unencrypted
   ▼
 DATABASES (not segmented)
   │ 4. ACIS needed 3 databases; creds reach 48 more
   │ 5. ~9,000 queries; 265 return PII, unencrypted at rest
   ▼
 EXFILTRATION
     6. results compressed into a web-reachable folder and
        pulled out in small pieces over encrypted web
        traffic, which nobody was inspecting (see box)

The flaw, CVE-2017-5638, was in Struts' Jakarta Multipart parser, the code that handles file uploads. A crafted Content-Type header made the server run the attacker's commands. That's remote code execution (RCE), the worst class of bug: NVD rates it critical (CVSS 3.x score 9.8). Working exploit code was public within days.

The dates

 2016 Jan 31 ── cert on the inspection box expires; no alert
      │
      │          (monitoring blind from here: ~19 months)
      │
 2017 Mar 7  ── Struts flaw disclosed; Apache ships the fix
      Mar 8  ── US-CERT alerts Equifax
      Mar 9  ── internal email to ~430 people: "patch in 48h"
      Mar 15 ── scans of 958 external IPs: "nothing found"
      May 13 ── attackers get in through ACIS ──────┐
      │                                             │ 76 days
      │         queries, web shells, exfiltration   │ unseen
      │                                             │
      Jul 29 ── 67 certs renewed; suspicious ───────┘
      │         traffic spotted almost at once
      Jul 30 ── ACIS taken offline
      Aug 2  ── Mandiant hired, FBI notified
      Sep 7  ── public announcement (143M, later ~148M)
      Sep 15 ── CIO and CSO retire;  Sep 26: CEO leaves
 2019 Jul 22 ── FTC/CFPB/states settlement, up to ~$700M

A note on "19 months": the House report dates the expiry to 31 January 2016, citing Equifax's own list of expired certificates. GAO said "about 10 months before the breach". The House figure is better documented; either way, the device was blind for the whole attack.

🔍 What went wrong

It also maps onto L03's kill chain: exploitation (the Struts request), installation (web shells), command and control (commands through them), and actions on objectives (queries, exfiltration).

Equifax fired a senior IT manager for not forwarding the patch email, and the CEO blamed "human error" before Congress. But the House report found systemic causes: a 2015 internal audit had already flagged no complete asset inventory and an "honor system" for patching, still unfixed in 2017. A process that fails when one person misses one email isn't a process.

🛡 What would have stopped it

Any one of these would have stopped the attack or shrunk it a lot:

• Patch within the policy's own 48 hours. The attack began 67 days after the fix shipped.
• An accurate software inventory, so the alert reached the ACIS owners. Staff had seen Struts in ACIS in January 2017, but nobody recorded it.
• Certificate expiry monitoring, plus an alert when the inspection box stops inspecting. Equifax had let over 300 certificates expire, 79 of them for monitoring business-critical domains (House report).
• Segmentation and scoped secrets: ACIS reaches only its 3 databases, with credentials from a vault, not a file share.
• Detection inside the network: file integrity monitoring (Mandiant said it could have spotted the web shells), query-rate alerts, and alerts on unusual outbound volume.

🧪 Worked example: the library you didn't know you had

Equifax's open-source scan ran on the root directory, not the subdirectory where Struts lived. In a Java service the same blind spot is a transitive dependency: a library pulled in by another library, so it never appears in your own pom.xml. Here's a made-up service:

$ mvn dependency:tree -Dincludes=org.apache.struts
[INFO] com.example:dispute-api:jar:3.4.1
[INFO] \- com.example.legacy:case-forms:jar:1.9.0:compile
[INFO]    \- org.apache.struts:struts2-core:jar:2.3.31:compile
                                           ▲
            vulnerable: CVE-2017-5638 fixed in 2.3.32

Grepping pom.xml finds nothing; it only names case-forms. The fix: pin the library's version wherever it comes from, then scan the built artifact or image, not the source folder.

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.apache.struts</groupId>
      <artifactId>struts2-core</artifactId>
      <!-- 2.3.32 was the 2017 fix; today, use a supported release -->
      <version>${struts.safe.version}</version>
    </dependency>
  </dependencies>
</dependencyManagement>

🧰 Lessons for your own work

1. Dependency patching in Java services. Use Dependabot or Renovate, and scan the final image (ECR image scanning, or Trivy in CI). Track open critical CVEs per service with an owner and a deadline, not an email to a list.
2. Certificate expiry monitoring. List every certificate you depend on: gateway certificates (Kong's, in the example stack) and upstream mTLS client certs, ingress certs, and any TLS-inspection or egress proxy. Alert at 30, 14, and 7 days. Separately, alert when a security tool goes quiet. Silence should page someone.
3. No plaintext credentials in config: not in application.yml, a ConfigMap, or the repo. Load them from AWS Secrets Manager (via IRSA or the External Secrets operator). A Kubernetes Secret is only base64: encoding, not encryption.
4. Database segmentation. Each service gets its own database user (in the example stack, a MongoDB user) with a role on its own database only, never readAnyDatabase. Add a Kubernetes NetworkPolicy so only that service's pods can reach the database port.
5. Egress and volume monitoring. Default-deny egress where you can, keep VPC Flow Logs, and alert when a service account suddenly reads far more than usual. 9,000 queries is loud if anyone is listening.

🛠 Try it (optional, ~5 min)

Check how long a public certificate has left, then make it a test (safe, read-only):

echo | openssl s_client -connect example.com:443 \
  -servername example.com 2>/dev/null \
  | openssl x509 -noout -subject -enddate -checkend 2592000
# notAfter=...  then "Certificate will not expire"
# or "Certificate will expire" if under 30 days remain

Then, for one service you own: which certificates does it depend on, who renews each, and what alerts if one expires? If nobody knows, you've found your Equifax.

🧠 Key takeaways

• A fix only helps once deployed; Equifax's alert never reached the right system.
• You can't patch what you don't know you run: asset and software inventory comes first.
• A monitoring tool that fails silently is worse than none, because everyone believes it's working.
• Segmentation and scoped credentials decide if one foothold becomes 51 databases.
• Breaches are chains of ordinary failures; break any link.

📖 Words to know

⏭ Next time

Module 2 starts with L07: how packets actually travel. The TCP/IP layers, ports, and the three-way handshake, the ground floor for everything about networks, firewalls, and segmentation.

The gap between "fix available" and "fix deployed"

Apache shipped the Struts fix on 7 March 2017. Attackers got in on 13 May. That 67-day gap is normal at big companies, not unusual. The House report shows why: Equifax's 2015 internal patch audit had already found no comprehensive IT asset inventory, patches applied reactively after an alert, an "honor system" to confirm patches, and no escalation when a critical patch was missed. Patching is slow less because of laziness than because of four questions nobody can answer fast: Where do we run this? Who owns it? Will the fix break it? Did the fix actually land?

The patch pipeline, and where it broke

 1. ADVISORY       CVE published, vendor fix released
      │            Equifax: ok (US-CERT alert on Mar 8)
      ▼
 2. MATCH          which apps/images contain the library?
      │            Equifax: BROKE (no inventory, wrong dir)
      ▼
 3. ASSIGN         ticket to the owning team, with deadline
      │            Equifax: BROKE (stale email list)
      ▼
 4. FIX + TEST     bump version, run tests, deploy
      │
      ▼
 5. VERIFY         rescan what is actually running
                   Equifax: BROKE (scan said "clean")

The fix is to make steps 2, 3, and 5 automatic, so a person only has to do step 4.

SBOM: an inventory that writes itself

A software bill of materials (SBOM) is a machine-readable list of every component in a build, including transitive dependencies and their versions. The two common formats are CycloneDX and SPDX. Generate one in CI for every build, store it next to the image, and "where do we run struts2-core 2.3.31?" becomes a query that takes seconds instead of a week of emails:

# across all stored SBOMs (CycloneDX JSON), made-up file names
jq -r '.metadata.component.name as $app
  | .components[]
  | select(.name=="struts2-core")
  | "\($app)  \(.version)"' sboms/*.json
# dispute-api   2.3.31
# forms-batch   2.5.10.1

Deadlines by severity, and the KEV catalog

A patch policy needs deadlines that scale with risk (L02). An example policy (made up, but typical):

CVSS tells you how bad a bug could be. The CISA Known Exploited Vulnerabilities (KEV) catalog tells you which bugs attackers are actually using, which is the stronger signal. Since November 2021, US federal agencies have been required (Binding Operational Directive 22-01) to fix KEV entries by set due dates. CVE-2017-5638 was in the first batch: added 3 November 2021, due 3 May 2022, and marked as known to be used in ransomware campaigns. EPSS, a daily score estimating how likely a CVE is to be exploited soon, is a useful tiebreaker for everything else.

Verify what's running, not what's in the repo

The Equifax scans said "clean" because they looked in the wrong place. Scan the deployed thing: the container image in your registry, or the running pods. Compare the image digest a pod is actually running with the one you patched. A ticket marked "done" isn't proof.

🛠 Try it (~5 min, public data and your own project)

Look up the Equifax CVE in CISA's public KEV feed:

KEV=https://www.cisa.gov/sites/default/files/feeds
curl -s $KEV/known_exploited_vulnerabilities.json \
  | jq '.vulnerabilities[] | select(.cveID=="CVE-2017-5638")
        | {cveID, dateAdded, dueDate, knownRansomwareCampaignUse}'

Then, in a Maven project of your own, generate an SBOM with mvn org.cyclonedx:cyclonedx-maven-plugin:makeAggregateBom and open target/bom.json. Count how many components are listed versus how many dependencies your pom.xml names.

🧠 Extra takeaways

• Patching is slow mainly because of missing inventory, unclear ownership, and no verification.
• An SBOM per build turns "where do we run X?" into a seconds-long query.
• Set patch deadlines by severity and exposure, and treat anything in CISA KEV as urgent.
• Verify the running image, not the ticket status or the repo.

Six weeks, then a rush

Equifax spotted the attack on 29 July 2017 and announced it on 7 September. In between, the investigation ran and, from mid-August, a team built the public response website and call center setup in about three weeks (House report). When the news broke, both were overwhelmed. The technical breach was over; the second failure, the response, was just starting.

A new domain teaches people to trust look-alikes

Equifax put its breach-check site on a brand-new domain, equifaxsecurity2017.com, instead of a page on equifax.com. Security experts warned it looked exactly like a phishing site. A security researcher proved the point by putting up a look-alike, securityequifax2017.com, with a pop-up explaining it was fake. Then, according to the House report (citing Gizmodo's reporting), Equifax's own Twitter account sent people to the fake site for nearly two weeks because an employee swapped the two words.

 real:   equifax.com/...       ◀── what people already trust
 used:   equifaxsecurity2017.com      new, unfamiliar domain
 fake:   securityequifax2017.com      words swapped, by a
                                      researcher; linked by
                                      Equifax's own tweets

 lesson: if your official site looks like phishing,
         people can't tell phishing from your site

Anyone can register a look-alike domain and get a valid HTTPS certificate for it, so the padlock proves nothing about who owns a site. Customers who learn "Equifax sends you to odd new domains" are easy targets for the next real phishing campaign.

The site and the call center buckled

The House report describes enrollment errors, people getting different answers about whether they were affected depending on whether they checked from a computer or a phone, and a backend that couldn't keep up. One executive compared it to a bathtub filling far faster than it could drain. Equifax had to bring in about 1,500 temporary call center agents in roughly a week.

What a better response looks like

For your team, the useful habit is to treat incident response like any other production feature: designed, tested, and owned before you need it.

🛠 Try it (~3 min, public data)

Certificate transparency logs record every publicly trusted certificate issued. Open https://crt.sh/?q=equifaxsecurity2017.com in a browser and look at when certificates were issued for the breach domain. Then think about your own organization: does anyone get alerted when a certificate is issued for a domain that looks like yours?

🧠 Extra takeaways

• Incident response is part of the breach: a bad response multiplies the harm.
• Keep incident pages on your main, trusted domain; new domains train people to fall for phishing.
• HTTPS proves the connection is encrypted, not that the site is legitimate.
• Prepare, load-test, and rehearse incident communications before you need them.