L28 · Case study: Uber (2022)

Module 4: Identity and Access · ~20 min · Lesson 28 of 71

🎯 What you'll learn

• How a bought password plus a flood of MFA push prompts got an attacker into Uber
• How one script on a file share turned a contractor login into admin access everywhere
• Why a password vault is a single point of failure, and how to protect the key to it
• How to map the failures to Module 4 (L21–L27) and to everyday SSO, scripts, and DAGs

🔥 Why this matters

On the evening of 15 September 2022, Uber employees saw a message in a company-wide Slack channel: "I announce I am a hacker and Uber has suffered a data breach" (as reported by The New York Times). No zero-day, no malware on Uber's servers: a stolen password, a tired contractor who tapped "Approve", and an admin password in a script were enough. Every protocol in this module worked as designed, and the attacker still got in.

🏢 The company and the setup

Uber runs ride-hailing, food delivery, and freight in dozens of countries. Like most large companies, its staff sign in once through single sign-on (SSO) (L24) to reach the VPN, Slack, Google Workspace, and cloud consoles. Sign-in used push MFA ("login approval requests", in Uber's words): the phone asks "Approve this sign-in?" and the user taps yes or no. The victim was an EXT contractor, Uber's term for an external worker with a corporate login.

Two more pieces matter:

• A privileged access management (PAM) system: a vault that stores and hands out the most powerful credentials (domain admin, cloud admin, service accounts). Uber's was reportedly Thycotic Secret Server (Thycotic is now part of Delinea).
• Internal network file shares where IT kept scripts, including PowerShell automation.

A note on sources. Uber published a short security update (19 September 2022, also filed with the SEC). It covers the entry point and some impact, but not the internal path. The script and the PAM vault come from the attacker's own messages and screenshots, relayed by researchers and reporters (BleepingComputer, The New York Times). Below, [Uber] marks Uber's statements and [claim] marks the attacker's account.

⏱ Attack timeline

The attack path

 ATTACKER
   │ 1. likely buys the contractor's Uber password on the
   │    dark web; malware on the contractor's personal
   │    device had exposed it                      [Uber]
   ▼
 CONTRACTOR'S PHONE
   │ 2. logs in again and again: push after push,
   │    denied at first                            [Uber]
   │ 3. WhatsApp message posing as Uber IT: "accept
   │    and they'll stop"                         [claim]
   │    contractor accepts one                     [Uber]
   ▼
 SSO + CORPORATE VPN
   │ 4. inside: scans the intranet                [claim]
   ▼
 NETWORK SHARE (readable by this user)
   │ 5. PowerShell script holding the username and
   │    password of an ADMIN in Thycotic (PAM)    [claim]
   ▼
 PAM VAULT
   │ 6. pulls secrets for other systems: domain admin,
   │    Duo, OneLogin, AWS, Google Workspace      [claim]
   ▼
 EVERYTHING ELSE
     7. AWS, GCP, vSphere, SentinelOne, HackerOne
        reports [screenshots, claim]; Slack, G-Suite,
        HackerOne, finance invoices            [Uber]
     8. posts in company Slack; changes OpenDNS to show
        a graphic image on internal sites      [Uber]

Uber's own version of steps 4–6 is shorter: the attacker "accessed several other employee accounts which ultimately gave the attacker elevated permissions to a number of tools". It neither confirmed nor denied the script. Everyone agrees on the shape: one login became admin over almost everything.

The dates

 2016 Oct-Nov ── separate earlier breach (see below)
 2022 Sep 15  ── attack; hacker announces it in Slack;
      │          Uber: "responding to a cybersecurity
      │          incident"; HackerOne pauses Uber's program
      Sep 19  ── Uber update: contractor account, dark web
      │          password, MFA approval, Lapsus$ affiliate
 2023 Aug 23  ── London jury finds a UK teenager carried
      │          out the Uber hack, among 12 offences
      Dec     ── indefinite hospital order

Who did it. Uber said the attacker was "affiliated with" Lapsus$, an extortion group behind loud, social-engineering-led breaches in 2022 (Okta, Microsoft, Nvidia). In August 2023 a London jury found that Arion Kurtaj, 18, had hacked Uber, Revolut, and others. Judged unfit to stand trial, he wasn't "convicted": the jury found he did the acts. Prosecutors said he hacked Uber on his own, not for Lapsus$.

What was (and wasn't) hit

Uber said it found no evidence of access to production systems, user accounts, or sensitive user data such as trip history, and no code changes. The attacker downloaded some Slack messages and saw invoice data and HackerOne bug reports (since fixed). Uber reset accounts, rotated keys, froze code changes, and forced everyone to sign in again.

🔍 What went wrong

In L03's Kill Chain terms there was no exploit and no malware on Uber's side, which is why identity-based intrusions are hard to spot: every step looks like a real user at work.

🛡 What would have stopped it

• Phishing-resistant MFA (FIDO2 security keys or passkeys, L21). The attacker would need the contractor's key next to their computer. No prompt to approve, so nothing to fatigue.
• At minimum, number matching plus push limits. The user types a number from the sign-in screen; after a few denials, pushes stop and security is alerted. But an attacker chatting with the victim can still read the number out (first deep dive).
• No credentials in scripts. The job runs as a service identity that can read one secret.
• Treat the PAM vault as the crown jewels. Its admins use hardware keys, no static admin password exists anywhere, and bulk secret reads raise an alert.
• Least privilege for contractors. A contractor account shouldn't read IT's script shares.

Context: Uber had been here before. In late 2016, attackers used a password from other breaches to get into an engineer's personal GitHub account, found an AWS access key in a private repo, and downloaded data on about 57 million riders and drivers from S3 (FTC revised complaint, 2018). Uber paid them $100,000 through its bug bounty and disclosed it only in November 2017. On 5 October 2022, weeks after the new breach, a federal jury convicted former CSO Joe Sullivan of obstructing the FTC and concealing a felony; he was sentenced to probation in 2023. Hard-coded credentials were the pivot both times, and hiding a breach can be a crime.

🧪 Worked example: the script on the share, and the same bug in a DAG

What the reported script looked like, in spirit (made-up names):

# \\files01.corp.example.com\it-scripts$\Sync-Accounts.ps1
$PamUrl  = "https://pam.corp.example.com/SecretServer"
$PamUser = "svc-pam-admin"        # vault ADMIN, not one app
$PamPass = "Winter2022!Example"   # static, never rotated
# ...log in to the vault, fetch a secret, do the job...

A static, plaintext password for an admin account, where many people can read it. The same bug in an Airflow DAG:

# dags/daily_recon.py  (vulnerable)
MONGO_URI = ("mongodb://recon_admin:S3cr3t-Example"
             "@mongo.example.com:27017/?authSource=admin")

Anyone who can read the DAG repo or the Airflow image now owns recon_admin. The fix keeps the secret out of code and narrows what it unlocks:

# dags/daily_recon.py  (fixed)
from airflow.providers.mongo.hooks.mongo import MongoHook

def load_recon():
    # "recon_mongo" is looked up at run time in AWS
    # Secrets Manager (Airflow secrets backend);
    # the DB user has readWrite on the recon db only
    hook = MongoHook(mongo_conn_id="recon_mongo")
    ...

The secrets backend (SecretsManagerBackend in the Amazon provider; on MWAA, set secrets.backend) reads airflow/connections/recon_mongo using the environment's IAM role, so no password sits in the repo. The Windows version is the same idea: run the job as a dedicated service identity allowed to read one secret, never a vault admin.

🧰 Lessons for your own work

1. Know your MFA. Does your IdP use number matching? Can you register a FIDO2 key or passkey? Ask for phishing-resistant MFA on admin roles and prod access.
2. An MFA prompt you didn't start means your password is stolen. Deny, report, change it. Real IT never asks you to approve a prompt over WhatsApp, a text, or a call they started.
3. No secrets in scripts, DAGs, manifests, or CI logs. Use GitHub secret scanning and push protection (L19), and workload identities and secrets backends (IRSA on EKS, Airflow's secrets backends).
4. The key to the vault is the real secret. Whatever unlocks Secrets Manager or your vault (an IAM role, a CI identity, a PAM API user) deserves your strongest protection.
5. Short sessions for powerful roles. If your organization uses IAM Identity Center, keep admin permission sets at 1 hour and know your revocation runbook (first deep dive), since a stolen session outlives a reset.

🛠 Try it (optional, ~5 min)

Plant three fake secrets in a scratch folder and see what a quick grep finds (local only):

mkdir -p ~/l28-try && cd ~/l28-try
printf '%s\n' '$PamUser = "svc-pam-admin"' \
  '$PamPass = "Winter2022!Example"' > Sync-Accounts.ps1
printf '%s\n' \
  'URI = "mongodb://recon:S3cr3t-Example@db.example.com"' \
  > daily_recon.py
printf '%s\n' '- name: DB_PASSWORD' '  value: "hunter2-ex"' \
  '- name: API_TOKEN' \
  '  valueFrom: {secretKeyRef: {name: api, key: token}}' \
  > deploy.yaml
grep -rnEi '(pass(word)?|secret|token|api_?key)[a-z_]*"? *[:=]' .
grep -rnE '://[^/:@ ]+:[^@ ]+@' .
# deploy.yaml:4:  valueFrom: {secretKeyRef: {name: api, ...
# Sync-Accounts.ps1:2:$PamPass = "Winter2022!Example"
# daily_recon.py:1:URI = "mongodb://recon:S3cr3t-Example@...

The first grep flagged the safe secretKeyRef line (a false positive) and missed the real hunter2-ex, whose name is on the line above. Grep is a start; real scanners (second deep dive) know hundreds of key formats. Try it on your own repo with --exclude-dir=.git, then delete ~/l28-try.

🧠 Key takeaways

• Push MFA without number matching can be beaten by annoyance; FIDO2 can't be fatigued.
• An unexpected MFA prompt means your password is already in someone else's hands.
• Credentials in scripts turn a low-level login into admin; the vault's own key is the top secret.
• SSO concentrates risk: protect the identity provider and MFA admin consoles like production.
• Hard-coded credentials were the pivot in both of Uber's breaches, six years apart.

📖 Words to know

⏭ Next time

Module 4 is done. Module 5, Web Application Security, opens with L29: the OWASP Top 10, what it is, how it's built, and a tour of the risks you'll spend the next nine lessons taking apart.

Two doors, same trick

Uber's attacker went after the user: flood their phone, then message them as "IT". A year later, the same playbook aimed at the help desk. In September 2023, MGM Resorts was hit by an attack that cost it about $100 million (MGM's SEC filing). The attackers claimed, via a security research account (vx-underground), that they found an employee on LinkedIn and phoned MGM's IT help desk pretending to be them; MGM's filings didn't name the attackers or the method. The group, Scattered Spider, is described in CISA advisory AA23-320A: posing as IT staff, push bombing, SIM swapping, and calling help desks to "reset passwords and/or transfer MFA tokens".

 DOOR 1: the user                 DOOR 2: the help desk
 ─────────────────                ─────────────────────
 attacker has password            attacker has name, ID,
   │                                manager (LinkedIn)
   │ 1. push, push, push            │ 1. calls help desk:
   │ 2. "Hi, Uber IT here,          │    "new phone, locked
   │    approve and it stops"       │    out, urgent demo"
   ▼                                ▼
 user taps Approve               agent resets password
   │                             and moves MFA to the
   │                             attacker's device
   ▼                                ▼
 attacker has a session  ◀──same result──▶  attacker
                                            enrolls MFA

Close door 1: make approving impossible to fake

• FIDO2 / passkeys for as many people as you can, starting with admins, IT, and help desk. CISA's advisory says these are "not susceptible to push bombing or SIM swap attacks".
• Number matching everywhere else, so a blind tap isn't enough. Microsoft made it mandatory for Authenticator push in May 2023; Duo calls it Verified Push.
• Rate-limit and alert. After 3 denials, stop sending pushes for a while and tell security.
• Show context in the prompt (app, location, approximate place) and give a "Report" button.

Close door 2: verify like the caller is lying

• Never verify with what's public. Name, employee ID, manager, and start date are on LinkedIn or in a breach dump.
• Call back on the number in HR records, never the inbound number, or verify in person or by video against a government ID.
• Manager or second-person approval for any MFA reset on a privileged account.
• After a reset, access is limited: no admin roles until the user has re-enrolled a FIDO2 key, and the user and their manager get a notification on a separate channel.
• Publish the rule: "IT will never message you on WhatsApp or ask you to approve a prompt."

Worked example: spotting push bombing in IdP logs

Most IdPs can export MFA events. A made-up evening, one JSON object per line in idp.jsonl:

{"ts":"21:02:10","user":"ext.lee","result":"DENIED"}
{"ts":"21:03:41","user":"ext.lee","result":"DENIED"}
{"ts":"21:05:02","user":"ext.lee","result":"TIMEOUT"}
{"ts":"21:06:30","user":"ext.lee","result":"DENIED"}
{"ts":"21:07:12","user":"a.khan","result":"APPROVED"}
{"ts":"21:08:55","user":"ext.lee","result":"DENIED"}
{"ts":"21:11:19","user":"ext.lee","result":"APPROVED"}
{"ts":"21:15:00","user":"b.ortiz","result":"DENIED"}
{"ts":"21:15:40","user":"b.ortiz","result":"APPROVED"}

Find users with three or more failed prompts whose last prompt was approved:

jq -s -c 'group_by(.user)
  | map({user: .[0].user,
         failed: (map(select(.result != "APPROVED"))
                  | length),
         then_ok: (last.result == "APPROVED")})
  | map(select(.failed >= 3 and .then_ok))' idp.jsonl
# [{"user":"ext.lee","failed":5,"then_ok":true}]

b.ortiz fumbled once, which is normal. ext.lee refused five times and then approved, which is the Uber pattern. A real rule adds a time window and the source IP and device. This should page someone and revoke the session automatically.

When it happens: revoke in IAM Identity Center

Resetting a password doesn't end sessions. AWS's documented order, for a compromised user:

 1. add the user's ID to a Deny policy (an SCP), so
    existing role sessions stop working
 2. remove their permission set assignments / groups
 3. disable the user in the identity source (your IdP)
 4. Identity Center console: delete active sessions
 5. disable user access in Identity Center
 6. keep the Deny for at least 12 hours (the longest
    permission set session), then reset and restore

Step 1 matters most: per AWS docs, ending an Identity Center session "doesn't end any active IAM role sessions". The portal session lasts 8 hours by default (configurable from 15 minutes to 90 days; your external IdP's own session settings also apply). Set the SCP up before you need it, which AWS's docs also recommend.

🛠 Try it (~5 min, local)

Save the nine lines above as idp.jsonl and run the jq command. Then change b.ortiz to deny three times before approving and run it again. Finally, find out who answers the IT help desk where you work and how they'd verify you if you called from a new phone. If the answer is "your employee ID", that's a finding.

🧠 Extra takeaways

• Attackers either wear down the user or talk the help desk into resetting MFA.
• FIDO2 closes door 1; strict, out-of-band identity checks close door 2.
• Denials followed by an approval is a high-signal alert; automate the response.
• Ending an SSO session isn't enough; know how to kill cloud role sessions too.

Where secrets hide

Uber's attacker found an admin password because someone had put it in a script years before. Every company has these. In a stack like the one in these examples, look here:

 PLACE                     WHAT LEAKS
 ───────────────────────   ──────────────────────────────
 repos + git history       keys deleted in a later commit
 scripts on shares         PowerShell/bash with passwords
 Airflow DAGs              URIs in code; Variables with
                           secrets in plain view in the UI
 k8s manifests, Helm       env "value:" literals, ConfigMaps
 container images          ENV/ARG lines, copied .env files
 CI logs                   secrets echoed or derived values
                           that masking doesn't catch
 notebooks, Slack, wikis   "here's the prod password"

The Uber attacker also downloaded Slack messages. Pasted credentials in chat are hard-coded credentials too.

The tools

• GitHub secret scanning and push protection (L19): scans repos for known key formats and blocks pushes containing them. For private org repos it needs GitHub Secret Protection (formerly part of Advanced Security). Ask your platform team what's on.
• gitleaks: open source, fast, regex-based. gitleaks git scans history; gitleaks dir scans files (the detect/protect commands were deprecated in v8.19). Its README says it is now feature complete, with only security fixes planned.
• trufflehog: open source, many detectors, and it can verify a hit by calling the provider's API (for an AWS key, GetCallerIdentity). Useful, but it sends the credential over the network: on a work laptop use --no-verification unless your security team agrees.

Run them only on code and systems you own or are authorized to scan: your team's repos and your own namespaces. Never point them at other people's repos or at company-wide data without permission. Finding secrets in someone else's system is the attacker's step 5.

Worked example: a first pass on your own repo and namespace

# after: brew install gitleaks trufflehog
cd ~/code/policy-service
gitleaks git --redact -v .        # whole history
gitleaks dir --redact -v .        # current files
trufflehog git file://$PWD --no-verification
#
# gitleaks prints one block per finding (illustrative):
#   Finding:  password = "REDACTED"
#   RuleID:   generic-api-key
#   File:     dags/daily_recon.py
#   Commit:   4f2c9e1...   (in history, not HEAD)
#   Fingerprint: 4f2c9e1...:dags/daily_recon.py:...

Then look for literal values in Kubernetes env, in a dev namespace you own (read-only):

kubectl -n recon-dev get deploy -o json | jq -r '
  .items[] | .metadata.name as $d
  | .spec.template.spec.containers[].env[]?
  | select(.value != null)
  | select(.name | test("PASS|SECRET|TOKEN|KEY"; "i"))
  | "\($d): \(.name) is a literal value"'
# recon-loader: DB_PASSWORD is a literal value

valueFrom.secretKeyRef is fine (though a k8s Secret is only base64, L19). A literal value is a finding. Also check docker history --no-trunc <image> for secrets in ENV or build ARG lines.

Found one: rotate first, then clean up

 FINDING
   │ 1. ASSUME COMPROMISED. Rotate or disable it now.
   │    Deleting the line doesn't un-leak it.
   ▼
   │ 2. CHECK USE. CloudTrail, MongoDB audit log, PAM
   │    access log: who used it since it was committed?
   ▼
   │ 3. REPLACE with an identity: IRSA / Pod Identity,
   │    Airflow secrets backend, GitHub OIDC (L19)
   ▼
   │ 4. PREVENT. Push protection, a pre-commit hook,
   │    a CI scan that fails the build
   ▼
     5. History rewrite only if policy needs it; it is
        cleanup, not containment (forks, clones, caches)

The order matters. Teams love to start at step 5 because it feels like fixing, but the window of danger is open until step 1. Use --redact when you share scan output, so the report itself isn't a new leak. For the Uber-style case, step 2 is the big one: if a PAM or vault admin credential leaks, check which secrets were read with it, and rotate all of those too.

🛠 Try it (~5 min, your own repo)

Pick one repo your team owns. Run git log -p | grep -nEi 'password|secret|token' | head to see what history holds, or gitleaks if you can install it. For any real hit, follow the five steps above, and tell the repo owner instead of pasting the value into chat.

🧠 Extra takeaways

• Secrets hide in history, images, logs, DAGs, and chat, not just in today's code.
• Scan only what you own; verification modes send the secret over the network.
• A found secret is a compromised secret: rotate first, clean history later.
• If the leaked credential unlocks a vault, everything in the vault is in scope.