L47 · Case study: Capital One (2019)

Module 7: Cloud, Containers, and Kubernetes · ~20 min · Lesson 47 of 71

🎯 What you'll learn

• How an SSRF through a misconfigured WAF turned into AWS credentials (L35 in real life)
• Why an over-privileged IAM role, not the SSRF alone, decided how much data left (L46)
• Why "encrypted at rest" didn't help, and why the breach was found by a stranger, not an alarm
• What to check on your own gateway, node roles and S3 buckets this week

🔥 Why this matters

On 29 July 2019, Capital One announced that someone had taken data on about 100 million people in the US and about 6 million in Canada, mostly from credit card applications. The data sat in S3. Nobody broke AWS or phished anyone. A security device in front of the apps was talked into fetching its own cloud credentials, which could read far too much. A typical API stack on AWS uses the same building blocks every day: a gateway, EC2 nodes with roles, and S3.

🏢 The company and the setup

Capital One is one of the largest US credit card issuers, and in the mid-2010s became one of the first big banks to commit publicly to running on AWS. In front of some applications it ran a web application firewall (WAF) (L12) on EC2 instances. Like every EC2 instance, the WAF could call the Instance Metadata Service (IMDS) at 169.254.169.254, and it had an IAM role (an instance profile). The criminal complaint calls the role *****-WAF-Role, with part of the name redacted.

In 2019 there was only IMDSv1: a plain GET returns the role's temporary keys (L35). IMDSv2, with its session token, arrived in November 2019, four months after the disclosure.

A note on sources. [court] marks the criminal complaint and trial reporting, [C1] Capital One's statements, and [reported] press and researcher details no official source confirmed. The exact technique is the main thing in that last group: the complaint says a firewall misconfiguration let commands reach the server, and Capital One spoke of a "configuration vulnerability". SSRF to IMDSv1 is the widely accepted reading, and the WAF was reportedly open-source ModSecurity, but neither company published a technical root cause.

⏱ Attack timeline

The attack path

 ATTACKER, connecting through TOR and a VPN             [court]
   │ 1. finds a Capital One WAF on EC2 that will forward
   │    requests where it's told                     [reported]
   ▼
 WAF / REVERSE PROXY (ModSecurity, reportedly)
   │ 2. crafted request: the WAF itself fetches
   │    169.254.169.254/latest/meta-data/iam/
   │    security-credentials/<role>  (SSRF, IMDSv1)  [reported]
   ▼
 INSTANCE METADATA SERVICE
   │ 3. returns temporary keys for *****-WAF-Role    [court]
   ▼
 ATTACKER'S OWN MACHINE, using those keys
   │ 4. "ls": lists 700+ S3 buckets and folders      [court]
   │ 5. "sync": copies data out                      [court]
   ▼
 S3 (encrypted at rest, decrypted for any allowed role)
     6. ~100M US + ~6M Canadian people's data          [C1]

Steps 3 to 5 were ordinary, signed AWS API calls with valid keys, made from outside.

The dates

 2019 Mar  ── access and copying (complaint)        [court]
      │       later talks about it on GitHub, Slack and
      │       Twitter                                [court]
      Jul 17 ── tip to Capital One's responsible
      │         disclosure inbox, pointing at a GitHub
      │         file with the data                   [C1]
      Jul 19 ── Capital One confirms the intrusion   [C1]
      Jul 29 ── announcement; FBI arrests a Seattle
      │         software engineer, ex-AWS            [court]
      Nov    ── AWS launches IMDSv2
 2020 Aug  ── OCC: $80M civil penalty; Fed order
 2021 Dec  ── $190M consumer class settlement agreed
 2022 Jun  ── jury convicts on wire fraud and computer
      │       fraud counts; acquits on identity theft
      Oct  ── sentence: time served + 5 years probation

Who did it. Paige Thompson, who had worked at AWS until 2016. Prosecutors said she scanned for misconfigured AWS customers, took data from about 30 other organizations and ran crypto miners on their servers; they had asked for seven years in prison [court]. Her AWS past was much discussed, but SSRF against the metadata endpoint had been presented publicly for years.

What was (and wasn't) hit

Mostly credit card applications from 2005 to early 2019: names, addresses, dates of birth, income, credit scores and balances [C1]. Also about 140,000 US Social Security numbers, 80,000 linked bank account numbers and 1 million Canadian Social Insurance Numbers. No card numbers or login credentials, Capital One said, and tokenized fields stayed protected.

🔍 What went wrong

Who was to blame is disputed. AWS said its services worked as designed and the fault was the customer's configuration (L45's line). Critics, including a US senator, argued IMDSv1 made a common bug far too valuable. Lawsuits named both companies; both denied liability.

🛡 What would have stopped it

Any one of these breaks the chain:

• No SSRF in the WAF. Proxy only to configured upstreams, never to a host the client names.
• IMDSv2 required, hop limit 1 (L35). A GET-only SSRF can't do the token PUT, and IMDSv2 rejects token requests carrying X-Forwarded-For, a common sign of a proxy. It didn't exist yet; a host firewall rule blocking the WAF process from 169.254.169.254 did.
• A role sized to the job. A WAF needs to write its logs, maybe read its config. Not ListAllMyBuckets, and not GetObject on customer data.
• A data perimeter (second deep dive). Bucket policies that only answer requests from your VPC endpoints, and instance credentials that fail outside their VPC.
• Identity-based detection. The WAF role listing every bucket from a TOR exit node is loud.

🧪 Worked example: the role the WAF didn't need

The real policy isn't public. Here is the shape of the problem (made-up names):

{
  "Effect": "Allow",
  "Action": ["s3:ListAllMyBuckets", "s3:ListBucket",
             "s3:GetObject"],
  "Resource": "*"
}

What CloudTrail would record when the stolen keys are used (trimmed, made-up values). The session name of an instance role is the instance ID:

eventName:   ListBuckets
userIdentity.arn: arn:aws:sts::111122223333:assumed-role/
                  edge-waf-role/i-0a1b2c3d4e5f60718
sourceIPAddress:  198.51.100.23     <- not your VPC or NAT
userAgent:        aws-cli/1.16.x ...

ListBuckets is a management event, logged by default. The GetObject calls that follow are data events, logged only if you enabled S3 data events, so without them you can't say what was read (L45's second deep dive).

The fixed role does one job:

{
  "Effect": "Allow",
  "Action": "s3:PutObject",
  "Resource": "arn:aws:s3:::example-waf-logs/edge-waf/*"
}

Stolen now, these keys can only add log files. With HttpTokens=required too, the SSRF yields nothing.

🧰 Lessons for your own work

1. A gateway such as Kong is the WAF in this story. A gateway forwards requests by design. Make sure no route, plugin or header lets a client pick the upstream host, and that the gateway's pods have no AWS role they don't need.
2. Check your node roles. Pods that reach IMDS borrow the node's role. Require IMDSv2 with hop limit 1 and give pods their own roles through IRSA or Pod Identity (L50).
3. Ask "if these keys leaked, what could they read?" for every role you own (L46).
4. Turn on S3 data events for the buckets that matter, or you can't say what was read.

🛠 Try it (optional, ~5 min)

A fake IMDS and a careless proxy that forwards to whatever Host the client sends, all on 127.0.0.1, with fake keys. Save as imds_lab.py:

import json, secrets, sys, threading, urllib.request
from http.server import BaseHTTPRequestHandler as Base
from http.server import ThreadingHTTPServer as Server

V2 = sys.argv[1:] == ["v2"]          # run with: v1 or v2
TOKENS = set()
CREDS = {"AccessKeyId": "ASIAEXAMPLEFAKE00001",
         "SecretAccessKey": "fakeEXAMPLEfakeEXAMPLE"}

class Handler(Base):
    def send(self, code, body):
        self.send_response(code)
        self.end_headers()
        self.wfile.write(body.encode() + b"\n")
    def log_message(self, *args):
        pass

class FakeIMDS(Handler):
    def do_PUT(self):
        if "X-Forwarded-For" in self.headers:
            return self.send(403, "proxied PUT refused")
        token = secrets.token_hex(8)
        TOKENS.add(token)
        self.send(200, token)
    def do_GET(self):
        token = self.headers.get("X-aws-ec2-metadata-token")
        if V2 and token not in TOKENS:
            return self.send(401, "token required")
        self.send(200, json.dumps(CREDS))

class CarelessProxy(Handler):
    def do_GET(self):
        # BUG: forwards to whatever Host the client names
        url = "http://" + self.headers["Host"] + self.path
        try:
            with urllib.request.urlopen(url, timeout=3) as r:
                self.send(200, r.read().decode().strip())
        except Exception as e:
            self.send(502, f"upstream error: {e}")

for port, cls in [(19254, FakeIMDS), (18080, CarelessProxy)]:
    srv = Server(("127.0.0.1", port), cls)
    threading.Thread(target=srv.serve_forever,
                     daemon=True).start()
input(f"v2={V2}: proxy :18080, fake IMDS :19254. "
      "Enter quits\n")

Run python3 imds_lab.py v1, and in a second terminal:

curl -s -H 'Host: 127.0.0.1:19254' \
  http://127.0.0.1:18080/latest/meta-data/iam/security-credentials/waf
# {"AccessKeyId": "ASIAEXAMPLEFAKE00001", ...}

Quit, run python3 imds_lab.py v2, and repeat: upstream error: HTTP Error 401: Unauthorized. The proxy only forwards GETs, like most SSRF, so it can't fetch a token.

🧠 Key takeaways

• SSRF supplied the keys; the over-privileged role decided the size of the breach.
• A WAF or gateway that forwards anywhere is an SSRF.
• IMDSv1 made one GET worth a bank's data; require IMDSv2 with hop limit 1.
• Encryption at rest didn't matter, because the role was allowed to decrypt.
• Stolen cloud keys look like normal API calls; add network conditions and identity alerts.

📖 Words to know

⏭ Next time

L48: secrets management in practice: why a Kubernetes Secret is only base64, getting secrets from Secrets Manager or Vault into pods and DAGs, and what to do when a password lands in git history.

Why teams leave v1 on

Requiring IMDSv2 is one setting, and most modern AWS SDKs already use it. Teams still hesitate because something old might break: a vendor agent, a shell script with curl 169.254.169.254/..., an SDK from 2018 baked into an image. So IMDSv1 stays on "until we check", and the check never happens. The way out is to measure first, so the switch is boring.

 1 MEASURE        2 FIX CALLERS      3 FLIP          4 GUARD
 ──────────       ─────────────      ──────          ───────
 who still        upgrade SDKs,      HttpTokens=     new instances
 calls without    agents, scripts    required,       can't launch
 a token?         (use a token)      hop limit 1     with v1 (SCP,
 (CloudWatch,                        per instance    account default)
  packet analyzer)                   or template
        ◀──────── alarm on rejected v1 calls ───────────┘

Step 1: measure

Every EC2 instance publishes the CloudWatch metric MetadataNoToken: the number of IMDS calls made without a token. If it's zero for a few weeks, including month-end batch jobs, nothing on that box uses v1. If it isn't, AWS's open-source IMDS packet analyzer runs on the instance and names the process making the call.

Find instances that still allow v1 (read-only):

aws ec2 describe-instances \
  --filters Name=metadata-options.http-tokens,Values=optional \
  --query 'Reservations[].Instances[].InstanceId' \
  --output text
# i-0a1b2c3d4e5f60718  i-0f9e8d7c6b5a40321

AWS Config's managed rule ec2-imdsv2-check and Security Hub control EC2.8 report the same thing continuously.

Step 2 and 3: fix, then flip

Old SDKs are the usual culprit; AWS's IMDS docs list the minimum SDK versions that use v2. Then flip one instance at a time. The change applies to a running instance, no restart:

aws ec2 modify-instance-metadata-options \
  --instance-id i-0a1b2c3d4e5f60718 \
  --http-tokens required \
  --http-put-response-hop-limit 1

On EKS, don't edit nodes by hand. Set MetadataOptions in the node group's launch template, create a new version, and let the managed node group roll the nodes. Check what your AMI defaults to: AMIs marked ImdsSupport: v2.0 (Amazon Linux 2023 among them) already require tokens, but the hop limit default may be 2, which lets pods reach the node role (L35). Version-dependent: read your template rather than trusting a default.

Step 4: guard

Two guardrails stop v1 from coming back:

• Account default (per Region, since 2024): aws ec2 modify-instance-metadata-defaults --http-tokens required --http-put-response-hop-limit 1. New launches get it unless they say otherwise.
• An SCP that denies ec2:RunInstances unless the ec2:MetadataHttpTokens condition key is required. AWS's docs also show a deny using ec2:RoleDelivery below 2.0, which rejects any API call signed with role keys that were fetched over IMDSv1.

Then alarm on MetadataNoTokenRejected: after the flip, a non-zero value is either a caller you missed or someone probing, and both deserve a look.

Worked example: the Kong node group

Say a Kong data plane runs on a managed node group. MetadataNoToken is 0 on all six nodes for 30 days. You set HttpTokens: required and HttpPutResponseHopLimit: 1 in a new launch template version and update the node group; nodes roll one by one. A pod-side check after the rollout, run from a debug container in a dev namespace you own:

curl -s -m 2 -X PUT http://169.254.169.254/latest/api/token \
  -H 'X-aws-ec2-metadata-token-ttl-seconds: 60' \
  || echo "no token: hop limit blocks pods"
# no token: hop limit blocks pods

That's the Capital One chain closed at the metadata step, for every pod on those nodes.

🧠 Extra takeaways

• Measure with MetadataNoToken before flipping, so requiring IMDSv2 isn't a gamble.
• The setting applies to running instances; on EKS, change the launch template and roll nodes.
• AMI and template defaults differ, especially the hop limit; check, don't assume.
• Guard with an account default and an SCP, and alarm on MetadataNoTokenRejected.

Keys are bearer tokens

The WAF role's keys worked from a TOR exit node exactly as they did from the WAF. AWS credentials are bearer credentials: whoever holds them can sign requests, from anywhere, until they expire. A data perimeter adds the missing question, "is this request coming from where it should?", as policy conditions. AWS describes three parts: trusted identities, trusted resources and expected networks. This deep dive covers the network part.

 WITHOUT A PERIMETER               WITH A PERIMETER
 ───────────────────               ────────────────
 WAF ──keys──▶ S3  ✔               WAF ─▶ VPC endpoint ─▶ S3  ✔
                                         vpce-0a1b...
 TOR ──same keys──▶ S3  ✔          TOR ──same keys──▶ S3  ✘
                                   bucket policy: Deny unless
                                   aws:SourceVpce = vpce-0a1b...

Resource side: the bucket only answers your endpoint

A VPC gateway endpoint for S3 lets instances reach S3 without the internet, and every request through it carries aws:SourceVpce. A bucket policy can deny everything else (made-up IDs):

{
  "Sid": "OnlyThroughOurEndpoint",
  "Effect": "Deny",
  "Principal": "*",
  "Action": "s3:*",
  "Resource": ["arn:aws:s3:::example-card-applications",
               "arn:aws:s3:::example-card-applications/*"],
  "Condition": {
    "StringNotEqualsIfExists": {
      "aws:SourceVpce": "vpce-0a1b2c3d4e5f60718"},
    "BoolIfExists": {"aws:PrincipalIsAWSService": "false"}
  }
}

A request from the internet has no aws:SourceVpce at all, and the deny must still apply. It does: negated operators such as StringNotEquals treat a missing key as "not equal", so the condition is true. The IfExists form gives the same result here and makes that intent explicit to reviewers. The second condition lets AWS services that act with their own service principal (log delivery, for example) keep working. Two warnings: this also blocks your engineers' laptops and the console, so add a break-glass exception on aws:PrincipalArn; and test in a non-prod bucket first, because a wrong deny on s3:* can lock you out until the account root user removes the policy.

Identity side: instance keys that stay home

Since 2022, requests signed with EC2 role credentials from IMDS carry two condition keys: aws:EC2InstanceSourceVPC and aws:EC2InstanceSourcePrivateIPv4, the VPC and IP of the instance the keys were issued to. AWS's published SCP pattern denies any call where the instance's VPC doesn't match the VPC the request came through:

{
  "Effect": "Deny",
  "Action": "*",
  "Resource": "*",
  "Condition": {
    "StringNotEquals": {
      "aws:ec2InstanceSourceVPC": "${aws:SourceVpc}"},
    "Null": {"ec2:SourceInstanceARN": "false"},
    "BoolIfExists": {"aws:ViaAWSService": "false"}
  }
}

Null ... false limits it to instance credentials; aws:ViaAWSService exempts calls AWS services make on your behalf. The catch: aws:SourceVpc only exists when the call goes through a VPC endpoint, so instances calling AWS APIs through a NAT gateway would be denied too. Roll out endpoints first, start in one OU, and copy AWS's current example rather than this sketch. These keys apply to instance credentials; pods using IRSA or Pod Identity get ordinary role sessions, so for them, rely on the resource-side policy.

Worked example: the alarm that should have rung

Even with a perimeter, you want to know when keys leave. GuardDuty raises UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS when instance credentials are used from an IP outside AWS (and an .InsideAWS variant for another AWS account). A made-up finding, trimmed:

type:     UnauthorizedAccess:IAMUser/
          InstanceCredentialExfiltration.OutsideAWS
severity: High
resource: accessKeyId ASIAEXAMPLEFAKE00001
          principal  edge-waf-role / i-0a1b2c3d4e5f60718
action:   ListBuckets from 198.51.100.23 (not AWS)

Route it to whoever is on call, and have a runbook: revoke the role's active sessions (IAM's "Revoke active sessions" adds a deny for tokens issued before now), then find how the keys left. In 2019, that finding didn't exist; this exact breach is a good part of why it does.

🧠 Extra takeaways

• AWS keys are bearer credentials; a perimeter adds "from where?" to "who?".
• aws:SourceVpce in a bucket policy makes stolen keys useless from the internet, where requests carry no endpoint ID.
• aws:EC2InstanceSourceVPC pins instance credentials to their VPC, but needs VPC endpoints.
• GuardDuty's InstanceCredentialExfiltration findings are the alarm for exactly this attack.