L65 · Tabletop exercise: the Saturday night export

Module 9: Detection and Incident Response · ~20 min · Lesson 65 of 71

🎯 What you'll learn

• How to run L63's lifecycle under pressure, from the first alert to the post-mortem
• Why each fix must hit the layer the attacker is using
• How to run this exercise with your own team in an hour

🔥 Why this matters

A tabletop exercise is a rehearsal: a facilitator reveals a made-up incident one inject (a new piece of evidence) at a time, and the team talks through what they'd do. Nothing is touched. You find the gaps (no access, no runbook, no weekend legal contact) while it's free. At each stage, pick A, B, C or D before you open the debrief, then score yourself.

📋 Briefing

Northwind Mutual is a fictional life insurer licensed in New York. You're the on-call engineer for the customer-data platform. Marcus Lee is the on-call incident commander (IC).

 Northwind Mutual, prd account 111122223333 (us-east-1)

 engineer laptop ──SSO──▶ IAM Identity Center
                             │ PlatformEngineer permission set
                             ▼
 Secrets Manager  ◀──read── AWSReservedSSO_PlatformEngineer role
 prd/airflow/kong-reporting-key
      ▲
      │ read at run time
 MWAA (Airflow): reporting_customers_nightly, 02:00 UTC
      │ egress via NAT 203.0.113.10 and 203.0.113.11
      ▼
 Kong (key-auth, consumer "airflow-reporting")
      │ GET /v1/reporting/customers?page=N&size=500
      ▼
 customers-api (EKS) ──▶ MongoDB customers (~206,000 docs)

The reporting route returns name, birth date, address, email, policy numbers and SSN last four. CloudTrail goes to a log archive account (L52).

Stage 1: The alert

Saturday 7 November, 01:52 UTC. Your phone buzzes:

SIEM  kong-consumer-unexpected-source
consumer   airflow-reporting
source IP  203.0.113.77      (expected 203.0.113.10, .11)
route      GET /v1/reporting/customers
last 28m   70 requests, all 200, 131 MB out

The DAG runs at 02:00; it hasn't started.

• A. It's probably the DAG starting early. Snooze the alert and check at 02:30.
• B. Declare a SEV1, page Marcus, open the incident channel, and export the consumer's Kong logs.
• C. Ask in the Airflow team's channel whether anyone changed the DAG, and wait.
• D. Delete the airflow-reporting consumer in Kong yourself, and write it up Monday.

Stage 2: Stop the bleeding

02:01. Marcus is IC, Jo Park is scribe. Requests still arrive about once a minute. Marcus wants containment that's fast and reversible.

• A. Block 203.0.113.77 at Kong and the WAF.
• B. Scale customers-api to zero replicas until morning.
• C. Export the consumer's config with decK, then delete its key-auth credential. Tonight's DAG run will fail; Marcus accepts that.
• D. Rotate the key's value in Secrets Manager so the attacker's copy is out of date.

Stage 3: How did they get the key?

02:15. The attacker had the right key from their first request. You search CloudTrail:

01:08:41 sso.amazonaws.com  GetRoleCredentials
  user   priya.n@northwind-mutual.example
  role   PlatformEngineer   src 203.0.113.77
01:12:05 secretsmanager     GetSecretValue
  arn    assumed-role/AWSReservedSSO_PlatformEngineer_3f9a1c/
         priya.n@northwind-mutual.example
  src    203.0.113.77  secret prd/airflow/kong-reporting-key

Priya's last sign-in, with MFA, was Friday 17:40 from home. No new sign-in at 01:08: someone is reusing her portal session, probably a cookie stolen by malware (L22).

• A. Put Priya's identitystore:userId in the pre-staged deny SCP, disable her sign-in, end her portal sessions; then call her on the number in HR records and have IT isolate her laptop.
• B. Reset Priya's password and require MFA re-enrollment.
• C. Call Priya first, and hold all action until she confirms it wasn't her.
• D. Rotate every secret in the prd account right now, before anything else.

Stage 4: Eradicate, then recover

03:10. You list every write call (readOnly=false) by Priya's session:

01:19:30 iam  CreateUser        svc-backup-sync
01:19:52 iam  AttachUserPolicy  SecretsManagerReadWrite
01:20:14 iam  CreateAccessKey   AKIAEXAMPLEBACKUP01
01:47:03 secretsmanager ListSecrets
              by svc-backup-sync  src 203.0.113.91

The SCP blocks Priya's identity, not this IAM user. Then Finance asks in Slack: "Month-end file is due Monday 09:00. Can we switch the DAG back on?"

• A. Delete svc-backup-sync, restore the old Kong key, rerun the DAG.
• B. Restore the customers collection from Friday's backup, to be safe.
• C. Keep everything off until an outside forensics firm reports, in about two weeks.
• D. Deactivate the user's key and deny it, hunt every write by both identities, rotate what PlatformEngineer could read, then run the DAG with a new key limited to the NAT IPs.

Stage 5: Who needs to know?

Sunday 10:00. Scoping (first deep dive) is done: 41,500 customer records, every field in the briefing. The VP of Engineering writes: "Contained in 14 minutes, nothing encrypted. Let's not drag legal in on a weekend."

• A. Agree, and wait for the forensic report before involving anyone else.
• B. Page legal, compliance and privacy per the IR plan; the scribe records when the incident was determined; the comms lead drafts a holding statement; counsel decides on notices.
• C. Post the details on the public status page now, to be transparent.
• D. Script an email to the 41,500 customers tonight from the reporting service.

Stage 6: The post-mortem

Wednesday. The draft says: "Root cause: engineer installed malware on her laptop."

• A. Keep it. Actions: disciplinary review and mandatory training for Priya.
• B. Skip the post-mortem: contained in 14 minutes, so the process worked.
• C. Rewrite it as a chain of system causes, each action with an owner and a date.
• D. Make the single action item "evaluate a new SIEM".

🧪 Scoring rubric

Score 3 for the best answer, 1 for a partial one, 0 otherwise. Maximum: 18.

16+: you could be IC. 11 to 15: solid. 10 or less: reread L63 and replay next week. Where each good decision cut the attack:

 1 Fri    infostealer takes Priya's portal cookie
 2 01:08  GetRoleCredentials from 203.0.113.77 ◀─ S3 deny SCP
 3 01:12  GetSecretValue kong-reporting-key    ◀─ S4 rotate
 4 01:19  CreateUser svc-backup-sync + key     ◀─ S4 hunt
 5 01:24  bulk GET /v1/reporting/customers     ◀─ S2 kill key
 6 01:52  SIEM alert, 28 minutes in            ◀─ S1 declare

🛠 Run it with your team (~60 min)

1. Cast it: facilitator, IC, scribe, comms lead, engineers, and legal if you can.
2. Localize it to a service you own; no real hostnames, account IDs or secrets in notes.
3. Reveal one inject at a time, about 8 minutes per stage.
4. Ask "can we actually do that?" Can the on-call revoke a gateway credential (a Kong key, in this scenario) at 2 a.m.? Who's legal's weekend contact?
5. End with action items: every "we don't know" gets an owner and a date.

🧠 Key takeaways

• Declare early; snoozing a real incident costs more than a false alarm.
• Contain at the layer the attacker uses: revoke the credential, don't chase IPs.
• A password reset doesn't end live sessions; for Identity Center, deny the user's ID in an SCP.
• Hunt all persistence before recovering, and recover with tighter credentials.
• Legal joins early because notice clocks start at determination.

📖 Words to know

⏭ Next time

L66 opens the last module with the attacker's oldest tool, people: phishing, fake vendor bank changes, help-desk calls, and a deepfake video meeting that cost about US$25 million.

In Stage 5 the team said "41,500 records". Regulators, counsel and customers will ask three things: who was affected, what fields, and how sure you are. Here's how you'd get there.

Your evidence, and what it can prove

Two details trip teams up. MongoDB auditing (Enterprise and Atlas) records authentication and schema changes by default; logging reads needs auditAuthorizationSuccess: true, which MongoDB warns costs performance (L53's deep dive). And the database stores more than the API returns. If the document has a full SSN but the serializer in the deployed image returns ssnLast4, the exposed field is ssnLast4. Check the image digest that was running at 01:24 (L14), not today's code.

Worked example: from Kong pages to customer IDs

Kong's log serializer records the consumer, client IP, query string and status. Count the pages the attacker fetched successfully:

$ jq -r 'select(.consumer.username=="airflow-reporting"
    and (.client_ip|IN("203.0.113.10","203.0.113.11")|not)
    and .response.status==200)
  | .request.querystring.page' INC-20261107-01-kong.jsonl \
  | sort -n | uniq | awk 'NR==1{f=$1} END{print f, $1, NR}'
1 83 83

Pages 1 to 83, no gaps, 83 distinct pages. The API pages with sort({_id: 1}).skip(...).limit(500), so restore a snapshot from 01:24 into an isolated account and replay the same query:

// mongosh, against the point-in-time restore (isolated account)
const ids = db.customers.find({}, {_id: 1})
  .sort({_id: 1}).skip(0).limit(83 * 500)
  .toArray().map(d => d._id);
ids.length            // 41500

Skip-based paging has a catch: a customer inserted during the 42-minute export shifts every later page by one. The oplog shows 37 inserts in that window, so the boundary records are "possible", not "confirmed".

Write the answer with its confidence

Affected: 41,500 customers CONFIRMED (IDs in INC-...-ids.csv)
          37 customers POSSIBLE (page-boundary shift)
Fields:   name, DOB, address, email, policy numbers, SSN last 4
Basis:    Kong logs + PITR replay + image sha256:9c1e... serializer
Not seen: no other consumer or route used by 203.0.113.0/24

Counsel usually notifies the possible group too: the cost of over-notifying 37 people is small, and being wrong in the other direction isn't. A design lesson for next time: for bulk routes, log the returned IDs (never the data) under the correlation ID, and scoping becomes a lookup.

🧠 Extra takeaways

• Scope means who, which fields, and how sure; write all three down.
• The API's serializer, not the database document, defines which fields were exposed.
• MongoDB doesn't audit reads by default; your API and gateway logs are often the only record.
• Skip/limit paging makes boundaries fuzzy; report confirmed and possible separately.

Stage 5 was about who to tell. This is about how to talk during an incident, starting with where.

Is the attacker in your chat?

Infostealers don't take one cookie. They take every session in the browser: AWS, Slack, email, Jira. If Priya's laptop was compromised, the attacker may be able to read the incident channel they're being hunted in.

 stolen browser profile
   ├── AWS access portal session  ─▶ cut in Stage 3
   ├── Slack session              ─▶ can read #inc-1107?
   ├── email session              ─▶ can read legal's mail?
   └── GitHub session             ─▶ can push code?

So: revoke the user's sessions in every SSO-connected app, not only AWS (most IdPs and Slack let admins sign a user out everywhere). For a serious incident, move the core team to an out-of-band channel: a bridge line and chat space set up in advance, on accounts that don't depend on the compromised identity provider. If the IdP itself might be compromised (L24's deep dive), this is the only safe channel.

The status update

Updates on a fixed rhythm stop executives from pinging engineers. A common shape, every 60 minutes for a SEV1, even if nothing changed:

INC-20261107-01 · SEV1 · update 4 · 04:00 UTC · next 05:00
Status:  contained, eradication in progress
Known:   one API consumer key misused 01:24-02:06; key revoked
         backdoor IAM user found and disabled
Unknown: number of customers affected (scoping, ETA 10:00)
Needs:   legal contact for Sunday; DBA for a PITR restore
IC: Marcus Lee  Scribe: Jo Park

Separate known from unknown. "We have no evidence of X" is honest; "X didn't happen" is a promise you may have to retract.

The holding statement

A holding statement is a short, pre-approved answer for anyone outside the team (support, partners, a reporter) while facts are still moving. Write the template before the incident, and get counsel to approve the final words:

"We're investigating a security incident affecting one of our systems. We acted quickly to contain it and are working with outside experts. If customers need to take any action, we will contact them directly. We won't ask for passwords or payment details by email or phone."

The last sentence matters: attackers often phish customers of a breached company, pretending to be the company.

Privilege: let counsel set it up

Many firms have outside counsel hire the forensics firm, hoping the report stays privileged. That isn't automatic: in 2020 a US federal court ordered Capital One to hand over its forensic report, partly because the work was arranged like ordinary business work. This isn't legal advice. The engineer's takeaway is simple: write facts, not guesses or blame, in every channel, and follow counsel's instructions on what goes where.

🧠 Extra takeaways

• Assume a compromised user's every browser session is compromised, chat included.
• Keep an out-of-band channel ready that doesn't depend on your normal identity provider.
• Status updates on a fixed rhythm, with known and unknown kept apart, protect the responders.
• Holding statements are written in advance and say only what's confirmed.