L63 · The incident response lifecycle

Module 9: Detection and Incident Response · ~20 min · Lesson 63 of 71

🎯 What you'll learn

• The NIST lifecycle: the classic four phases (SP 800-61 Rev. 2) and how Rev. 3 (2025) re-maps them onto CSF 2.0
• Who does what in an incident: incident commander, comms lead, scribe and the experts
• How to contain a compromised pod on EKS without destroying the evidence
• Why credentials must be revoked, not just "rotated later", and why post-mortems are blameless

🔥 Why this matters

02:14, Saturday. A detection rule (L52, L62) fires: the payments-api pod's IAM role credentials are listing S3 buckets from 203.0.113.45, outside AWS. The on-call's first instinct is kubectl delete pod. Now the attacker's process, files and memory are gone, and the copied credentials still work until they expire. Incident response is doing the right things, in the right order, under pressure.

1. The lifecycle: two versions of the same idea

An incident is an event that actually or potentially harms the confidentiality, integrity or availability (L01) of your systems or data. Most alerts (L62) aren't. Someone has to declare one, which starts a process everyone already knows.

For thirteen years the reference was NIST SP 800-61 Rev. 2 (2012). Its lifecycle has four phases, drawn as a loop:

 ┌─────────────┐   ┌─────────────┐   ┌────────────────┐
 │ 1. PREPARE  │──▶│ 2. DETECT & │◀─▶│ 3. CONTAIN,    │
 │ plans,tools,│   │   ANALYZE   │   │   ERADICATE &  │
 │ access,logs │   │ real? scope?│   │   RECOVER      │
 └──────▲──────┘   └─────────────┘   └───────┬────────┘
        │   lessons feed the next round      │
        │        ┌──────────────────┐        │
        └────────│ 4. POST-INCIDENT │◀───────┘
                 │ ACTIVITY         │
                 └──────────────────┘

Phases 2 and 3 loop: containing one host often reveals another.

In April 2025 NIST replaced it with SP 800-61 Rev. 3, now titled a CSF 2.0 Community Profile. NIST's reasoning: incidents are no longer rare, one-day events for a separate team; they run for weeks and touch everything. So Rev. 3 maps incident response onto the six functions of the NIST Cybersecurity Framework 2.0 (L67 covers the CSF itself):

 TOP: INCIDENT RESPONSE
   DETECT ───────▶ RESPOND ───────▶ RECOVER
   find, analyze   manage, analyze, restore assets,
                   contain, report  communicate
          ▲                ▲               ▲
 MIDDLE:  └──── IMPROVEMENT (ID.IM) ───────┘
          lessons from every function, all the time
          ▼                ▼               ▼
 BOTTOM: PREPARATION (ongoing risk management)
   GOVERN          IDENTIFY         PROTECT
   policy, roles   assets, risks    safeguards

Two changes matter. Lessons learned are continuous, not a meeting at the end. And Rev. 3 drops step-by-step handling procedures, which NIST says change too fast for a static document. It also says to use whichever model suits you, so many playbooks (and this lesson) keep Rev. 2's phase names for what you actually do.

2. Prepare, then detect and analyze

What makes a 2 a.m. incident go well happens months earlier:

• Playbooks for likely incidents: leaked AWS key, compromised pod, ransomware (L64), data exposed through an API.
• Access: a tested break-glass role for responders, so nobody waits for an access request.
• Evidence that exists: CloudTrail in a log archive account (L52), EKS audit logs on (L45), application logs with correlation IDs (L60).
• Severity levels, so "declare a SEV1" means the same thing to everyone.

When an alert fires, analysis answers three questions fast. Is it real? What's the scope (which identities, workloads, data)? What's the severity? Write down every finding with a UTC timestamp. If it's plausibly a real compromise of production or customer data, declare. Declaring a false alarm costs an hour; not declaring a real one costs days.

3. Roles: one person decides, one writes, one talks

The roles come from the fire service's Incident Command System:

                ┌─────────────────────┐
                │ INCIDENT COMMANDER  │ decides, assigns, sets
                │ (IC)                │ the next check-in time
                └──┬──────┬───────┬───┘
       tasks       │      │       │   updates
       ┌───────────┘      │       └──────────────┐
 ┌─────▼─────┐      ┌─────▼─────┐       ┌────────▼────────┐
 │ TECH LEAD │      │ SCRIBE    │       │ COMMS LEAD      │
 │ + experts │      │ timeline, │       │ execs, support, │
 │ (EKS,     │      │ decisions,│       │ legal/compliance│
 │ AWS, DB)  │      │ evidence  │       │ (regulators,    │
 └───────────┘      └───────────┘       │ customers)      │
                                        └─────────────────┘

• The incident commander (IC) coordinates and decides; they do not type commands. An IC who starts debugging stops commanding.
• The tech lead and experts investigate and act, checking with the IC before destructive steps.
• The scribe keeps the timeline: who did what, when, and why. It becomes the post-mortem draft.
• The comms lead keeps everyone else informed, so engineers aren't interrupted.

The clocks are why legal and compliance join early in a regulated company such as a bank. A public company must file an SEC Form 8-K Item 1.05 within four business days of deciding an incident is material, and NYDFS Part 500 requires notice within 72 hours of determining that certain incidents occurred. L68 covers these rules; for now, know that someone must track them from hour one.

4. Contain, eradicate, recover without destroying evidence

Containment stops the damage spreading. Eradication removes the attacker and the hole they used. Recovery brings clean systems back. The trap is doing them in the wrong order.

The rule from RFC 3227 is the order of volatility: collect the most short-lived evidence first.

 most volatile ─────────────────────────────▶ least volatile
 process memory, ─▶ container ─▶ node disk ─▶ CloudTrail,
 open connections   filesystem   (EBS)        S3 logs
 (gone on delete)   (gone on     (snapshot    (archived,
                    delete)      it)          L52)

On EKS that means isolate, don't delete. Take the pod out of its Service and ReplicaSet by changing its labels (Kubernetes documents this for debugging), block its traffic with a NetworkPolicy, cordon its node, and snapshot the node's EBS volume. The Deployment starts a fresh replacement, so customers barely notice.

Credential revocation is containment too. Deleting a pod doesn't revoke the STS credentials already copied out of it: they stay valid until they expire. For an IAM role, "Revoke active sessions" adds a deny for every session issued before now. For IAM users, deactivate access keys. In Identity Center, disabling a user doesn't end role sessions they already hold. Then rotate every secret the workload could read (L48).

Eradication means fixing the root cause (the vulnerable image, the leaked key) and hunting for persistence: new access keys, changed trust policies, EKS access entries (L52). Recover from known-good images and keep watching, because attackers often come back.

5. Post-incident: the blameless post-mortem

A post-mortem is a written review that explains what happened and changes something. It's blameless: it asks why the system let this happen, not who messed up. Blame teaches people to hide mistakes, and hidden mistakes repeat.

❌ "Dev X committed the key to git." ✅ "A key could reach a public repo because no pre-commit or CI secret scan existed (L57), and the key was long-lived because the job didn't use OIDC."

A good post-mortem has a UTC timeline (from the scribe), impact, root cause and contributing factors, what went well, and action items with owners and due dates. Under Rev. 3, those lessons feed Improvement (ID.IM) and so the next Preparation.

🧪 Worked example: quarantining a pod on EKS

The pod is payments-api-7d9f8c6b5-x2kqp in payments. Evidence first, then isolation.

P=payments-api-7d9f8c6b5-x2kqp; NS=payments; T=INC-20261107-01
# 1. Record it before changing anything
kubectl -n $NS get pod $P -o yaml > $T-pod.yaml
kubectl -n $NS logs $P --all-containers --timestamps > $T-pod.log

# 2. Pull it out of the Service and ReplicaSet
#    ("app-" removes the label)
kubectl -n $NS label pod $P app- quarantine=true incident=$T

# 3. Deny all traffic to and from quarantined pods
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: quarantine, namespace: payments}
spec:
  podSelector: {matchLabels: {quarantine: "true"}}
  policyTypes: ["Ingress", "Egress"]    # no rules = none allowed

NetworkPolicies are additive (L51), so check no other policy still selects the pod and allows traffic. Whether connections that are already open get cut depends on your CNI.

# 4. Stop new pods landing on the node, then snapshot its disk
N=$(kubectl -n $NS get pod $P -o jsonpath='{.spec.nodeName}')
kubectl cordon $N
aws ec2 create-snapshot --volume-id vol-0a1b2c3d4e5f67890 \
  --description "$T node $N" \
  --tag-specifications \
  'ResourceType=snapshot,Tags=[{Key=incident,Value=INC-20261107-01}]'

// 5. Revoke the pod role's existing sessions (what the console's
//    "Revoke active sessions" adds, as AWSRevokeOlderSessions)
{"Effect": "Deny", "Action": "*", "Resource": "*",
 "Condition": {"DateLessThan":
   {"aws:TokenIssueTime": "2026-11-07T02:31:00Z"}}}

Healthy pods fetch new credentials, issued after 02:31; the copied ones are dead. (Fargate has no node to cordon or snapshot; plan a different capture route.)

🛠 Try it (optional, ~5 min)

A thought exercise for one service you own (no real identifiers in your notes):

1. If its pod were compromised at 2 a.m., who would be IC? Where is the playbook?
2. Could the on-call snapshot a node and add a deny policy to the pod's role without raising an access request?
3. How long do its credentials stay valid (for example role sessions, database passwords, gateway API keys)?
4. Which logs would show what the attacker did, and are they on today?

Every "I don't know" is a Preparation action item.

🧠 Key takeaways

• Rev. 2's four phases still describe the work; Rev. 3 (2025) maps it onto CSF 2.0 and makes improvement continuous.
• One IC decides, a scribe records, a comms lead talks; the IC doesn't type commands.
• Isolate, don't delete: relabel, NetworkPolicy, cordon, snapshot, in order of volatility.
• Revoke credentials actively; deleting a pod doesn't kill credentials already stolen.
• Post-mortems are blameless and end in owned, dated action items.

📖 Words to know

⏭ Next time

L64 takes apart a ransomware attack, from a stolen VPN password to the ransom note, and shows which backups survive an attacker with admin rights.

Evidence you collect may end up in front of regulators, insurers, law enforcement or a court. If you can't show where it came from and that nobody altered it, it's worth much less. Cloud evidence is easy to collect and just as easy to collect badly.

What chain of custody means

Chain of custody is the documented history of a piece of evidence: what it is, who collected it, when, how, where it's been stored, and everyone who has handled it since. Each item gets a cryptographic hash (L14) at collection, so anyone can later prove the copy is unchanged.

 collect ──▶ hash ──▶ record ──▶ store ─────▶ analyze a COPY
 (snapshot,  SHA-256  who, when, forensics    never the
 logs, yaml) of file  source, tool account,   original
                                 locked

A forensics account

Don't leave evidence in the compromised account, where the attacker may still have admin. Keep a separate forensics account in the AWS Organization, with its own small group of users:

• EBS snapshots: share or copy the snapshot to the forensics account. Snapshots encrypted with the default aws/ebs key can't be shared, so you must first copy it with a customer managed KMS key whose policy allows the forensics account (L19). Prepare that key before an incident.
• Analysis: create a volume from the snapshot in the forensics account and attach it, read-only, to an isolated instance with no internet access. Never boot the evidence.
• Files (pod YAML, logs, memory images): store them in an S3 bucket with Object Lock (L52's deep dive) so even an investigator can't overwrite them.
• CloudTrail: already in the log archive account. Note the digest files that prove integrity; don't copy the logs around and lose that proof.

Memory: the evidence you lose first

A snapshot captures the disk, not memory. Fileless malware, decrypted secrets and the attacker's open connections live only in RAM. Capturing node memory needs a tool installed or ready to run (AVML and LiME are open-source Linux examples), and running it changes the system slightly, which you record. Many teams decide memory capture is out of scope for routine incidents; just make that decision in Preparation, not at 2 a.m.

Worked example: the evidence log

The scribe keeps one row per item. Timestamps in UTC, always.

INC-20261107-01 evidence log
ID   UTC time   Item                     Collected by  SHA-256
E01  02:36:10   pod yaml (payments-api-  on-call eng   3f9a..c41e
                7d9f8c6b5-x2kqp)
E02  02:36:40   pod logs, all containers on-call eng   b27d..09aa
E03  02:41:05   snap-0f1e2d3c4b5a69788   on-call eng   n/a: AWS ID,
                of vol-0a1b2c3d4e5f67890               tagged
E04  02:58:30   E03 copied to forensics  IC-approved   n/a: AWS ID
                acct, CMK re-encrypted
E05  03:12:00   CloudTrail 01:00-03:00   tech lead     digest valid
                (Athena query + results)

# hash files at collection; store the hash with the file
shasum -a 256 INC-20261107-01-pod.yaml INC-20261107-01-pod.log \
  > INC-20261107-01-hashes.txt

Note that rows E03 and E04 rely on AWS's own record (CloudTrail logs the CreateSnapshot and CopySnapshot calls, with who and when). That's part of the chain too.

🧠 Extra takeaways

• Hash every file at collection and record who, when, how and where it's stored.
• Move evidence to a separate forensics account; analyze copies, never originals.
• Prepare a shareable KMS key for snapshots in advance; default-key snapshots can't be shared.
• Disk snapshots miss memory; decide in advance whether you'll capture it and how.

The main lesson revoked the pod's IAM role sessions. That's one credential. A compromised pod usually had access to many, and the attacker had time to copy all of them. Eradication fails if any one keeps working.

Inventory what the pod held

 compromised pod
  ├─ IAM role via IRSA/Pod Identity ─▶ revoke older sessions
  ├─ ServiceAccount token (mounted) ─▶ bound to the pod: valid
  │                                    until the pod is deleted
  │                                    or the token expires
  ├─ Secrets mounted / env vars ────▶ rotate: Mongo password,
  │   (L48)                            Kong consumer key, API keys
  ├─ Signing keys it could use ─────▶ JWT signing key? rotate +
  │                                    publish new JWKS
  ├─ Anything reachable by network ─▶ other services, IMDS,
  │                                    internal admin APIs (L43)
  └─ What the attacker created ─────▶ new keys, users, roles,
                                       access entries (L52)

The order matters

Revoke in the wrong order and the attacker uses one credential to restore another.

1. Contain the network first (the quarantine NetworkPolicy), so the pod itself can't fetch new credentials. Once the evidence is captured, delete the quarantined pod: that invalidates its bound ServiceAccount token (L50), which a copy could otherwise use against the API server.
2. Kill the identities with the widest reach: the AWS role sessions, then anything the attacker created with them (CloudTrail by the role's ARN: CreateAccessKey, CreateUser, UpdateAssumeRolePolicy, CreateAccessEntry).
3. Rotate secrets the pod could read, starting with ones that grant data access (database, partner API keys). Rotation-safe secrets (L19's deep dive) make this quick instead of an outage.
4. Check the tokens it issued or signed. If the pod could sign JWTs, rotate the key and drop the old one from your JWKS so existing tokens fail validation (L25).

Worked example: hunting for persistence in CloudTrail

An Athena query (assuming the standard CloudTrail table, L52) for write calls made by the pod's role during the incident window:

SELECT eventtime, eventname, sourceipaddress,
       json_extract_scalar(requestparameters, '$.userName')
FROM cloudtrail_logs
WHERE useridentity.sessioncontext.sessionissuer.arn =
      'arn:aws:iam::111122223333:role/payments-api-irsa'
  AND eventtime BETWEEN '2026-11-07T00:00:00Z'
                    AND '2026-11-07T03:00:00Z'
  AND readonly = 'false'
ORDER BY eventtime;

2026-11-07T02:09:41Z  CreateUser       203.0.113.45  svc-backup2
2026-11-07T02:10:02Z  CreateAccessKey  203.0.113.45  svc-backup2
2026-11-07T02:11:30Z  PutObject        203.0.113.45  (null)

svc-backup2 is persistence: a user that survives the role revocation. Deactivate its key, record it as evidence, and only then delete it. (This role should never have had iam:CreateUser; that's an action item for the post-mortem, L46.)

🧠 Extra takeaways

• List every credential the workload held or could reach, not just its IAM role.
• Isolate the network, revoke the widest identities, remove persistence, then rotate secrets.
• Search CloudTrail for write calls by the compromised identity to find what the attacker created.
• Rotating a JWT signing key invalidates tokens it signed, once the old key leaves the JWKS.