L62 · SIEM and detection engineering

Module 9: Detection and Incident Response · ~20 min · Lesson 62 of 71

🎯 What you'll learn

• What a SIEM does with your logs: collect, normalize, correlate, alert
• How to write a detection as a Sigma rule for a CloudTrail event, and keep it in git with tests
• How to tune false positives and set severity so alerts stay worth reading
• How an analyst triages an alert, and how to measure detections with MTTD and precision

🔥 Why this matters

At 02:10 on a Saturday, someone signed in to the prod AWS account through Identity Center and read 40 Secrets Manager secrets, including the MongoDB admin password. CloudTrail recorded every call within minutes (L52). The team found out on Tuesday, during an unrelated audit: nobody had written a rule that looked. Detection engineering turns "it's in the logs" into "the right person was told within minutes, with enough context to act".

1. What a SIEM does

A SIEM (security information and event management system) gathers logs, makes them comparable, runs rules over them and raises alerts. Splunk, Elastic Security, Microsoft Sentinel and Google SecOps are common ones.

 CloudTrail  EKS audit  Kong http-log  app events  GuardDuty
     │           │            │             │           │
     └───────────┴─────┬──────┴─────────────┴───────────┘
                       │ 1. COLLECT: ship off the box (L60)
                       ▼
        ┌──────────────────────────────────────┐
        │ 2. NORMALIZE                         │
        │ sourceIPAddress ──┐                  │
        │ client_ip       ──┼──▶ source.ip     │
        │ remote_addr     ──┘    times in UTC  │
        └──────────────────┬───────────────────┘
                           │ store and index
                           ▼
        ┌──────────────────────────────────────┐
        │ 3. CORRELATE: rules run on a         │
        │ schedule or on the stream; join      │
        │ sources; add owner and data class    │
        └──────────────────┬───────────────────┘
                           │ 4. ALERT: a rule matched
                           ▼
              queue or page, with a severity

Normalization maps each source's field names onto one schema (Elastic's ECS, Splunk's CIM, or OCSF, which Amazon Security Lake uses), so one search covers every source. Correlation joins events across sources and time: "the IP that failed 30 logins at Kong, then got a token".

One catch: most SIEMs charge by volume ingested, so teams quietly drop noisy sources like S3 data events. Decide that on purpose; a source you don't collect can't alert.

2. Writing a detection

A detection is a question you ask the logs over and over. Four common shapes: a match (one event is enough, like StopLogging), a threshold (50 failed logins from one IP in 5 minutes), a sequence (a new access key, then its first use from an unknown IP) and an anomaly (this role's first call from a new country).

Back to Saturday. Prod workloads read secrets with their own roles (IRSA or Pod Identity, L48, L50), so a person reading one through an SSO role is rare, and it's what an attacker with a stolen SSO session does (ATT&CK T1555.006, L61).

Sigma is an open, vendor-neutral YAML format for detection rules, started in 2017 by Florian Roth and Thomas Patzke (now the SigmaHQ project). A converter (pySigma, via the sigma CLI) turns each rule into your SIEM's query language.

title: Prod Secret Read by a Human SSO Role
id: 6f1c2a7e-3b9d-4c51-9a0e-2d7b8e4f1a63
status: test
description: A person read a Secrets Manager secret in prod
author: detections@example.com
date: 2026-11-08
tags:
  - attack.credential-access
  - attack.t1555.006
logsource:                  # which logs this rule reads
  product: aws
  service: cloudtrail
detection:
  selection:                # every key must match (AND)
    eventSource: secretsmanager.amazonaws.com
    eventName: GetSecretValue
    recipientAccountId: '111122223333'      # prod account
    userIdentity.arn|contains: ':assumed-role/AWSReservedSSO_'
  condition: selection
falsepositives:
  - Approved break-glass access (route it, don't filter it)
level: high

In a selection, keys are ANDed and a list of values is ORed. |contains is a modifier, like |startswith and |re. Identity Center roles are always named AWSReservedSSO_<PermissionSet>_<suffix>, which makes this rule cheap and precise.

3. Detection-as-code

Detection-as-code means rules live in git and follow the same flow as application code.

 engineer               GitHub                       SIEM
    │ 1. rule + sample     │                           │
    │    events (tp, fp)   │                           │
    │ ──────── PR ───────▶ │ 2. CI: lint the rule,     │
    │                      │    convert it, run it on  │
    │                      │    samples: tp must match │
    │                      │    and fp must not        │
    │ ◀──── 3. review ──── │                           │
    │                      │ 4. merge: deploy via API ▶│
    │                      │                           │ 5. alerts
    │ ◀──── 6. triage outcome: false positive? ─────── │
    │       tune the rule and add that event as a test │

Tests make tuning safe. Each rule ships with an event that must match and one that must not; each excluded false positive becomes a new must-not-match test, and the attack sample proves you didn't blind the rule. Splunk's contentctl and Elastic's detection-rules repo are vendor versions of the same pattern.

4. Tuning, severity and alert fatigue

Every alert ends with one of these outcomes:

Tune with exclusions on fields an attacker can't control: role ARN, account, secret name. Never exclude on userAgent or a header, which the caller writes. Give every exclusion an owner and a review date, and never switch a rule off to silence one source.

Severity combines impact and confidence: a confident rule on prod financial data pages someone, a shaky one on a dev account opens a ticket, and informational rules only feed correlations. Alert fatigue, when people stop reading because most alerts are noise, helped Target miss its breach (L13's deep dive covers the fixes). If nobody would act on an alert, it shouldn't page.

5. Triage and measurement

Triage is the analyst's first look: is this real, and how bad? It ends at a decision; if it's real, L63's incident process takes over.

 ALERT: Prod Secret Read by a Human SSO Role (high)
   │
   ▼ 1. UNDERSTAND  what is this rule looking for?
   ▼ 2. ENRICH      who is this person? on call? a change
   │                ticket open? what does the secret unlock?
   ▼ 3. SCOPE       what else did this session and IP do,
   │                one hour either side? (CloudTrail, Kong)
   ▼ 4. DECIDE
   ├── false positive ────────▶ close; tune rule, add test
   ├── benign true positive ──▶ close with the ticket ID
   └── true positive / unsure ▶ escalate as an incident (L63)

MTTD (mean time to detect) is the average time from the first malicious event to the alert (teams differ on the start point, so state yours). It sums log delivery (CloudTrail averages about 5 minutes, not guaranteed), ingestion and the rule's schedule: an hourly search adds up to an hour. Record every triage outcome; it's what you measure with.

Precision is true positives ÷ (true positives + false positives): when it fires, how often is it right? Recall is the share of real attacks it catches, which you can only estimate by simulating attacks, for example with Datadog's open-source Stratus Red Team in a sandbox account you own.

🧪 Worked example: tuning the secret-read rule

Week one in prod, the rule raises 31 alerts:

Precision is 2 ÷ (2 + 26) ≈ 7% (benign ones tracked separately), so analysts will learn to ignore it. The fix is narrow:

  filter_rotation:                      # owner: platform team
    userIdentity.arn|contains: '/AWSReservedSSO_PlatformAdmin_'
    requestParameters.secretId|startswith: 'prod/manual-rotation/'
  condition: selection and not 1 of filter_*

Both fields must match, so the platform team reading other secrets still alerts. (secretId is whatever the caller passed, name or ARN; an ARN still alerts, the safe direction.) One rotation event becomes a must-not-match test. Break-glass stays unfiltered: it should always alert, routed as "verify the ticket". The real fix is upstream: automate the rotation so humans never read those secrets, then delete the filter. Every exclusion is a small blind spot.

🛠 Try it (optional, ~5 min)

Lint and convert the rule locally in a throwaway virtualenv. (Tested with Python 3.13; on 3.15, pip failed to build the google-re2 dependency.)

mkdir -p ~/tmp/sigma-l62 && cd ~/tmp/sigma-l62
python3 -m venv venv && . venv/bin/activate
pip install -q sigma-cli && sigma plugin install splunk

Save section 2's rule as aws_prod_secret_read_by_human.yml (output from sigma-cli 4.0.0, pySigma 2.0.2, Splunk backend 3.0.0; wrapped):

$ sigma check aws_prod_secret_read_by_human.yml
issue=NumberAsStringIssue severity=low ... string=111122223333
Found 0 errors, 0 condition errors and 1 issues.

$ sigma convert -t splunk --without-pipeline \
    aws_prod_secret_read_by_human.yml
eventSource="secretsmanager.amazonaws.com"
eventName="GetSecretValue" recipientAccountId="111122223333"
userIdentity.arn="*:assumed-role/AWSReservedSSO_*"

The low issue is deliberate: account IDs can start with 0, so keep them quoted. Real use swaps --without-pipeline for a processing pipeline that maps fields to your SIEM's schema. Now add the worked example's filter, convert again, and check both conditions sit inside the new NOT (...) clause.

🧠 Key takeaways

• A SIEM collects, normalizes, correlates and alerts; a source you don't collect can't alert.
• Write rules in Sigma, keep them in git, and test each with must-match and must-not-match events.
• Exclude on fields the attacker can't control, with an owner and a review date; fix noise at the source when you can.
• Severity is impact × confidence; if nobody would act on an alert, it shouldn't page.
• Measure MTTD and precision per rule, and estimate recall by simulating attacks.

📖 Words to know

⏭ Next time

L63: the alert was real. The NIST incident response lifecycle, who does what, and how to contain a compromised pod without destroying evidence.

Most rules fight a trade-off: catch more and you alert more on harmless activity. A honeytoken sidesteps it. It's a fake credential or record that no legitimate process ever touches, so any use of it is suspicious by definition. Precision close to 100%, and almost no tuning.

Kinds you can plant in a typical API stack

 PLANT                          WHERE IT ALERTS
 ───────────────────────────    ─────────────────────────────
 AWS access key for a user      CloudTrail: any call with
 with no permissions            that accessKeyId
 fake customer C-90417733       Kong http-log: any request
 in MongoDB                     whose URI contains the ID
 MongoDB user report_ro whose   MongoDB log: any login
 password is in a config file   attempt as that user
 internal-only decoy route      Kong http-log: any hit on
 /v1/internal/export            that route

The AWS key is the classic. Create an IAM user with no policies (or an explicit deny-all), make an access key, and leave it somewhere an intruder looks: a jump box's ~/.aws/credentials, a config file in a private repo, an Airflow Connection nobody uses. CloudTrail records calls even when they fail with AccessDenied, and sts:GetCallerIdentity, the attacker's usual first call (L52), needs no permissions and is logged too.

Worked example: the canary key rule

title: Canary AWS Access Key Used
id: d41f7a36-95c2-4e08-b1d7-8a3e6c2f0b59
status: stable
logsource:
  product: aws
  service: cloudtrail
detection:
  selection:
    userIdentity.accessKeyId: AKIAEXAMPLEHONEY0001
  condition: selection
falsepositives:
  - None expected; nothing legitimate holds this key
level: critical

Converted, it's a one-field search (userIdentity.accessKeyId="AKIAEXAMPLEHONEY0001"). Run it on the event stream (an EventBridge rule on CloudTrail events) rather than an hourly search, so MTTD is minutes. Read-only calls such as GetCallerIdentity only reach EventBridge if the rule is set to receive all CloudTrail management events (an option AWS added in 2023), so check yours. The alert tells you more than "someone is in": the key was only in one place, so you know which place was read. Plant different keys in different places.

The fake customer works the same way, with one twist. Don't watch for reads in MongoDB's audit log, because batch jobs (such as Airflow full-load DAGs) read every document in the collection, including the fake one. Watch the API path in Kong instead, where only targeted access happens. And if the data feeds billing or reporting, register the record as fake with those teams, so it never gets a statement or lands in a regulatory report.

How attackers spot them

Honeytokens aren't invisible, so make yours look boring:

• Names leak. An AccessDenied message includes the caller's ARN, so a user called honeytoken-user gives the game away. Use something like svc-backup-sync.
• Account IDs leak. sts:GetAccessKeyInfo returns the account a key belongs to, and AWS logs that call in the caller's account, not yours. Keys from well-known free canary services sit in known accounts, and some secret scanners (TruffleHog, for one) recognize them. A self-managed key in one of your normal accounts is harder to spot.
• Unlogged APIs. Researchers have repeatedly found AWS calls that CloudTrail didn't record and whose error messages revealed the key's ARN. AWS has closed many of these; assume a careful attacker may still find one, and treat honeytokens as one layer, not the plan.
• Never plant a working key in a public repo. Bots scrape public GitHub within minutes, so you'd drown in alerts from people who aren't targeting you. GitHub secret scanning also reports exposed AWS keys to AWS, which may quarantine the key.

🧠 Extra takeaways

• Any use of a honeytoken is suspicious, so its rule needs almost no tuning.
• Plant a different token in each place, so the alert tells you which place was read.
• Detect fake records at the API, not the database, where batch jobs read everything.
• Make tokens look ordinary: plausible names, your own accounts, never public repos.

A password spray is many failed logins, each one ordinary. The attack only shows up when you count them, and it only becomes urgent when one of them succeeds. Single-event rules can't see that. Sigma's correlation rules (added in version 2.0 of the Sigma specification, 2024) can.

The four correlation types

Each base rule gets a name; correlation rules refer to those names, can refer to other correlations, and group results with group-by inside a timespan. Check the current specification for newer types, and check what your backend actually supports.

Worked example: spray, then success

Kong's http-log doesn't contain the request body, so the username comes from the login service's own security events (L60). The custom logsource below needs a processing pipeline to map it to your index. Several rules can share one file, separated by ---:

title: Login Failed
name: login_failed
logsource: {product: example-auth, service: security}
detection:
  selection: {event: login, outcome: failure}
  condition: selection
level: informational
---
title: Login Succeeded
name: login_ok
logsource: {product: example-auth, service: security}
detection:
  selection: {event: login, outcome: success}
  condition: selection
level: informational
---
title: Password Spray From One IP
name: spray
correlation:
  type: value_count
  rules: [login_failed]
  group-by: [client_ip]
  timespan: 10m
  condition: {field: user, gte: 20}
level: medium
---
title: Spray Followed by a Successful Login
correlation:
  type: temporal
  rules: [spray, login_ok]
  group-by: [client_ip]
  timespan: 30m
level: high

(Each real rule also needs an id and status; they're cut here for space.) Converting this with the Splunk backend (pySigma Splunk backend 3.0.0) produced a multisearch whose core is:

| bin _time span=10m
| stats dc(user) as value_count by _time client_ip
| search value_count >= 20
...
| bin _time span=30m
| stats dc(event_type) as event_type_count by _time client_ip
| search event_type_count >= 2

Read the generated query

Two things in that output matter more than the YAML.

First, bin _time span=30m cuts time into fixed buckets. A spray ending at 10:28 and a success at 10:31 fall in different buckets, so the rule misses them. That's the same boundary problem as fixed rate-limit windows (L42). Some SIEMs support sliding windows; check what yours generates and test an event pair that straddles a boundary.

Second, the same backend rejected temporal_ordered outright ("not supported by backend"), which is why the example uses temporal. Support varies by backend and version, and a conversion that silently changes meaning is worse than one that fails, so put a straddling sample in your tests.

Also know the rule's blind spot: grouping by client_ip misses a spray spread across hundreds of IPs. A second rule grouped by user (many IPs failing for one user) covers the other direction. And client_ip must be the real client, taken from a trusted X-Forwarded-For hop (L43), or the attacker chooses it.

🧠 Extra takeaways

• Some attacks only show up as counts or sequences; Sigma correlation rules express them portably.
• value_count counts distinct values, which is what separates a spray from one user's typos.
• Read the converted query: fixed time buckets and unsupported types change what the rule catches.
• Group by IP and by user, and make sure the IP field can't be set by the client.