L60 · Logging for security

Module 9: Detection and Incident Response · ~20 min · Lesson 60 of 71

🎯 What you'll learn

• Which events a service must log, and the five fields every security event needs
• What must never reach a log, and how a redaction filter catches what slips through
• How structured JSON logs and correlation IDs let you follow one request from the gateway (Kong in the examples) through every service
• How attackers abuse logs (forged lines, Log4Shell-style lookups), and how to keep logs trustworthy and kept for the right time

🔥 Why this matters

Logging fails in two directions. In 2018 Twitter urged all its users to change their passwords after a bug wrote passwords to an internal log before hashing them. In 2024 Ireland's data protection regulator fined Meta €91 million over passwords found stored in plain text internally in 2019. Too much logging turns your logs into a breach. Too little means that when a token is stolen, nobody can answer "which accounts did it touch?" OWASP lists this as A09, Security Logging and Alerting Failures (L29).

1. What to log: security events, with five fields

Application logs answer "why did my code break?" Security logs answer "who did what, and did it work?" The OWASP Logging Cheat Sheet and ASVS list the events that matter. Platform logs (CloudTrail, EKS audit, L52) can't see them; only your code knows them.

Every event needs five fields. Missing any one makes the line useless in an investigation:

 WHO      sub=C-2001, client=mobile-app, ip=203.0.113.24
 WHAT     action=statement.read, resource=STM-88213
 WHEN     2026-11-07T10:42:13.512Z  (UTC, synced clock)
 WHERE    service=statements-svc, pod=statements-7d9f-x2k,
          route=GET /v1/statements/{id}, env=prod
 OUTCOME  denied, reason=not_owner, status=404
          + corr=7c1e9a02-5f3b-4d7e-9a41-0b6c2d8e1f55

The client gets a bare 404 (L34); the log says why. A burst of not_owner from one caller is enumeration, which L62 turns into an alert.

2. What never to log

PCI DSS lets you display at most the BIN and last four digits of a card number. For logs, last four is the safe default.

Secrets rarely get logged on purpose. They leak through side doors:

• Exception messages. MongoDB's duplicate-key error includes the value: E11000 ... index: ssn_1 dup key: { ssn: "123-45-6789" }.
• Whole objects. log.info("req={}", request) calls toString(), which may include every field.
• URLs. A token in a query string (?access_token=...) ends up in gateway (Kong), load balancer and proxy logs.
• Debug wire logging. HTTP client wire logging at DEBUG (Apache HttpClient's wire logger) prints every header, Authorization included.
• Scheduler task logs. Airflow, for example, masks connection passwords and Variables with names such as password, secret or api_key, but not a token a DAG fetched and printed.

3. Structured logs and correlation IDs

A text line like Login failed for alice from 203.0.113.24 must be parsed with a regex before you can search it. A structured log writes each event as JSON, with named fields:

{"ts":"2026-11-07T10:42:13.512Z","level":"WARN",
 "event":"authz_fail","service":"statements-svc",
 "user":"C-2001","resource":"STM-88213","outcome":"denied",
 "reason":"not_owner","corr":"7c1e9a02-5f3b-4d7e-9a41-0b6c2d8e1f55",
 "trace_id":"4bf92f3577b34da6a3ce929d0e0e4736"}

Spring Boot 3.4+ can write JSON natively (logging.structured.format.console=ecs), or use logstash-logback-encoder; in Python, structlog or python-json-logger. Agree on field names across teams (ECS or OpenTelemetry).

A correlation ID is one ID attached to every log line a single request produces, in every service it touches. At the gateway, Kong's correlation-id plugin adds one (here X-Correlation-ID, as in L43), and tracing adds a W3C traceparent header (00-<trace-id>-<span-id>-<flags>):

 client
   │ GET /v1/statements/STM-88213
   │ (X-Correlation-ID from the client? strip it, see below)
   ▼
 Kong ── correlation-id plugin: X-Correlation-ID: 7c1e9a02-...
   │     opentelemetry plugin:  traceparent: 00-4bf9...-00f0...-01
   │     http-log: corr=7c1e9a02-...
   ▼
 statements-svc ── reads header → logging context (MDC)
   │               every log line: "corr":"7c1e9a02-..."
   │ calls downstream, FORWARDS both headers
   ▼
 accounts-svc ──── same IDs in its logs ──▶ MongoDB query log

In Java, a servlet filter puts the ID in SLF4J's MDC (mapped diagnostic context); in Python, a contextvars variable plus a logging filter does the same.

The security catch: a header the client sent is attacker input. Kong's correlation-id plugin keeps an existing header instead of replacing it. A client can then reuse a victim's ID to muddle an investigation, or send a newline or 10 KB of junk. On public routes, remove the incoming header first (for example with request-transformer) and let Kong generate a fresh one. Services should still accept only a well-formed UUID. The W3C spec likewise says you may restart traces at trust boundaries, and must keep personal data out of trace headers.

4. Log injection: when the input writes the log

A login form takes this username, with a newline inside it:

 username = "alice\nINFO login ok user=admin"

 text log, before:              JSON log, after:
 INFO login failed user=alice   {"msg":"login failed user=alice
 INFO login ok user=admin         \nINFO login ok user=admin"}
      └─ forged line                  └─ one event, \n escaped

That's log injection (CWE-117). It can fake events, hide real ones, or break your parser. JSON encoders escape \r and \n for free. For text logs, encode them: Log4j 2 %enc{%m}{CRLF}, or Logback %replace(%msg){'[\r\n]', '_'}. Also remember who reads logs: a web log viewer that renders <script> has stored XSS (L32), and a terminal will act on ANSI escape codes.

The worst case is a logger that interprets the data. Log4Shell (L36) happened because Log4j 2 expanded ${jndi:...} lookups inside logged messages. Keep log4j-core on a current 2.x release, and treat any logging feature that evaluates expressions in messages as a bug.

5. Tamper resistance and retention

An attacker who gets in will try to delete the evidence. Logs are only trustworthy if they leave the place the attacker can reach, quickly:

 pod stdout ─▶ node file ─▶ Fluent Bit ─▶ CloudWatch Logs / S3
 (app can     (rotated,     (DaemonSet)    in a log archive
  write)       root can                    account: services
               edit)                       can write, nobody
                                           in prod can delete
        seconds here ◀─────────────────▶ kept for the retention period

Ship off the node within seconds, into an account your workloads can't administer (L52's deep dive). Use UTC and synced clocks (Amazon Time Sync), or timelines won't line up.

Retention is a decision, not a default. A new CloudWatch log group keeps data forever unless you set a period. PCI DSS v4.0 (10.5.1) requires at least 12 months of audit log history, with the latest three months immediately available; a compliance team may require longer for audit trails. Privacy law pushes the other way for debug logs full of personal data. Set it per log type.

🧪 Worked example: a redaction filter as a safety net

Your first defense is not passing secrets to the logger at all. Redaction catches what slips through. In Python, attach the filter to the handler: a filter on a logger isn't applied to records from its child loggers.

PATTERNS = [
    (re.compile(r"(?i)\bbearer\s+[\w.~+/=-]+"), "Bearer [REDACTED]"),
    (re.compile(r"\b\d{3}-\d{2}-\d{4}\b"), "[SSN]"),
    (re.compile(r"\b(?:\d[ -]?){9,15}(\d{4})\b"), r"[PAN ...\1]"),
]
SECRET_FIELDS = {"password", "token", "authorization", "ssn", "pan"}

class RedactFilter(logging.Filter):
    def filter(self, record):
        msg = record.getMessage()               # merges %s args
        for rx, repl in PATTERNS:
            msg = rx.sub(repl, msg)
        record.msg, record.args = msg, None
        for k in SECRET_FIELDS & record.__dict__.keys():
            setattr(record, k, "[REDACTED]")    # extra={...} fields
        return True                             # keep, redacted

Real output (Python 3.15):

INFO upstream call Authorization: Bearer [REDACTED]
INFO kyc check ssn=[SSN] card=[PAN ...1111]

The Java equivalent with logstash-logback-encoder masks named JSON fields and regex matches:

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <decorator
      class="net.logstash.logback.mask.MaskingJsonGeneratorDecorator">
    <defaultMask>[REDACTED]</defaultMask>
    <paths>password,token,authorization,ssn</paths>
    <value>(?i)bearer\s+[\w.~+/=-]+</value>
  </decorator>
</encoder>

Limits: regexes miss formats you didn't predict and can hit innocent 16-digit order IDs (a Luhn check helps). Stack traces are formatted later, so this filter never sees them. Unit-test the filter like any other security control (L59).

🛠 Try it (optional, ~5 min)

Save the filter as redact_demo.py, adding import logging, re at the top and this at the bottom:

handler = logging.StreamHandler()
handler.addFilter(RedactFilter())
handler.setFormatter(logging.Formatter("%(levelname)s %(message)s"))
log = logging.getLogger("payments.api")
log.addHandler(handler); log.setLevel(logging.INFO)
log.info("call Authorization: Bearer eyJhbGciOi.fake.sig")
log.info("kyc ssn=%s card=%s", "123-45-6789", "4111 1111 1111 1111")
log.info("login failed user=%s", "alice\nINFO login ok user=admin")

Run python3 redact_demo.py. The first two lines come out redacted. The third prints two lines: a forged login ok. Now add .replace("\n", "\\n") to the filter's final msg and run it again.

🧠 Key takeaways

• Log authentication, authorization failures, admin actions, validation failures and high-value business actions, each with who, what, when, where and outcome.
• Never log passwords, tokens, session IDs, full PANs or SSNs; log an ID, a hash or the last four instead.
• Structured JSON makes logs searchable and escapes newlines; correlation IDs tie one request together across the gateway and every service.
• Treat incoming correlation IDs as input: strip or validate them at the edge.
• Ship logs off the box fast, to a place prod can't delete, and set retention on purpose.

📖 Words to know

⏭ Next time

L61: MITRE ATT&CK, the shared map of attacker tactics and techniques that defenders use to decide which of these log events matter most.

Every request through Kong can be logged by http-log, file-log, tcp-log and similar plugins. They all use Kong's log serializer, which builds one JSON object per request. It's your best record of who called what, and an easy place to leak credentials.

What's in the object

From Kong's documentation (fields vary a little by version):

 {
   "request":  { method, uri, url, querystring,
                 headers: {...every request header...}, size },
   "response": { status, headers, size },
   "route", "service", "consumer", "authenticated_entity",
   "client_ip", "upstream_uri", "latencies", "started_at",
   "correlation_id"   ← added when correlation-id is on
 }

Kong's PDK docs say the Authorization and Proxy-Authorization request headers are redacted by default, and Kong support says there's no setting to turn that off. Everything else is logged as sent:

Removing fields

custom_fields_by_lua runs Lua in the log phase. Kong's docs say dots in the key create nested fields and returning nil removes a field. Applying that to nested headers is the natural reading of the docs, but verify it on your version:

plugins:
- name: http-log
  config:
    http_endpoint: http://log-collector.logging.svc:8080/kong
    custom_fields_by_lua:
      request.headers.apikey: "return nil"
      request.headers.cookie: "return nil"
      request.headers.x-partner-token: "return nil"

Two traps. Fields set this way are shared with logging plugins that run later, because they edit the same log table. And the query string appears in three places, so removing querystring alone leaves the token in url and upstream_uri. The real fix is upstream: don't accept credentials in query strings at all (L22, L25).

Also set hide_credentials: true on key-auth, so Kong strips the API key before proxying and your services never see (or log) it. Because the serializer reads headers late in the request, this often removes it from Kong's own log too; check on your version.

Worked example: checking your own gateway on localhost

Run Kong in DB-less mode in Docker on your laptop with a file-log plugin writing to /dev/stdout, a dummy route, and key-auth with a fake consumer key (container name kong-l60). Then:

curl -s 'localhost:8000/demo?access_token=FAKE-QS-TOKEN' \
  -H 'apikey: FAKE-KEY-123' \
  -H 'Authorization: Bearer FAKE.JWT.VALUE' \
  -H 'Cookie: SESSION=FAKE-SESSION'
docker logs kong-l60 2>&1 | tail -1 | grep -oE 'FAKE[-A-Z0-9.]*'

Each FAKE... string that grep prints is a credential the gateway's logs would keep. Fix the config, rerun, and repeat until only the expected one-line request summary remains.

🧠 Extra takeaways

• Kong redacts Authorization and Proxy-Authorization, not API key headers, cookies or query strings.
• Remove fields with custom_fields_by_lua returning nil, and remember the query string appears three times.
• hide_credentials: true keeps API keys away from upstream services and their logs.
• Test what a gateway logs with fake credentials on localhost; don't assume.

L52 protects logs by putting them where nobody can delete them. Sometimes you also need to prove an audit trail wasn't edited: a regulator asks who raised a customer's transfer limit, and the answer must hold up. An application-level audit trail can make edits detectable with a hash chain.

How a chain works

Each record stores the hash of the previous record, and its own hash covers both:

  rec 1                  rec 2                  rec 3
 ┌───────────────┐      ┌───────────────┐      ┌───────────────┐
 │ event: add    │      │ event: limit  │      │ event: delete │
 │ prev: 000...0 │  ┌──▶│ prev: h1      │  ┌──▶│ prev: h2      │
 │ hash: h1 ─────┼──┘   │ hash: h2 ─────┼──┘   │ hash: h3      │
 └───────────────┘      └───────────────┘      └───────────────┘
   h_n = SHA-256( h_(n-1) + canonical JSON of event_n )

Change record 2 and its hash no longer matches. Recompute it, and record 3's prev no longer matches. To hide an edit, you must rewrite everything after it.

Worked example: a payee audit trail

GENESIS = "0" * 64

def entry_hash(prev: str, event: dict) -> str:
    body = json.dumps(event, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256((prev + body).encode()).hexdigest()

def verify(path: str) -> str:
    prev = GENESIS
    with open(path) as f:
        for n, line in enumerate(f, 1):
            rec = json.loads(line)
            if rec["prev"] != prev or \
               rec["hash"] != entry_hash(prev, rec["event"]):
                return f"BROKEN at line {n}"
            prev = rec["hash"]
    return f"OK, head={prev[:12]}"

Three events (payee.add, limit.change, payee.delete by u-ops-7) were appended, then tampered with. Real output:

original                             OK, head=d39901fb3258
"limit.change" edited to look benign BROKEN at line 2
line 2 deleted                       BROKEN at line 2
last line deleted                    OK, head=99c38bc812cb

The last row is the weakness: chopping off the end, or rewriting the whole chain, still verifies. sort_keys matters too: the hash must be computed over a canonical form, or the same event can hash differently.

Anchoring the head

To close that gap, regularly copy the latest hash (the head) somewhere the writer can't change:

• Write it every few minutes to an S3 bucket with Object Lock in the log archive account (L52). The verifier then checks that the chain still contains each anchored head.
• Or sign checkpoints with an asymmetric KMS key (kms:Sign) that the audit service can use but not export. Anyone with the public key can verify them (L17).

CloudTrail's log file validation does the same: hourly digest files hold hashes of the log files and chain to the previous digest, signed by AWS. AWS also sold a managed ledger database (QLDB) built on the same idea before retiring it in 2025. For MongoDB, L53's audit log records database operations; a hash chain records business events that only your service understands.

🧠 Extra takeaways

• A hash chain makes edits and deletions in the middle of a log detectable.
• Truncation and full rewrites still verify unless you anchor the head somewhere the writer can't touch.
• Hash a canonical serialization, or verification fails on harmless re-encoding.
• Tamper evidence complements, not replaces, storage nobody in prod can delete.