L17 · Digital Signatures and MACs

Module 3: Cryptography Essentials · ~20 min · Lesson 17 of 71

🎯 What you'll learn

• Why SHA-256(secret + message) is a broken MAC, and how HMAC fixes it
• Why comparing two MACs with == can leak the right answer, one byte at a time
• Why a signed message (like a JWT) is still readable by anyone
• How to verify GitHub and Stripe webhooks correctly

🔥 Why this matters

Say your team exposes POST /webhooks/payments through an API gateway such as Kong, so a payment partner can tell payments-events when a premium has been paid. The URL isn't secret: it's in a config repo, a runbook and the partner's dashboard. If the handler trusts any request, anyone can send {"policy_id":"PL-100247","status":"PAID"}. The usual defence is an HMAC over the body, which is easy to get almost right, and "almost right" means forgeable or replayable.

1. Picking the tool: MAC or signature?

L01's Q&A compared the two: a MAC (message authentication code) uses one shared secret, so anyone who can verify can also create; a digital signature uses a key pair, so only the private-key holder can create, and anyone with the public key can verify. Here's where each one already shows up in a typical API stack:

Most webhook "signatures" are really MACs; judge by the key model, not the name. Rule of thumb: if every verifier could safely also create, a MAC is simpler and faster. If many parties verify, or you need proof a third party will accept (non-repudiation, L01), use a signature.

2. Why hash(secret + message) is broken

The obvious homemade MAC is tag = SHA-256(secret || message). Only someone with the secret can compute it, right? Not quite. SHA-256 (L14) processes input in 64-byte blocks, carrying an internal state between them, and its output is the final state. An attacker can load a valid tag as the starting state and keep hashing, never knowing the secret. This is a length extension attack: given H(secret || msg) and the secret's length, the attacker computes a valid tag for secret || msg || padding || extra.

 LEGIT   H( [secret][amount=100&to=bob][pad] ) ──▶ tag T
                                                       │
     attacker loads T as the state, hashes [&to=eve] ◀─┘

 FORGED  H( [secret][amount=100&to=bob][pad][&to=eve] ) = tag T'
            └──?───┘└ known (it was sent) ─┘└──new──┘

 ? = the secret, never seen. The server recomputes T' and accepts.

In 2009, Thai Duong and Juliano Rizzo showed Flickr's API signed requests as MD5(secret || parameters), and forged calls by extending them. MD5, SHA-1 and SHA-256 are all vulnerable. SHA-3 (whose "sponge" design never outputs its full state) and truncated variants like SHA-512/256 are not, but the real fix is a construction designed as a MAC.

3. HMAC in one picture

HMAC (hash-based MAC, RFC 2104) runs the hash twice, with the key mixed in each time:

 key ⊕ ipad ──┐
              ├─▶ SHA-256 ─▶ inner digest ──┐
 message ─────┘                             │
                                            ▼
 key ⊕ opad ────────────────────────▶ SHA-256 ─▶ HMAC tag

 ipad = 0x36 repeated, opad = 0x5c repeated (one block long)

The outer keyed hash means the tag you see is never a state an attacker can extend. Use a random key at least as long as the hash output (32 bytes for SHA-256); RFC 2104 "strongly discourages" shorter ones. And don't invent the encoding: if GitHub says "hex digest of the raw body", compute exactly that.

4. Comparing tags: constant time

The natural code to check a tag is if sent == expected. Most == implementations stop at the first byte that differs, so a wrong first byte returns slightly faster than a wrong tenth.

 expected:  a 7 3 f 9 c ...
 guess 1:   0 . . . . .   ✗ stop after 1 byte   (fastest)
 guess 2:   a 0 . . . .   ✗ stop after 2 bytes  (a bit slower)
 guess 3:   a 7 0 . . .   ✗ stop after 3 bytes  (slower again)
            └─ timing says "first byte right", move to the next

Over a network that's nanoseconds hidden in jitter, but enough samples can reveal it; in 2009 Nate Lawson found Google's Keyczar library comparing HMACs this way. A constant-time comparison checks every byte wherever the first mismatch is: hmac.compare_digest in Python, MessageDigest.isEqual in Java, crypto.timingSafeEqual in Node. GitHub's docs say it directly: "Never use a plain == operator."

5. Signing is not encrypting

Back to the L01 weak spot (integrity vs confidentiality): a MAC or signature proves integrity and authenticity, not confidentiality. A made-up HS256 JWT, split up:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9                 ← header
.eyJzdWIiOiJDLTIwNDE3Iiwicm9sZSI6ImFnZW50IiwiZXhwIjox
NzkzOTg3MTg1fQ                                       ← payload
.fCvq6wgx9GSV34wU-_GsbKLFCSj38GZl30PUHLgTGUc         ← HMAC tag

Anyone can base64-decode the middle part, no key needed:

$ echo 'eyJzdWIiOiJDLTIwNDE3Iiwicm9sZSI6ImFnZW50Iiwi'\
'ZXhwIjoxNzkzOTg3MTg1fQ==' | base64 -d
{"sub":"C-20417","role":"agent","exp":1793987185}

So a tax ID in a signed-only token or webhook is not protected. If you need both properties, combine them in the right order:

• Encrypt-then-MAC: MAC the ciphertext and check it before decrypting, as AEAD modes like AES-GCM do (L01, L15). TLS 1.2's older MAC-then-encrypt CBC suites enabled padding-oracle attacks like Lucky 13 (2013).
• Sign-then-encrypt for signatures: sign, then encrypt the signed result, hiding the signature too. RFC 7519 recommends this order for nested JWTs.

🧪 Worked example: verifying a webhook

The two formats (from GitHub's and Stripe's docs) look like this:

X-Hub-Signature-256: sha256=<hex HMAC-SHA256(secret, raw body)>

Stripe-Signature: t=1793983585,v1=<hex HMAC-SHA256(
                  secret, "1793983585" + "." + raw body)>

GitHub signs only the body. Stripe also signs a timestamp, and sends one v1= per active secret during a rotation.

Vulnerable (Flask, running in payments-events behind Kong):

@app.post("/webhooks/payments")
def payments_webhook():
    event = request.get_json()                  # (1)
    body = json.dumps(event).encode()           # (1)
    expected = hmac.new(SECRET, body,
                        hashlib.sha256).hexdigest()
    sig = request.headers.get("X-Signature", "")
    if sig != expected:                         # (2)
        abort(401)
    mark_paid(event["policy_id"])               # (3)
    return "", 200

1. Parsed, then re-serialized. The partner signed exact bytes; json.dumps changes spacing and number formats (1.10 becomes 1.1), so real requests fail, and someone "fixes" it by turning verification off.
2. != on strings leaks timing, as in section 4.
3. No timestamp. A request captured once (say, from a log) verifies forever: a replay.

Fixed (tested with Python 3 on made-up data):

import hashlib, hmac, json, time

TOLERANCE = 300  # seconds; Stripe's libraries default to 5 min

def verify_stripe_style(raw: bytes, header: str,
                        secret: bytes, now=None) -> bool:
    try:
        items = [p.split("=", 1) for p in header.split(",")]
        ts = next(v for k, v in items if k == "t")
        sigs = [v for k, v in items if k == "v1"]
        age = (now or time.time()) - int(ts)
    except (ValueError, StopIteration, AttributeError):
        return False                       # fail closed (L04)
    if abs(age) > TOLERANCE or not sigs:
        return False                       # too old: replay
    signed = ts.encode() + b"." + raw
    mac = hmac.new(secret, signed, hashlib.sha256).hexdigest()
    return any(hmac.compare_digest(mac, s) for s in sigs)

def verify_github(raw: bytes, header: str,
                  secret: bytes) -> bool:
    if not header or not header.startswith("sha256="):
        return False
    mac = hmac.new(secret, raw, hashlib.sha256).hexdigest()
    return hmac.compare_digest("sha256=" + mac, header)

@app.post("/webhooks/payments")
def payments_webhook():
    raw = request.get_data()               # exact bytes
    hdr = request.headers.get("Stripe-Signature", "")
    if not verify_stripe_style(raw, hdr, SECRET):
        abort(401)
    event = json.loads(raw)                # parse AFTER
    ...

verify_github returns True for GitHub's published test vector; a changed amount, a garbage header, or an hour-old timestamp all return False. In Java (Spring), take @RequestBody byte[] and compare with MessageDigest.isEqual. GitHub sends no timestamp, so dedupe on its X-GitHub-Delivery ID (next section). And make sure nothing in front, like a Kong request-transformer, rewrites the body.

🛠 Try it (optional, ~5 min)

All local. Output is from Homebrew OpenSSL 3.x+; LibreSSL differences are at the end.

1. Reproduce GitHub's test vector:

$ printf 'Hello, World!' | \
    openssl dgst -sha256 -hmac "It's a Secret to Everybody"
SHA2-256(stdin)= 757107ea0eb2509fc211221cce984b8a37570b6d7586c22c
46f4379c8b043e17

It matches GitHub's docs. Use printf, not echo, which adds a newline.

2. Change one character of the body or the key: the HMAC changes completely. Compare printf 'Hello, World!' | shasum -a 256, a plain hash anyone can compute without the secret.

3. Compute a Stripe-style signature with a fake secret:

SECRET='whsec_test_0000'; TS=1793983585
BODY='{"id":"evt_0001","type":"payment.succeeded","amount":125000}'
printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET"

Add 1 to TS and rerun: the signature changes, so an old request can't be made to look fresh.

4. Other syntaxes (same result; hexkey: takes a binary key):

printf '%s.%s' "$TS" "$BODY" | \
  openssl dgst -sha256 -mac HMAC -macopt "key:$SECRET"
printf '%s.%s' "$TS" "$BODY" | \
  openssl mac -digest SHA256 -macopt "key:$SECRET" HMAC

macOS notes: /usr/bin/openssl (LibreSSL 3.3.6) supports dgst -hmac and -mac HMAC -macopt but prints bare hex for stdin, with no SHA2-256(stdin)= prefix. It lacks openssl mac (OpenSSL 3.0+ only, uppercase output). A key typed on the command line lands in shell history and ps, so use test secrets only.

🧠 Key takeaways

• hash(secret || message) is forgeable by length extension; use HMAC (or a library's MAC).
• Compare tags in constant time: hmac.compare_digest, MessageDigest.isEqual.
• Verify over the exact raw bytes received, then parse.
• A valid MAC doesn't mean a fresh request: sign a timestamp and reject old ones.
• Signing gives integrity and authenticity, never confidentiality: a signed JWT is readable.

📖 Words to know

⏭ Next time

L18: storing passwords, and why a fast hash like SHA-256 is exactly the wrong tool for it.

The gap

A MAC answers one question: "did a key holder produce these exact bytes?" It doesn't answer "is this the first time I've seen them?" or "were they sent recently?" A request that was genuine once stays genuine forever, byte for byte. An attacker who gets a copy doesn't need the secret; they just send it again.

How do copies leak? Rarely through TLS itself. More often from places you control: a gateway that logs full request headers and bodies, a debugging proxy, an APM tool that captures payloads, a support ticket with a pasted curl command, or a webhook URL that was briefly pointed at a test server.

 PARTNER            LOG / PROXY              payments-events
    │  POST /webhooks/payments                    │
    │  Stripe-Signature: t=1793983585,v1=ccd9...  │
    │  {"policy_id":"PL-100247",...}              │
    │───────────────────┬────────────────────────▶│ ✔ processed once
    │                   │ copy saved              │
    │                   ▼                         │
    │              [ attacker ]                   │
    │                   │ same bytes, days later  │
    │                   └────────────────────────▶│ ✔ processed again?

Three defences, used together

1. Freshness: a signed timestamp. Stripe puts t= inside the signed string, so it can't be changed without breaking the MAC, and rejects anything older than a tolerance (5 minutes by default in Stripe's libraries; Kong's hmac-auth plugin uses a 300-second clock_skew; AWS rejects most requests more than 5 minutes from their timestamp). An unsigned X-Sent-At header is useless: the attacker just edits it. Stripe's docs also warn that a tolerance of 0 turns the check off entirely in their libraries, and that your servers need NTP, or real requests fail.

2. Uniqueness: remember what you've processed. A timestamp still leaves a 5-minute window, and GitHub sends no timestamp at all. So record each event's ID and refuse duplicates. GitHub gives you X-GitHub-Delivery, a GUID per event; Stripe gives you the event id (evt_...). Dedupe on the event ID, not on the signature: Stripe's retries carry a new timestamp and a new signature for the same event, and GitHub's redeliveries keep the same delivery GUID.

3. Idempotent handlers. Even with both checks, design the effect so running it twice is harmless: "set status of PL-100247 to PAID for invoice INV-5521", not "add $1,250 to the balance". This is the same idempotency idea as PUT in L09.

🧪 Worked example: dedupe with MongoDB

A unique index makes the database enforce "process once", even with several pods racing:

db.webhook_events.createIndex({ eventId: 1 }, { unique: true })
// keep IDs for 7 days, longer than the sender's retry window
db.webhook_events.createIndex(
  { receivedAt: 1 }, { expireAfterSeconds: 604800 })

from datetime import datetime, UTC
from pymongo.errors import DuplicateKeyError

def handle(event_id: str, raw: bytes) -> int:
    try:
        events.insert_one({"eventId": event_id,
                           "receivedAt": datetime.now(UTC)})
    except DuplicateKeyError:
        return 200          # seen it: ack, do nothing
    enqueue_for_processing(raw)
    return 200

Two details matter. Return 200 for a duplicate, or the sender will keep retrying. And choose the TTL from the sender's retry window, not the timestamp tolerance: Stripe retries for up to three days, so IDs must live at least that long. For GitHub, with no timestamp, the dedupe store is your only replay defence, so keep IDs for as long as an old delivery would still do harm.

🧠 Extra takeaways

• A valid MAC proves origin and integrity, not freshness or uniqueness.
• Put the timestamp inside the signed data, and keep the tolerance small but nonzero.
• Dedupe on the event ID with a unique index; return 200 for duplicates.
• Make the handler idempotent, so a replay that slips through does no harm.

Every AWS call is a MAC

When boto3 in an Airflow task or aws s3 ls on your laptop calls AWS, it never sends your secret access key. If you sign in through IAM Identity Center, you get temporary credentials (access key ID, secret key, session token). The SDK uses the secret to compute an HMAC over the request, and AWS, which also knows the secret, recomputes it. This is Signature Version 4 (SigV4).

The four steps

 1. CANONICAL REQUEST            2. STRING TO SIGN
    GET                             AWS4-HMAC-SHA256
    /                               20261106T164625Z
    Action=ListUsers&Version=...    20261106/us-east-1/iam/
    host:iam.amazonaws.com            aws4_request
    x-amz-date:20261106T164625Z     hex(SHA-256(canonical req))
    host;x-amz-date                        │
    hex(SHA-256(body))                     │
                                           ▼
 3. SIGNING KEY (chain of HMACs)  4. SIGNATURE
    k1 = HMAC("AWS4"+secret,        HMAC(kSigning,
              "20261106")                string to sign)
    k2 = HMAC(k1, "us-east-1")       ─▶ hex, into the
    k3 = HMAC(k2, "iam")                Authorization header
    kSigning = HMAC(k3, "aws4_request")

The result goes in a header like this (made-up key ID; one line in practice):

Authorization: AWS4-HMAC-SHA256
  Credential=AKIAIOSFODNN7EXAMPLE/20261106/us-east-1/iam/aws4_request,
  SignedHeaders=host;x-amz-date,
  Signature=<64 hex chars>

What each piece protects

• Canonical request: method, path, query and the listed headers are normalized, so both sides hash identical bytes (the same raw-bytes problem as webhooks, solved by defining one exact format). Change any signed part and the signature fails: that's integrity.
• Payload hash: the body is covered too, via its SHA-256 (S3 can use UNSIGNED-PAYLOAD instead, which leaves the body unprotected by the signature).
• x-amz-date: AWS rejects most requests more than 5 minutes from this time, which limits replay. Within that window a captured request could still be resent, so TLS still matters.
• Scoped signing key: the key chain bakes in date, region and service. A leaked kSigning works only for that day, region and service, unlike the secret access key itself.
• x-amz-security-token: with temporary credentials, the session token travels with the request (some services require it in the signed headers, others only alongside), and AWS checks it hasn't expired.

SigV4a is the asymmetric variant for multi-region requests (for example S3 Multi-Region Access Points): it derives an ECDSA P-256 key pair from your secret, so it's a real digital signature, and AWS only needs to store the public key. Presigned S3 URLs are SigV4 in the query string: anyone holding the URL can use it until X-Amz-Expires runs out, so treat one like a password.

🛠 Try it: derive a SigV4 signing key locally (~3 min)

This uses AWS's public documentation example secret, so no real credentials are involved:

hm() { openssl dgst -sha256 -mac HMAC -macopt "$1" \
       | awk '{print $NF}'; }
K='wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY'
k1=$(printf 20120215     | hm "key:AWS4$K")
k2=$(printf us-east-1    | hm "hexkey:$k1")
k3=$(printf iam          | hm "hexkey:$k2")
printf aws4_request      | hm "hexkey:$k3"
# f4780e2d9f65fa895f9c67b32ce1baf0b0d8a43505a000a1a9e090d414db404d

Homebrew OpenSSL, /usr/bin/openssl (LibreSSL) and Python's hmac module all gave this same value when the lesson was written. Change the date to 20120216 and watch the whole key change: that's the scoping at work. To see a real canonical request from your own account, aws sts get-caller-identity --debug prints it (read-only, but it does call AWS with your credentials).

🧠 Extra takeaways

• SigV4 is HMAC over a canonical form of the request; the secret key never leaves your machine.
• Date, region and service are baked into the signing key, so leaks are scoped.
• The 5-minute timestamp window limits replay but doesn't eliminate it.
• SigV4a and presigned URLs are variations: a real ECDSA signature, and a bearer link.