L09 · HTTP under the microscope

Module 2: Networking for Defenders · ~20 min · Lesson 9 of 71

🎯 What you'll learn

• How to read a raw HTTP request and response, line by line
• Which methods are safe and idempotent, and why that matters for retries and side effects
• Why every part of a request, including headers a gateway "adds", is attacker-controlled
• What the Secure, HttpOnly, and SameSite cookie attributes do
• How status codes can leak information (403 vs 404, verbose 500s)

🔥 Why this matters

A team protects an admin endpoint with an "office IP only" check. The service reads the client IP from the X-Forwarded-For header and compares it with 10.0.0.0/8. A tester sends the request again from home with one extra line, X-Forwarded-For: 10.0.0.5, and gets in. Nothing was broken or exploited: the service simply believed a header the client wrote. Today is about learning to see which parts of a request you can trust (very few) and which you can't (the rest).

1. Anatomy of a request and response

HTTP is text with a strict layout. Here is a request as it travels from Kong to a service, with each part labelled:

 PATCH /v1/policies/PL-100246?notify=true HTTP/1.1   ◀─ request line
 │      │                     │                         method,
 │      path                  query string              target,
 method                                                 version
 Host: api.example.com              ◀─ which site (virtual host)
 Content-Type: application/json     ◀─ how to parse the body
 Authorization: Bearer eyJhbGci...  ◀─ credentials (L25)
 Cookie: session=7f3a9c             ◀─ state from earlier visits
 X-Forwarded-For: 198.51.100.23     ◀─ client IP, per the proxy
 X-Consumer-ID: C-20417             ◀─ added by Kong after auth
 Content-Length: 25                 ◀─ body size in bytes
                                    ◀─ blank line ends headers
 {"email":"a@example.com"}          ◀─ body

The response has the same shape: a status line (HTTP/1.1 200 OK), headers, a blank line, and an optional body. Headers are name-value pairs that carry metadata about the message; names are case-insensitive.

HTTP/2, which curl negotiates with most sites today, sends the same information in binary frames instead of text. The request line becomes pseudo-headers (:method, :path, :authority, :scheme), but the meaning is identical, so everything in this lesson applies to both.

2. Methods: safe and idempotent

The method says what the client wants done. Two properties, defined in RFC 9110, matter for security and reliability. A safe method is one the client expects not to change anything on the server. An idempotent method is one where sending the same request twice has the same effect as sending it once.

Proxies, browsers, crawlers, and retry logic all assume GET is safe. A GET /v1/payments/approve?id=PAY-5521 that changes state can be triggered by a link prefetch, a crawler, or another site embedding it as an image (the root of CSRF, L33). And a gateway that retries a timed-out POST can submit a payment twice, which is why payment APIs use idempotency keys: a client-chosen ID that lets the server spot and drop a repeated request.

The method itself is chosen by the client, too. If a check is bound to GET but the framework also routes HEAD or X-HTTP-Method-Override: DELETE to the handler, your check may not run.

3. Everything in the request is attacker-controlled

Here's the one rule to remember from today: the client writes every byte of the request. A browser or the official mobile app is only one possible client; curl can send anything.

So what can you trust? Only facts established by something you control: the TCP peer address of the connection that reached you, a header that your own proxy set and that you know it overwrote, and claims inside a token whose signature you verified. Draw the trust boundary from L05 and ask, "who wrote this field, and could anyone else have written it?"

   CLIENT WRITES (untrusted)           KONG WRITES (trusted only if
   ─────────────────────────           the service can be reached
                                       ONLY through Kong)
   method, path, query                 ────────────────────────
   Host                                X-Consumer-ID   (auth ok)
   Authorization, Cookie               X-Credential-Identifier
   X-Forwarded-For: <anything>  ─────▶ X-Forwarded-For: <client
   X-Consumer-ID: <anything>           value>, <IP that connected>
   body                                X-Kong-Request-Id

 [ client ] ──HTTPS──▶ ( ALB ) ──▶ ( Kong ) ──▶ ( policy-svc )
          ─ ─ ─ TB1 ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ TB2 ─ ─

Two details from Kong's source code are worth knowing. Its authentication plugins (key-auth, for example) overwrite or clear X-Consumer-ID on routes where they run, but on a route with no auth plugin a client-supplied X-Consumer-ID passes straight through. And Kong appends to X-Forwarded-For rather than replacing it, so whatever the client put there is still the leftmost value. Behaviour can differ by version and plugin, so test your own setup.

4. Cookies in 60 seconds

A server sets a cookie with a Set-Cookie response header; the browser sends it back in a Cookie header on later requests. Attributes control when that happens:

Set-Cookie: __Host-session=7f3a9c; Path=/; Secure; HttpOnly;
            SameSite=Lax

• Secure: only sent over HTTPS, so it doesn't leak on plain HTTP.
• HttpOnly: JavaScript can't read it via document.cookie, which limits what XSS (L32) can steal.
• SameSite: Strict, Lax, or None controls whether it's sent on requests started by other sites (CSRF, L33). Some browsers, notably Chrome, treat a cookie with no SameSite as Lax; don't rely on that default.
• __Host- prefix: the browser accepts it only with Secure, Path=/, and no Domain, so a sibling subdomain can't overwrite it.

None of these stops someone using curl from sending any cookie they like. They protect browser users from other websites, not your server from clients. Sessions get a full lesson in L22.

5. Status codes that leak

Status codes are grouped by first digit: 2xx success, 3xx redirect, 4xx client error, 5xx server error. Two codes are often mixed up: 401 Unauthorized means "I don't know who you are" (missing or bad credentials), and 403 Forbidden means "I know who you are, and the answer is no."

That distinction can leak information. If GET /v1/policies/PL-100247 returns 403 for a policy that exists and 404 for one that doesn't, an attacker can loop over IDs and learn which policies exist without reading any of them. This is called enumeration. RFC 9110 explicitly allows a server to answer 404 instead of 403 to hide that a resource exists.

The other leak is the verbose error: a 500 whose body contains a Java stack trace, a Mongo query, or a library version. Return a generic message with a correlation ID, and keep the details in your logs (L38).

🧪 Worked example: an IP allowlist that trusts X-Forwarded-For

Setup. report-svc on EKS has an /admin/export endpoint that should work only from the corporate network, 10.0.0.0/8. Traffic flows client → AWS ALB → Kong → service. An attacker at home (198.51.100.23) sends:

$ curl https://api.example.com/admin/export \
    -H 'X-Forwarded-For: 10.0.0.5'

What arrives at the service. The ALB appends the IP that connected to it (its default append mode); Kong appends the IP that connected to it, the ALB's private address:

 X-Forwarded-For: 10.0.0.5, 198.51.100.23, 10.20.1.15
                  │         │              │
                  │         │              └ added by Kong: the ALB
                  │         └ added by the ALB: the real client
                  └ written by the attacker

Vulnerable code (Python):

def client_ip(request):
    xff = request.headers.get("X-Forwarded-For", "")
    return xff.split(",")[0].strip()   # leftmost = attacker's

if ip_address(client_ip(request)) in ip_network("10.0.0.0/8"):
    return export_all_reports()        # "10.0.0.5" -> allowed

Fixed code. Walk the list from the right, skipping hops you know are your own proxies; the first address that isn't one of them is the real client:

TRUSTED = [ip_network("10.20.0.0/16")]   # ALB/Kong subnets

def client_ip(request):
    hops = [h.strip() for h in
            request.headers.get("X-Forwarded-For", "").split(",")
            if h.strip()]
    for hop in reversed(hops):
        if not any(ip_address(hop) in n for n in TRUSTED):
            return hop                   # "198.51.100.23"
    return None                          # fail closed (L04)

Better still, don't make the app do this at all. Enforce network allowlists at the edge (an AWS security group, or Kong's ip-restriction plugin with trusted_ips and real_ip_header set so Kong itself resolves the real client), and require real authentication on admin endpoints. An IP is a weak signal, not an identity (zero trust, L04).

🛠 Try it (optional, ~5 min)

1. Watch a real exchange. Run:

$ curl -v https://example.com -o /dev/null
* ALPN: server accepted h2
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
*  issuer: ...
* using HTTP/2
> GET / HTTP/2
> Host: example.com
> User-Agent: curl/8.7.1
> Accept: */*
>
< HTTP/2 200
< content-type: text/html; charset=utf-8
< server: cloudflare
< allow: GET, HEAD
< cf-cache-status: HIT

Lines starting > are what you sent, < is what came back, * is curl's commentary. HTTP/2 lowercases header names, and server: cloudflare reveals who's in front of the site. Look at the issuer: line: if it names your employer rather than a public CA, a corporate proxy is decrypting and re-encrypting your traffic (more in L10 and L11). On macOS, curl is built with LibreSSL, so its TLS lines look a little different from Linux output.

2. Status codes. example.com only allows GET and HEAD:

$ curl -s -o /dev/null -w '%{http_code}\n' -X POST \
    https://example.com/
405
$ curl -s -o /dev/null -w '%{http_code}\n' \
    https://example.com/nope
404

3. See that the client writes everything. In one terminal, start a fake server on your Mac with nc -l 8080. In another, send:

$ curl -X DELETE 'http://localhost:8080/v1/policies/PL-100246' \
    -H 'X-Consumer-ID: C-00001' -H 'X-Forwarded-For: 10.0.0.5' \
    -H 'Host: admin.internal'

The nc window prints the raw request, with every forged header intact. Nothing answers, so press Ctrl-C in both windows when you're done.

🧠 Key takeaways

• A request is a method, target, headers, and body, and the client writes all of it.
• GET must not change state; non-idempotent POSTs need idempotency keys to survive retries.
• Trust a header only if your own proxy set it, overwrote any client copy, and is the only way in.
• X-Forwarded-For's leftmost value is client-supplied; read from the right, past known proxies.
• Return 404 for "exists but not yours", and keep stack traces out of 500 responses.

📖 Words to know

⏭ Next time

L10: the TLS handshake, the few round trips that turn an open TCP connection into an encrypted, authenticated channel, and the things TLS still doesn't protect.

The gateway is a header firewall

Every header falls into one of three buckets. Some a gateway should add for the services behind it. Some it should strip because only the gateway may set them. And some it should never trust, because the client wrote them and no amount of parsing makes them true. The same thinking applies to responses going the other way.

           INBOUND (client → Kong → service)
 ┌──────────────────────────────────────────────────────────┐
 │ STRIP from client     X-Consumer-*, X-Credential-*,      │
 │ (only Kong sets)      X-User-Id, X-Roles, X-Authenticated│
 │                       X-Original-URL, X-Rewrite-URL,     │
 │                       X-HTTP-Method-Override             │
 │ ADD                   X-Consumer-ID (after auth),        │
 │                       request/correlation ID,            │
 │                       X-Forwarded-For (append)           │
 │ NEVER TRUST as-is     X-Forwarded-For (leftmost),        │
 │                       X-Forwarded-Host/Proto from        │
 │                       untrusted peers, Host, Referer,    │
 │                       User-Agent                         │
 └──────────────────────────────────────────────────────────┘
           OUTBOUND (service → Kong → client)
 ┌──────────────────────────────────────────────────────────┐
 │ STRIP                 X-Powered-By, Server versions,     │
 │                       internal hostnames, debug headers  │
 │ ADD                   security headers (HSTS etc., L37), │
 │                       the request ID, for support calls  │
 └──────────────────────────────────────────────────────────┘

Why each bucket matters

Identity headers (X-Consumer-ID, X-User-Id, X-Roles) are the most dangerous. If a client can set one and reach a service whose code reads it, they've chosen who they are. Kong's auth plugins overwrite or clear the X-Consumer-* headers on routes where they run, but a route with no auth plugin, or a team's own custom header such as X-Roles, gets no such protection.

Routing-override headers like X-Original-URL and X-Rewrite-URL are honoured by some frameworks and proxies to change the path after the gateway has already applied its rules. A gateway that blocks /admin by path can be sidestepped if the backend reroutes on X-Original-URL: /admin. Strip them unless you know you need them.

X-Forwarded-Host and X-Forwarded-Proto influence redirects and absolute URLs the service builds. Kong's source honours these from a client only if the client's IP is in trusted_ips; otherwise it writes its own values. That's the right model: forwarding headers are believable only from a proxy you configured.

Response headers can leak what you run. Kong's default headers setting (in recent 3.x versions) includes server_tokens and latency_tokens, which add a Via/Server header naming Kong and its version, plus latency timings. Many teams turn those off at the edge to keep version information away from scanners.

🧪 Worked example: a stripping trap with plugin order

A team wants to block forged identity headers, so they add Kong's request-transformer plugin to remove X-Consumer-ID on every route. Requests from the mobile app start arriving at policy-service with no consumer ID at all. Why?

 request ─▶ key-auth (priority 1250) ─▶ request-transformer (801)
            sets X-Consumer-ID:         remove.headers:
            C-20417                     X-Consumer-ID   ✗ removes
                                                          Kong's own
                                                          value
 Higher priority runs first in Kong, so the transformer runs AFTER
 authentication and deletes the header Kong just set.

The fix depends on the route. On authenticated routes, rely on the auth plugin, which already overwrites or clears its headers, and remove the transformer rule. On unauthenticated routes, strip identity headers there (or in an early pre-function). And whatever you do at the gateway, the service must accept traffic only from Kong, with a Kubernetes network policy or mTLS (L43, L51), or every rule above can be bypassed by calling the pod directly. Priorities are from Kong's current source; check the plugin docs for your version.

🛠 Try it (~5 min)

Pick one service you own. List every request header its code reads (search for getHeader, request.headers, or @RequestHeader). For each, write who sets it: the client, the gateway (Kong in the examples), or another proxy. Any header marked "gateway" that would also be accepted if a pod called the service directly is a finding.

🧠 Extra takeaways

• Sort headers into add, strip, and never-trust, for both requests and responses.
• Identity and routing-override headers must be set only by the gateway.
• Forwarding headers are believable only from proxies listed in trusted_ips.
• Know the gateway's plugin order before you add a rule that removes headers.

Two parsers, one stream of bytes

Behind Kong or an ALB, many clients' requests share a few long-lived connections to the backend. That's efficient, but it means the front-end and back-end must agree exactly where one request ends and the next begins. Request smuggling (also called an HTTP desync) happens when they disagree: the attacker hides part of a second request inside the first, and the back-end treats it as the start of the next request on that connection, which may belong to another user.

Where the ambiguity comes from

In HTTP/1.1, a body's length is given in one of two ways: Content-Length: 27 (a byte count) or Transfer-Encoding: chunked (the body arrives in chunks, ending with a zero-length chunk). A message with both is ambiguous. RFC 9112 says Transfer-Encoding wins and that such a message "ought to be handled as an error", but older or lenient parsers disagree, and that gap is the bug.

HTTP/2 fixes this end to end: every frame carries its own length. RFC 9113 says a message with Transfer-Encoding is malformed, and so is one whose content-length doesn't match the data frames. The trouble is downgrading: the front-end speaks HTTP/2 to the client but HTTP/1.1 to the back-end, and rewrites the request. If it copies a client's content-length or transfer-encoding through without checking, the back-end sees an HTTP/1.1 message whose length the attacker chose.

 CLIENT ══ HTTP/2 ══▶ FRONT-END ── HTTP/1.1 (shared) ──▶ BACK-END
                      (ALB, CDN,                          (service)
                       gateway)
 1. Attacker sends one HTTP/2 request. Its frames say:
    "body = 60 bytes", but it also carries a header that
    says the body is shorter.
 2. Front-end trusts the frame length: one request, 60 bytes.
    It rewrites to HTTP/1.1 and copies the bad length header.
 3. Back-end trusts the header: request ends early. The rest
    of the 60 bytes is read as the START of the next request.
 4. A real user's request arrives on the same connection and
    gets glued onto the attacker's leftover bytes.
        ┌──────────────── 60 bytes from front-end ───────────┐
        │ request A (attacker)  │ smuggled prefix ...        │
        └───────────────────────┴────────────┬───────────────┘
                                             │ + user's request B
                                             ▼
          back-end runs "prefix + B" as one request, using
          B's cookies or token, or returns B's response to
          the attacker

Real-world scale

James Kettle of PortSwigger revived the topic with "HTTP Desync Attacks" (2019) and showed the HTTP/2 downgrade variants in "HTTP/2: The Sequel is Always Worse" (Black Hat USA 2021). His findings included an H2.CL issue at Netflix (CVE-2021-21295, in Netty) and an H2.TE issue in AWS's Application Load Balancer, which accepted Transfer-Encoding in HTTP/2 requests; both were fixed. The lesson for you: even managed infrastructure can get this wrong, so stay current.

Defences (what you control)

• Keep every proxy patched: ALB is managed by AWS, but self-run proxies and servers such as Kong, NGINX ingress, Netty, and Tomcat are yours to upgrade (L02's patching priorities apply).
• Reject, don't repair: configure proxies and servers to return 400 and close the connection on ambiguous requests (both CL and TE, duplicate CL, malformed chunks). Most modern servers do this by default; verify yours.
• Use one protocol end to end where you can (HTTP/2 or HTTP/1.1 throughout), and avoid chains of different proxy products that each parse requests their own way.
• Don't treat the gateway as the only check. Smuggled requests skip gateway rules, so services must still authenticate and authorize each request (defence in depth, L04).
• Watch for symptoms: unexplained 400s or timeouts, or users reporting someone else's data, can indicate a desync.

🧠 Extra takeaways

• Smuggling exploits disagreement between two parsers about where a request ends.
• HTTP/2 framing removes the ambiguity, but downgrading to HTTP/1.1 can bring it back.
• Patch proxies, reject ambiguous messages, and keep authorization in the service.