L11 · Certificates and PKI

Module 2: Networking for Defenders · ~20 min · Lesson 11 of 71

🎯 What you'll learn

• What a certificate binds together, and how a chain of signatures leads back to a root you already trust
• Why revocation mostly doesn't work, and why certificate lifetimes are shrinking toward 47 days
• How certificate transparency lets you spot certificates nobody asked for
• What mTLS adds, and where it shows up between a gateway such as Kong and the services behind it

🔥 Why this matters

In L10 your laptop checked example.com's certificate in a few milliseconds. That check is the only thing stopping someone on the network path from impersonating an API gateway such as Kong or a MongoDB cluster. Certificates fail in two directions. Equifax (L06) let one expire, and its traffic inspection went blind for about 19 months. In 2025, Cloudflare found that a CA had issued twelve certificates for its 1.1.1.1 DNS service without permission, spotted months late in public logs.

1. What a certificate says

An API client connects to api.example.com and receives a public key. How does it know the key isn't from whoever runs the coffee-shop Wi-Fi? A public key carries no name.

A certificate (formally an X.509 certificate) is a signed statement: "this public key belongs to these names, for this time window." It is signed by a certificate authority (CA), an organization whose job is to check that claim first. This is the digital signature from L01 at work: anyone with the CA's public key can verify it, but only the CA could have made it.

 ┌─ X.509 certificate ──────────────────────────────────┐
 │ Subject     CN=api.example.com      (legacy field)   │
 │ SAN         DNS:api.example.com                      │
 │ Public key  EC P-256  04:9f:3a:...                   │
 │ Issuer      CN=Example Issuing CA 1                  │
 │ Validity    2026-10-01 → 2027-04-19  (200 days)      │
 │ Extensions  CA:FALSE · EKU: TLS server auth          │
 ├──────────────────────────────────────────────────────┤
 │ Signature   issuer's PRIVATE key, over all of above  │
 └──────────────────────────────────────────────────────┘

The Subject Alternative Name (SAN) is the list of DNS names (or IPs) the certificate is valid for. Clients match the hostname against the SAN; browsers stopped using the older Common Name (CN) years ago. A wildcard *.example.com covers exactly one label: api.example.com, but not a.b.example.com or example.com itself.

A certificate is public. What proves the server owns it is the TLS handshake (L10), where the server signs with the matching private key. A copied certificate is useless; a stolen key is a disaster.

For most certificates a CA checks only domain validation (DV): proof that you control the name, such as a DNS record or a file at a URL. So the padlock means "you're talking to whoever controls this name," not "this company is honest."

2. The chain of trust

Your laptop doesn't know the CA that signed example.com's certificate either. It knows a small set of root CAs: self-signed certificates preinstalled in a trust store (158 on the Mac this lesson was written on). Roots are kept offline; they sign intermediate CAs, which sign the everyday leaf certificates. This is example.com's public chain as of November 2026:

 TRUST STORE (macOS keychain, the JVM's cacerts, Kong's CA list)
 ┌──────────────────────────────────────┐
 │ ROOT   SSL.com TLS ECC Root CA 2022  │ self-signed, offline
 └──────────────────┬───────────────────┘
                    │ signs
 ┌──────────────────▼───────────────────┐
 │ INTERMEDIATE  SSL.com TLS Transit    │ CA:TRUE
 │               ECC CA R2              │
 └──────────────────┬───────────────────┘
                    │ signs
 ┌──────────────────▼───────────────────┐
 │ INTERMEDIATE  Cloudflare TLS Issuing │ CA:TRUE, pathlen:0
 │               ECC CA 3               │
 └──────────────────┬───────────────────┘
                    │ signs
 ┌──────────────────▼───────────────────┐
 │ LEAF   CN=example.com                │ CA:FALSE, 90 days
 │        SAN example.com,*.example.com │
 └──────────────────────────────────────┘
  server sends: leaf + intermediates   client already has: root

To accept the leaf, a client checks that:

1. each signature verifies with the issuer's public key, ending at a root in its trust store
2. every certificate is inside its validity dates
3. the hostname matches a SAN
4. each issuer is marked CA:TRUE, and the leaf is allowed for TLS server authentication
5. nothing is revoked (best effort, see section 4)

The server must send the intermediates. Browsers often fetch a missing one themselves, so a broken chain "works in Chrome" but fails in curl, Java, or Kong.

Which roots are trusted is decided by root programs (Apple, Google, Microsoft, Mozilla) and the CA/Browser Forum's Baseline Requirements. A CA that breaks the rules can be removed, as Symantec's was in 2018. A company can also run its own private CA: only machines with its root installed trust it, and public rules such as lifetime limits don't apply.

3. Lifetimes are shrinking

CA/Browser Forum ballot SC-081 (passed 11 April 2025) cuts the maximum lifetime of publicly trusted TLS certificates in steps:

Today the limit is 200 days; AWS Certificate Manager (ACM) has issued 198-day public certificates since February 2026. The reasoning: revocation is unreliable (next section), so expiry is the only guaranteed way a bad certificate stops working.

4. Revocation: taking a certificate back

If a private key leaks, the CA must revoke the certificate: declare it invalid early. Two ways to tell clients:

• CRL (certificate revocation list): a signed list of revoked serial numbers, published at a URL written in the certificate.
• OCSP (Online Certificate Status Protocol): the client asks the CA's responder "is serial X still good?" It's slow and leaks which sites you visit. Worse, if the responder can't be reached, clients soft-fail and accept the certificate anyway, and an attacker in the middle can block that query.

OCSP stapling has the server fetch a signed OCSP response and attach it to the handshake, which fixes speed and privacy but not soft-fail.

The industry is leaving OCSP. Ballot SC-063 made it optional for public CAs (from March 2024) and CRLs mandatory. Let's Encrypt dropped OCSP URLs from new certificates on 7 May 2025 and shut its responders down on 6 August 2025. Browsers ship their own revocation lists (Chrome's CRLSets, Firefox's CRLite); most API clients (Java, Python requests, curl) check nothing by default.

5. Certificate transparency

A compromised or careless CA can issue a valid certificate for your domain, and you'd never know. In 2011 the hacked Dutch CA DigiNotar issued a fake *.google.com certificate used to spy on users in Iran. Certificate transparency (CT) makes every public certificate visible.

  ┌────┐ 1. precertificate ──────▶ ┌──────────────────────────┐
  │ CA │                           │ CT LOGS                  │
  │    │ ◀── 2. SCTs (signed ───── │ public, append-only      │
  └─┬──┘        promise to log)    └────────────┬─────────────┘
    │ 3. final cert,                            │ 4. anyone reads
    │    SCTs embedded                          ▼
    ▼                              MONITORS: crt.sh, your alerts
  your server                      on *.example.com
    │ 5. TLS handshake
    ▼
  browser: "SCTs from enough logs?" no ─▶ reject

Before issuing, the CA submits a precertificate to several logs and gets back SCTs (signed certificate timestamps), which it embeds in the certificate. Chrome (for certificates issued since April 2018) and Apple reject public certificates without enough of them.

CT doesn't prevent mis-issuance; it makes it detectable, if someone watches. Cloudflare admitted it lacked regular CT checks, so the 1.1.1.1 certificates (issued from February 2024) went unseen until 2025. CT also leaks names: admin-internal.example.com in a public certificate is free recon for an attacker (L03). Use a private CA for internal names.

6. mTLS: both sides show a certificate

In ordinary TLS only the server proves who it is. In mutual TLS (mTLS), the server also asks for a client certificate, and the client proves it holds the key. In the example stack: Kong presenting a client certificate to an upstream, or a service mesh on EKS giving every pod one. mTLS proves which workload is calling, not which user (more in the first deep dive).

🧪 Worked example: Kong can't reach policy-service after a renewal

policy-service's certificate was renewed overnight. Every call through Kong now returns 502, and Kong's error log shows (made-up data):

2026/11/02 08:14:07 [error] 2231#0: *88412 upstream SSL
certificate verify error: (20:unable to get local issuer
certificate) while SSL handshaking to upstream, upstream:
"https://10.20.3.41:8443/v1/policies/PL-100246"

Error 20 means Kong got a leaf but couldn't find the certificate that signed it. Look at what the server actually sends:

$ openssl s_client -connect policy-svc.internal.example.com:8443 \
    -servername policy-svc.internal.example.com \
    -showcerts </dev/null
Certificate chain
 0 s:CN=policy-svc.internal.example.com
   i:CN=Example Corp Issuing CA 2
---
Verify return code: 20 (unable to get local issuer certificate)

Only certificate 0. The renewal moved to a new intermediate, and the deployment mounts only the leaf. Kong trusts the company root, but has no link between the two:

 BEFORE (broken)                  AFTER (fixed)
 server sends: [leaf]             server sends: [leaf][CA 2]
 Kong trusts:  [root]             Kong trusts:  [root]
 leaf ──?──▶ root   ✗             leaf ──▶ CA 2 ──▶ root   ✓

The fix: serve the full chain, leaf first, then intermediates. cert-manager's tls.crt normally contains it already, so check the app reads the whole file, not just the first PEM block.

The wrong fix: setting tls_verify: false on the Kong service. Calls work again, and anyone who gets between Kong and the pod can now impersonate policy-service. That's a security control silently switched off, the Equifax pattern (L06, and L04's fail secure).

🛠 Try it (~5 min, public read-only data)

1. Save example.com's chain and list each certificate:

echo | openssl s_client -connect example.com:443 \
  -servername example.com -showcerts 2>/dev/null \
  | awk '/BEGIN CERT/{n++} /BEGIN CERT/,/END CERT/{
      print > ("cert" n ".pem")}'
for f in cert*.pem; do
  openssl x509 -in "$f" -noout -subject -issuer -enddate
done

2. Read the leaf: openssl x509 -in cert1.pem -noout -text. Find these lines (public certificate, trimmed):

Issuer: C=US, O=SSL Corporation, CN=Cloudflare TLS Issuing ECC CA 3
Not Before: Sep 26 22:49:11 2026 GMT
Not After : Dec 25 22:56:35 2026 GMT       ← 90 days
X509v3 Basic Constraints: critical
    CA:FALSE                               ← can't sign certs
X509v3 Subject Alternative Name:
    DNS:example.com, DNS:*.example.com
X509v3 CRL Distribution Points: ...        ← revocation list
CT Precertificate SCTs:                    ← 2 SCTs here

3. Open https://crt.sh/?q=example.com in a browser. You'll see each certificate twice: the precertificate and the final one. Click one to see its SHA-256 fingerprint, and compare it with openssl x509 -in cert1.pem -noout -fingerprint -sha256.

On a work laptop, you may see a different chain. When these commands ran on a corporate Mac, the leaf came from the company's own CA (call it "Example Corp Inspection CA"): four certificates, a 395-day leaf, no SCTs. That's a TLS inspection proxy, like Equifax's device: it decrypts, inspects, and re-signs traffic with a CA your laptop was told to trust. crt.sh shows the truth: example.com's real certificates come from SSL.com and Sectigo.

macOS notes: /usr/bin/openssl is LibreSSL 3.3.6: it lacks -ext (use -text) and may report verify error:num=19 if a corporate root is missing from its trust store. crt.sh often returns 502 when busy; retry in a minute.

🧠 Key takeaways

• A certificate binds names (the SAN) to a public key, signed by a CA; the server proves it owns the certificate by signing with the private key.
• Trust comes from a chain: leaf → intermediates → a root already in your trust store. Servers must send the intermediates.
• Revocation is weak (OCSP soft-fails and is being retired), so lifetimes are shrinking: 200 days now, 100 from March 2027, 47 from March 2029.
• Certificate transparency makes every public certificate visible, but only helps if you monitor your domains.
• mTLS adds a client certificate, proving which workload is calling, not which user.

📖 Words to know

⏭ Next time

L12: firewalls, segmentation, proxies, and WAFs: north-south vs east-west traffic, why an API gateway is a reverse proxy, and what a WAF can't catch.

The handshake, with the extra steps

In normal TLS the server sends its certificate and the client checks it. mTLS adds one message from the server (CertificateRequest) and two from the client (its own Certificate, and a CertificateVerify signed with its private key). Here's TLS 1.3 between Kong and an upstream:

 Kong (client)                         policy-service (server)
   │ ── 1. ClientHello ─────────────────────────────▶ │
   │ ◀─ 2. ServerHello, Certificate (server chain) ── │
   │       CertificateRequest   ◀── the "mutual" part │
   │       CertificateVerify, Finished                │
   │  3. Kong checks server chain + SAN               │
   │ ── 4. Certificate (Kong's client chain) ───────▶ │
   │ ── 5. CertificateVerify (signed, Kong's key) ──▶ │
   │ ── 6. Finished ────────────────────────────────▶ │
   │                     7. server checks client chain│
   │                        and Kong's identity       │
   │ ══ 8. encrypted HTTP request ══════════════════▶ │

Step 5 is what makes it more than "Kong sent a file": a signature over the handshake that only the holder of Kong's private key could produce, the same trick the server uses in step 2.

What it proves, and what it doesn't

The last row is the classic mistake. If policy-service accepts any certificate signed by the company CA, then every service, batch job, and laptop holding such a certificate can call it. Trusting a CA means trusting everything it ever issued. Always check the identity too: allow the specific SAN or SPIFFE ID of the expected caller.

mTLS is also only as end-to-end as your network. If a load balancer terminates TLS between Kong and the pod, the pod's mTLS partner is the load balancer, not Kong.

🧪 In the example stack, option 1: Kong to upstream

Kong's Gateway Service entity has the fields for both halves (decK format, made-up UUIDs):

services:
  - name: policy-service
    url: https://policy-svc.policy.svc.cluster.local:8443
    tls_verify: true          # check the upstream's certificate
    tls_verify_depth: 3
    ca_certificates:          # trust store for this upstream
      - 6f1c2a9e-0000-4000-8000-00000000c0a1
    client_certificate:       # what Kong presents (mTLS)
      id: 0b7d4e21-0000-4000-8000-00000000c1e2

client_certificate alone gives you half of mTLS: the upstream can verify Kong, but Kong isn't verifying the upstream. tls_verify and ca_certificates are the other half. Kong's docs say 3.14 and later verify upstream certificates by default and can refuse config with tls_verify off; older versions don't, so check yours. On the upstream side, require a client certificate (for Spring Boot, server.ssl.client-auth=need) and check the caller's SAN against an allow-list.

🧪 In the example stack, option 2: a service mesh on EKS

A mesh such as Istio or Linkerd does mTLS for you. In Istio, each workload gets a certificate whose SAN is a SPIFFE ID built from its Kubernetes service account, for example spiffe://cluster.local/ns/policy/sa/policy-service. Certificates last 24 hours by default and rotate automatically. Two policies make it real:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata: {name: default, namespace: policy}
spec:
  mtls: {mode: STRICT}        # refuse plaintext callers
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata: {name: only-kong, namespace: policy}
spec:
  selector: {matchLabels: {app: policy-service}}
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/kong/sa/kong-gateway"]

STRICT alone is the "any cert from our CA" mistake; the AuthorizationPolicy is what says only Kong's identity may call. Together they mitigate threat row 2 from L05's worked example: a pod that skips Kong and sends X-Consumer-ID: C-20417 directly now gets rejected. It's zero trust from L04: the network location proves nothing; the identity does.

🧠 Extra takeaways

• mTLS adds CertificateRequest, a client Certificate, and a client CertificateVerify signature.
• It proves which workload holds which key, not which user or what they're allowed to do.
• Verify both directions, and check the peer's identity, not just its CA.
• A mesh automates certificates; authorization policies decide who may call whom.

The only outage with a date printed on it

Most outages surprise you. Certificate expiry doesn't: the date is inside the certificate. Yet it keeps happening. Equifax (L06) let hundreds of certificates lapse, including the one that blinded its traffic inspection. On 6 December 2018, an expired certificate in Ericsson software took down mobile data for tens of millions of O2 customers in the UK and SoftBank customers in Japan for most of a day. On 3 February 2020, an expired authentication certificate locked users out of Microsoft Teams for several hours.

It will get harder. Under SC-081, a certificate that was renewed once a year now needs it about twice a year, about four times a year from March 2027, and about eight from March 2029. A calendar reminder and a ticket won't keep up. You need three things: an inventory, monitoring, and automation.

1. Inventory: where certificates hide

Most teams know about the certificate on the public endpoint. The outages come from the others:

The last row bites hardest. On 30 September 2021 an old Let's Encrypt root (DST Root CA X3) expired, and devices with outdated trust stores started rejecting perfectly valid certificates.

2. Monitoring: alert well before the date

Pick thresholds and make them alerts, not dashboards:

 day 0            133 (2/3)               170       193    200
 │ issued          │ auto-renew            │ WARN    │ PAGE │ expiry
 ├─────────────────┼───────────────────────┼─────────┼──────┤
 │                 └─ fails? retry daily   │ <30 d   │ <7 d │

For one endpoint, openssl x509 -checkend (from L06) returns non-zero if the certificate expires within N seconds, so it fits in any cron job or CI check. For Kubernetes, a loop over TLS secrets gives you a quick inventory (run in a dev namespace you're allowed to read; output is made up):

kubectl get secrets -A --field-selector type=kubernetes.io/tls \
  -o custom-columns=NS:.metadata.namespace,N:.metadata.name \
  --no-headers |
while read ns name; do
  end=$(kubectl get secret -n "$ns" "$name" \
    -o jsonpath='{.data.tls\.crt}' | base64 -d |
    openssl x509 -noout -enddate | cut -d= -f2)
  echo "$end  $ns/$name"
done

Apr 19 09:00:00 2027 GMT  policy/policy-svc-tls
Nov 19 09:00:00 2026 GMT  kong/upstream-client-cert   ← 13 days

Better still, use what your platform already emits. cert-manager exports certmanager_certificate_expiration_timestamp_seconds to Prometheus, so an alert can fire when fewer than 7 days remain. ACM sends a daily "ACM Certificate Approaching Expiration" event to EventBridge, starting 45 days before expiry, for every active certificate, including imported ones it can't renew.

Watch the effect too. Equifax's real failure was a control that stopped working without anyone noticing. Alert when a TLS inspection device, WAF, or log pipeline stops seeing traffic, not only when its certificate nears expiry.

3. Automation: renew without a human

• ACM renews its own public certificates automatically when they're DNS-validated (email validation needs a human click). Its 2026 user guide says renewal starts 45 days before expiry.
• cert-manager renews at two-thirds of a certificate's lifetime by default (set renewBefore to override) through ACME or a private CA issuer.
• Reloading matters as much as renewing. cert-manager can update a Secret on time, and an app that reads its certificate only at startup keeps serving the old one until it expires. Kong entities updated by hand, or keystores built into images, have the same problem. Test that rotation reaches the running process.

🛠 Try it (~5 min)

List every certificate one service you own depends on: its ingress certificate, its upstream client certificates in the gateway, its database TLS, any partner certificates, and the CA bundle it trusts. For each one, write down its expiry date, who renews it, and what alert fires 30 days before. Any blank cell is your next outage.

🧠 Extra takeaways

• Expiry is predictable, so an expiry outage is a process failure.
• Shorter lifetimes (200 → 100 → 47 days) make manual renewal unsustainable.
• Inventory everything: gateway certs, keystores, partner client certs, and CA certificates.
• Alert at 30 and 7 days, automate renewal, and test that the running app reloads.