L43 · API gateways as a security control

Module 6: API Security · ~20 min · Lesson 43 of 71

🎯 What you'll learn

• What a gateway like Kong can check on every request, and in what order
• How Kong's authentication plugins (key-auth, jwt, openid-connect, mTLS) differ, and their traps
• Which checks a gateway can never make: object-level and business-logic authorization
• Why the gateway is itself a target, and how to keep its config honest

🔥 Why this matters

A pen test finding on an API behind a Kong gateway: a customer token reads another customer's statement through GET /v1/accounts/ACC-2002/statements. In stand-up someone says "but Kong handles security". It did: the token was valid, the scope right, the rate normal, and the log shows a tidy 200. This lesson is about where that line is, so nobody assumes a check happened that never could.

1. The gateway as a checkpoint

Think of an airport. Passport control checks who you are and that you may enter the country. It doesn't check whether the hotel room you're heading to is booked in your name.

An API gateway is a reverse proxy that every external API call passes through, applying the same checks to all of them. In L27 terms it's a coarse policy enforcement point (PEP): it enforces rules that it can decide from the request alone. In Kong, a service is an upstream API, a route is a path or host that leads to it, a consumer is a known caller, and a plugin is a check or transformation attached globally, to a service, a route or a consumer.

 client
   │ 1. HTTPS (TLS ends here, or mTLS: L10, L11)
   ▼
 ┌──────────────────────── Kong ────────────────────────┐
 │ 2. match route  ─▶ no route? 404                     │
 │ 3. authenticate ─▶ key-auth / jwt / oidc / mtls 401  │
 │ 4. authorize    ─▶ acl, ip-restriction, scopes  403  │
 │ 5. hygiene      ─▶ size limit, validation, rate  4xx │
 │ 6. transform    ─▶ strip client headers, add IDs     │
 └──────────────────────────┬───────────────────────────┘
                            │ 7. proxied, X-Consumer-ID set
                            ▼
                      accounts-svc (EKS)
                            │ 8. response back through Kong
                            ▼
                 9. log plugin: who, what, status, latency

Any plugin can stop the request with an error, so the upstream sees only traffic that passed every check (the first deep dive covers the order they run in).

2. Proving who's calling

Kong's authentication plugins answer "who is this?" and attach a consumer to the request. They prove different things:

(Editions change between Kong versions; check the plugin hub for yours.)

Look again at jwt: it checks exp and nbf only if listed in config.claims_to_verify, which is empty by default. A validly signed token that expired last year passes. You'll try this below.

3. Coarse authorization and hygiene

Once Kong knows the consumer, it can decide anything that needs only the consumer, route and request.

• acl allows or denies consumer groups per route: only partner-payments may call /v1/payments. A missing match returns 403 You cannot consume this service.
• Scopes: openid-connect can require accounts:read on a route. That's function-level authorization (BFLA, L40) done at the edge.
• ip-restriction limits admin or partner routes to known address ranges.
• request-size-limiting rejects large bodies. Its default limit is 128 MB, so set a real one.
• Request validation checks bodies against a schema (second deep dive); rate limiting caps calls per consumer or IP (L42); cors applies L37's browser rules.

Logging plugins (http-log, file-log, tcp-log, datadog and others) record every request: consumer, route, status, latency and client IP. Add correlation-id so one ID follows the request through your services. Don't assume secrets are masked: depending on version, API keys in headers or query strings, and anything added with custom_fields_by_lua, can be logged as-is. Check.

4. What a gateway can't decide

Here is the line from the pen test, drawn as what each layer knows:

 GET /v1/accounts/ACC-2002/statements    token: sub=C-1001

 KONG KNOWS                        ACCOUNTS-SVC KNOWS
 ───────────────────────────       ─────────────────────────────
 token is signed and fresh         ACC-2002 belongs to C-2002
 consumer = mobile-app             C-1001 is not a joint holder
 scope accounts:read ✓             the account is closed
 path matches /v1/accounts/*       statement has a legal hold
 rate: 3 req/min ✓                 this is the 400th account
                                     read by C-1001 today
          │                                  │
          ▼                                  ▼
     "allowed" ✓                  "is this caller allowed THIS
                                   object, in THIS state?" ✗

Kong could parse ACC-2002 out of the path, but ownership lives in MongoDB, not in the request. So two classes of check always belong in the service:

• Object-level authorization (BOLA, L34 and L40): is this object yours?
• Business logic: four-eyes approval, limits, state ("can't refund a refunded payment"), and flow abuse (L39's API6). Every request is valid; only the business data says no.

A gateway also can't protect a service that doesn't go through it. If accounts-svc is reachable directly inside the cluster and trusts X-Consumer-ID, any pod can claim to be any consumer (L09, L34). Allow ingress to the service only from Kong with a NetworkPolicy, and better, have the service verify the token or an mTLS identity itself (L04's zero trust).

5. The gateway is a target too

Kong holds the credentials and rules for everything, so its configuration is a crown jewel.

• The Admin API (port 8001/8444) can delete an auth plugin or copy all traffic elsewhere. In open-source Kong it has no authentication of its own (L38's deep dive).
• Configuration drift. A route added by hand with no auth plugin is live for everyone. Manage config as code with decK from git, review it, and alert on drift.
• Secrets in config. Consumer keys and JWT secrets in a decK file are secrets in git (L19). Use Kong's vault references ({vault://...}); which vault backends are free depends on version.
• Its own CVEs. Patch Kong like any internet-facing server (L06).

🧪 Worked example: one route, every layer

A decK file for a made-up partner route. The comments show what each block catches.

_format_version: "3.0"
services:
- name: accounts-svc
  url: http://accounts-svc.accounts.svc:8080
  routes:
  - name: partner-accounts
    paths: [/v1/accounts]
    methods: [GET]               # no PUT/DELETE here at all
    plugins:
    - name: openid-connect       # Enterprise; who is calling?
      config:
        issuer: https://idp.example.com/
        auth_methods: [bearer]
        audience_required: [accounts-api]
        scopes_required: [accounts:read]
    - name: acl                  # which apps may use it?
      config:
        allow: [partner-readers]
    - name: request-size-limiting
      config:
        allowed_payload_size: 16
        size_unit: kilobytes
    - name: correlation-id
      config:
        header_name: X-Correlation-ID
        echo_downstream: true

(With openid-connect, mapping a token to an acl group needs extra settings such as authenticated_groups_claim; see Kong's docs.)

What the gateway does with five requests:

 request                                    Kong   why
 ─────────────────────────────────────────  ─────  ────────────────
 1. no token                                401    step 3
 2. token, aud=marketing-api                401    wrong audience
 3. right token, app not in partner-readers 403    acl
 4. DELETE /v1/accounts/ACC-2002            404    no such route
 5. valid partner token for ACC-2002, but   200 ⚠  Kong can't see
    the partner has no rights to ACC-2002          ownership

Request 5 is the pen test finding. The fix is in accounts-svc: check the caller may use that account, and return 404 if not (L34).

🛠 Try it (optional, ~5 min)

See the jwt trap locally: Kong in Docker, DB-less, on 127.0.0.1 only, with request-termination standing in for a service. kong/kong-gateway runs free plugins without a license; any 3.x tag you can pull works.

mkdir -p ~/l43-try && cd ~/l43-try
cat > kong.yml <<'EOF'
_format_version: "3.0"
services:
- name: demo
  url: http://127.0.0.1:9
  routes:
  - name: demo
    paths: [/demo]
    plugins:
    - name: jwt
      config: {claims_to_verify: []}   # same as default
    - name: request-termination
      config: {status_code: 200, message: "reached upstream"}
consumers:
- username: demo-app
  jwt_secrets:
  - {key: demo-iss, secret: local-only-not-a-real-secret,
     algorithm: HS256}
EOF
docker run -d --name kong-l43 -e KONG_DATABASE=off \
  -e KONG_DECLARATIVE_CONFIG=/cfg/kong.yml \
  -v "$PWD:/cfg" -p 127.0.0.1:8000:8000 kong/kong-gateway:3.9

Make a token that expired an hour ago (Python standard library only):

TOKEN=$(python3 - <<'EOF'
import base64, hashlib, hmac, json, time
b64 = lambda b: base64.urlsafe_b64encode(b).rstrip(b"=").decode()
h = b64(json.dumps({"alg": "HS256", "typ": "JWT"}).encode())
p = b64(json.dumps({"iss": "demo-iss",
                    "exp": int(time.time()) - 3600}).encode())
key = b"local-only-not-a-real-secret"
sig = hmac.new(key, f"{h}.{p}".encode(), hashlib.sha256).digest()
print(f"{h}.{p}.{b64(sig)}")
EOF
)
curl -s -o /dev/null -w '%{http_code}\n' \
  -H "Authorization: Bearer $TOKEN" http://127.0.0.1:8000/demo

Expect 200: an expired token got through. Now turn on the expiry check and repeat the curl:

sed -i '' 's/claims_to_verify: \[\]/claims_to_verify: [exp]/' kong.yml
docker restart kong-l43

(sed -i '' is the macOS form; on Linux drop the ''.)

Expect 401, with a body naming the expired claim (wording varies by version). These outputs are what Kong 3.x is documented to do, not captured from a run. Clean up: docker rm -f kong-l43.

🧠 Key takeaways

• A gateway is a coarse PEP: it decides from the request alone, the same way for every route.
• Authentication plugins differ in what they prove; Kong's free jwt plugin skips exp unless told.
• Kong can enforce scopes, groups, sizes and rates, but never "is this object yours?" or business rules.
• A gateway only protects traffic that goes through it; lock the upstream to the gateway.
• The gateway's Admin API and config are crown jewels: config as code, reviewed, with drift alerts.

📖 Words to know

⏭ Next time

L44 closes Module 6 with a case study: Optus (2022) and T-Mobile (2023), where APIs handed millions of customer records to whoever asked.

Most gateway outages are loud. The dangerous misconfigurations are quiet: every test passes, and a route is open to the internet. Two Kong behaviours cause most of them.

Plugins run by priority, not by the order you wrote them

Each Kong plugin has a fixed priority. In the access phase, higher numbers run first. Some Kong 3.x values (they change between versions; Kong's "plugin execution order" page lists yours):

 priority   plugin                  what it means
 ────────   ─────────────────────   ─────────────────────────────
   2000     cors                    preflights answered first
   1450     jwt                     ┐
   1250     key-auth                ├ authentication
   1050     openid-connect (Ent.)   ┘
    990     ip-restriction
    950     acl                     needs a consumer, so after auth
    910     rate-limiting           ⚠ after auth (see below)
     12     http-log                logs every outcome, even 401s

Notice rate-limiting sits below the auth plugins. When key-auth rejects a bad key with 401, the request ends there and the rate limiter never counts it. A script guessing API keys isn't slowed down by a rate limit on that route. Defend against guessing with long random keys (L20) and alerting on 401 bursts, or add an IP rate rule in front of Kong (for example AWS WAF). Kong Enterprise can change the order per plugin (ordering), which is powerful and easy to get wrong.

Two auth plugins: AND by default, OR with anonymous

Put key-auth and jwt on the same route and, by default, a caller must pass both. Teams who want "either one" set config.anonymous to a placeholder consumer. Now a failed check doesn't reject the request; it continues as the anonymous consumer:

 request with no credentials
   │
   ├─ jwt:      fails ─▶ anonymous set ─▶ continue as "anon"
   ├─ key-auth: fails ─▶ anonymous set ─▶ continue as "anon"
   ▼
 ┌──────────────────────────────────────────────┐
 │ does any later plugin block "anon"?          │
 │   yes (acl / request-termination) ─▶ 401/403 │
 │   no  ─▶ proxied to upstream  ⚠ OPEN ROUTE   │
 └──────────────────────────────────────────────┘

Kong's docs pair anonymous with a second plugin that blocks the anonymous consumer. Forget that plugin, and the route is public, while every test with a valid token still passes.

🧪 Worked example: the route that went public

# BEFORE: "accept partner keys or JWTs"
routes:
- name: statements
  paths: [/v1/statements]
  plugins:
  - name: jwt
    config: {anonymous: anon, claims_to_verify: [exp]}
  - name: key-auth
    config: {anonymous: anon, hide_credentials: true}
consumers:
- username: anon            # nothing ever blocks this consumer

Kong forwards anonymous requests with the header X-Anonymous-Consumer: true, and this upstream ignores it. The fix makes "no valid credential" fail closed:

# AFTER
  plugins:
  - name: jwt
    config: {anonymous: anon, claims_to_verify: [exp]}
  - name: key-auth
    config: {anonymous: anon, hide_credentials: true}
  - name: acl
    config: {allow: [statement-readers]}   # anon isn't in it

Add a test to your pipeline that calls each route with no credentials and expects 401 or 403. It catches this bug and the "route added by hand with no plugin" bug in one go.

The most specific plugin config wins

A plugin set on a route replaces the same plugin set globally; Kong doesn't merge them. Global key-auth with hide_credentials: true, plus a route-level key-auth added for a different key_names, means that route forwards keys upstream again. When you review a decK diff, read each route's plugins as the complete set, not as additions to the global ones.

🧠 Extra takeaways

• Kong runs plugins by priority; auth runs before rate limiting, so failed logins aren't counted.
• Multiple auth plugins are AND by default; anonymous turns them into OR and opens a fail-open path.
• Every anonymous needs a matching block (acl or request-termination) on the anonymous consumer.
• A "call every route with no credentials" test in CI catches most gateway fail-open bugs.

What a schema can say

Your OpenAPI spec already describes what a valid request looks like: types, required fields, formats, lengths, enums, and whether extra fields are allowed. A gateway that enforces it rejects malformed requests before your code parses them. In Kong, the request-validator plugin checks bodies and parameters against JSON Schema, and oas-validation checks against a whole OpenAPI spec. Both are Enterprise plugins at the time of writing; the free alternative is the same check in the service (Bean Validation in Spring, or an OpenAPI validator middleware).

 PATCH /v1/customers/C-1001/profile
 {"email":"a@example.com","role":"admin","limit":1e9}
   │
   ▼
 ┌─ request-validator ────────────────────────────┐
 │ email: string, format email          ✓         │
 │ additionalProperties: false  "role"  ✗ ─▶ 400  │
 │ limit: not in schema                 ✗         │
 └────────────────────────────────────────────────┘
   │ (only clean bodies continue)
   ▼
 customers-svc

🧪 Worked example: blocking mass assignment at the door

L41's attack slips {"role":"admin"} into a PATCH. A strict schema rejects any field it doesn't list:

- name: request-validator          # Enterprise plugin
  config:
    version: draft4                # JSON Schema draft 4
    allowed_content_types: [application/json]
    verbose_response: false        # don't echo schema details
    body_schema: |
      {"type": "object",
       "additionalProperties": false,
       "properties": {
         "email": {"type": "string", "maxLength": 254},
         "phone": {"type": "string",
                   "pattern": "^\\+[0-9]{8,15}$"}}}
    parameter_schema:
    - name: customerId
      in: path
      required: true
      style: simple
      explode: false
      schema: '{"type":"string","pattern":"^C-[0-9]{4,10}$"}'

The route's path must name the parameter (for example a regex path with a named capture group called customerId), and the parameter_schema format varies a little by version, so test it locally. The service should still bind into a DTO with only these fields (L41): the schema is a second layer, not the fix.

What it can't catch

A schema checks shape. Every one of these requests is perfectly shaped:

It also can't help with routes that aren't in the spec at all, which is exactly where forgotten endpoints live (API9, L39).

Keeping the spec and the gateway in sync

Validation is only as good as the spec it enforces. If the spec allows additionalProperties by default (JSON Schema does), you've validated nothing. Generate gateway config from the spec in CI (decK's deck file openapi2kong does this, and can carry plugin settings through x-kong-plugin-* extensions; check your decK version), so a field added in code but not in the spec fails fast in test, not silently in production.

🧠 Extra takeaways

• Schema validation rejects malformed requests early: unknown fields, wrong types, huge strings.
• Set additionalProperties: false on write bodies, or the schema doesn't stop mass assignment.
• A perfectly shaped request can still be BOLA, a broken business rule, or a scrape.
• Generate gateway validation from the spec in CI so the two can't drift apart.