L42 · Rate limiting and resource consumption

Module 6: API Security · ~20 min · Lesson 42 of 71

🎯 What you'll learn

• What "resource consumption" covers besides request counts
• How fixed windows, sliding windows, token buckets and leaky buckets work
• How to choose the key you limit by (IP, consumer, account) and price expensive calls higher
• How to configure Kong's rate-limiting plugin, and why its policy changes the real limit

🔥 Why this matters

At 2 a.m. a partner's batch job retries in a tight loop: 3,000 requests a second to GET /v1/positions, each a MongoDB aggregation. The pods hit the HPA maximum, MongoDB's CPU pins at 100%, and every other client times out. Nobody attacked you; the API simply had no ceiling. That's API4:2023 Unrestricted Resource Consumption (L39).

1. What you're actually protecting

Every request spends something: CPU, memory, database time, bandwidth, and sometimes money (an SMS, a credit-bureau lookup). Counting requests is only one of the limits you need.

  client
    │
    ▼
 ┌─────────────────────────────────────────────────────┐
 │ Kong     requests per minute per consumer           │
 │          max body size, connection timeouts         │
 ├─────────────────────────────────────────────────────┤
 │ service  max page size (?limit=100000 → 100)        │
 │          per-account limits on login, OTP, export   │
 │          cost-based budget for expensive endpoints  │
 ├─────────────────────────────────────────────────────┤
 │ MongoDB  maxTimeMS on queries, indexes, pool size   │
 ├─────────────────────────────────────────────────────┤
 │ vendors  daily spend cap on SMS / bureau calls      │
 └─────────────────────────────────────────────────────┘

The gateway counts requests. Only the service knows that one ?limit=100000 call costs as much as a thousand normal ones, so you need limits at every layer (defense in depth, L04).

2. Four ways to count

Every limiter asks whether a key has used up its allowance. They differ in how they remember.

Fixed window. Keep one counter per key per clock window ("minute 14:03") and reject above the limit. It's cheap, but it allows a boundary burst: a whole allowance at the end of one window and another at the start of the next.

 limit: 10 per minute, fixed windows
      window 14:03                    window 14:04
 |............................████|████............................|
                           10 req ▲ 10 req
                                  14:04:00
 fixed window:   20 requests in ~2 s, all allowed
 sliding window: at 14:04:01 the last 60 s already hold
                 ~10 requests, so the second burst is refused

Sliding window. Count the last 60 seconds from now. A sliding window log stores every timestamp (exact, but memory grows with traffic). A sliding window counter keeps two counts, this window's and the previous one's weighted by how much of it still overlaps: a good estimate.

Token bucket. Each key has a bucket that holds up to B tokens and refills at r tokens per second. A request takes a token, or is refused if none are left. Short bursts up to B are fine, and the long-run rate can't exceed r.

        refill: r = 10 per minute (one every 6 s)
                │
                ▼
         ┌─────────────┐
         │ ● ● ● ● ●   │  holds at most B = 10 tokens
         │ ● ● ● ● ●   │  (the largest burst allowed)
         └──────┬──────┘
                │ a request takes 1 token
                │ (an export might take 20)
                ▼
   token left? ──yes──▶ forward to the service
        │
        no ────────────▶ 429 Too Many Requests + Retry-After

That makes cost-based limits natural: an export simply costs more tokens.

Leaky bucket. Requests join a queue that drains at a fixed rate; when the queue is full, new requests are dropped. The output is perfectly smooth, at the cost of added latency. (Some tools use the name for a meter equivalent to a token bucket.)

3. Who are you limiting?

The key decides who shares a counter, and it matters as much as the number.

• By IP is the fallback for unauthenticated routes. It's weak: an office shares one NAT address, while a botnet gets a fresh allowance per address.
• By consumer or credential (the authenticated client) is the right default behind a gateway: a runaway job hurts only its owner.
• By the account being acted on protects login, OTP and password reset: 5,000 IPs guessing one customer's code are still 5,000 attempts on one account (L31 made the same point).
• A global ceiling per route protects the backend no matter how many keys share it.

Set limits per route: a 6-digit OTP has a million values, so five attempts per code makes guessing hopeless, while a limit sized for GET /v1/quotes doesn't.

4. Kong's rate-limiting plugin

The open-source rate-limiting plugin counts fixed windows (second through year). In decK format, on a route:

plugins:
- name: rate-limiting
  route: transactions-read
  config:
    minute: 600              # per consumer
    hour: 20000              # both limits apply
    limit_by: consumer       # needs an auth plugin on the route
    policy: redis            # local | cluster | redis
    redis:                   # Kong 3.6+; older: redis_host, ...
      host: redis.kong.svc.cluster.local
      port: 6379
      timeout: 2000          # ms
    fault_tolerant: true     # Redis down: proxy unlimited
    hide_client_headers: false

With the Kong Ingress Controller, the same config goes in a KongPlugin resource. If Kong can't identify a consumer, the plugin falls back to limiting by IP.

The policy decides where counters live, and therefore what "600 per minute" really means:

                  load balancer
          ┌─────────────┼─────────────┐
          ▼             ▼             ▼
 policy: local  kong-1: 600   kong-2: 600   kong-3: 600
                → up to 1,800/min get through (and 2,400
                  after the HPA adds a fourth pod)

 policy: redis  kong-1        kong-2        kong-3
                   └──INCR───────┼──────INCR───┘
                                 ▼
                         ┌──────────────┐
                         │ Redis: 1     │ → 600/min, plus one
                         │ counter/key  │   network hop each
                         └──────────────┘

local is fine for a coarse "stop the flood" limit, if you remember it's per pod. For limits you promise partners, or that protect money or credentials, use redis. Set policy explicitly rather than relying on your version's default.

🧪 Worked example: the 2 a.m. partner job, with limits

With the plugin above, the partner's 601st request in a minute gets:

HTTP/1.1 429 Too Many Requests
RateLimit-Limit: 600
RateLimit-Remaining: 0
RateLimit-Reset: 23              ◀ seconds until the window resets
X-RateLimit-Limit-Minute: 600
X-RateLimit-Remaining-Minute: 0
Retry-After: 23                  ◀ a well-behaved client waits

{"message":"API rate limit exceeded"}

Kong answers without touching the service. The service still caps what one allowed request costs:

// application.yml: spring.data.web.pageable.max-page-size: 100
// (Spring Data's default cap is 2000)
@GetMapping("/v1/positions")
Page<PositionView> list(@AuthenticationPrincipal Jwt jwt,
                        Pageable page) {          // size ≤ 100
    Query q = Query.query(where("accountId").is(jwt.getSubject()))
                   .with(page)
                   .maxTime(Duration.ofSeconds(2)); // maxTimeMS
    return positions.find(q);                     // runs in Mongo
}

One call can now spend at most two seconds of MongoDB time and return 100 rows. (positions.find stands in for a MongoTemplate query plus a count.) Cap body size too: Kong's request-size-limiting plugin, or express.json(), which refuses bodies over 100 KB by default.

🛠 Try it (optional, ~5 min)

Compare three algorithms on a boundary burst. Save this as limits.py:

# limits.py: one burst, three limiters. Limit: 10 per 60 s.
L, W = 10, 60.0
burst = [59 + i / 20 for i in range(10)] + \
        [60 + i / 20 for i in range(10)]   # just before/after 1:00

def fixed(ts):
    n = {}
    for t in ts:
        w = t // W
        if n.get(w, 0) < L:
            n[w] = n.get(w, 0) + 1
    return sum(n.values())

def sliding(ts):
    n = {}
    for t in ts:
        w = t // W
        est = n.get(w - 1, 0) * (1 - t % W / W) + n.get(w, 0)
        if est < L:
            n[w] = n.get(w, 0) + 1
    return sum(n.values())

def bucket(ts, tokens=L, last=0.0):
    ok = 0
    for t in ts:
        tokens = min(L, tokens + (t - last) * L / W)  # refill
        last = t
        if tokens >= 1:
            tokens, ok = tokens - 1, ok + 1
    return ok

for f in (fixed, sliding, bucket):
    print(f"{f.__name__:8} accepted {f(burst):2} of 20")

Real output:

$ python3 limits.py
fixed    accepted 20 of 20
sliding  accepted 11 of 20
bucket   accepted 10 of 20

The sliding window's 11 is its estimate being slightly generous after the boundary. Change the second burst to start at 120 and predict the results before you run it.

🧠 Key takeaways

• API4 is about cost, not just request counts: cap size, time, page length and spend as well.
• Fixed windows allow 2× bursts at the boundary; sliding windows and token buckets smooth them.
• Limit by consumer or by target account, not only by IP, and set limits per route.
• Kong's local policy multiplies the limit by the pod count; redis shares one counter, and fault_tolerant decides what a Redis outage does.
• The gateway counts requests; only the service can bound what one request costs.

📖 Words to know

⏭ Next time

L43 steps back to the gateway as a whole: what Kong does well as a security control (authentication, mTLS, logging, limits) and what it can't do for you.

When you write your own limiter (a per-account OTP limit in the service, say), you face the problem Kong's redis policy solves: many pods, one counter. The naive version is wrong in a way that's easy to miss in testing.

The race

Two pods read the counter, both see 9 under a limit of 10, and both write 10. Two requests got through, and the counter says one did.

  pod A                     Redis               pod B
    │ GET otp:C-2001 ─────────▶│                   │
    │◀──────────────── 9 ──────│◀──── GET ─────────│
    │                          │───── 9 ──────────▶│
    │ 9 < 10: allow            │      9 < 10: allow│
    │ SET otp:C-2001 10 ──────▶│                   │
    │                          │◀── SET ... 10 ────│
    ▼                          ▼                   ▼
   counter = 10, but 11 requests were allowed

Under real load (or an attacker sending requests in parallel on purpose), "read, decide, write" leaks. It's the same class of bug as a double-spend: a race condition between a check and an update.

Atomic options

1. INCR first, decide after. INCR is atomic, so each request gets a unique count back. If the count is over the limit, refuse. No two requests can both see 10.
2. Set the expiry safely. INCR then EXPIRE is two commands; a crash between them leaves a counter with no expiry, which locks the key out forever. Use a script (below), or Redis 7's EXPIRE key seconds NX, which only sets an expiry if none exists.
3. Lua scripts run atomically. For a token bucket you need to read tokens, compute the refill and write back as one step, so put it in a script.

A fixed-window limiter as one atomic script:

-- KEYS[1] = "rl:otp:C-2001:" .. window number
-- ARGV[1] = limit, ARGV[2] = window length in seconds
local n = redis.call("INCR", KEYS[1])
if n == 1 then
  redis.call("EXPIRE", KEYS[1], ARGV[2])
end
if n > tonumber(ARGV[1]) then
  return 0          -- refuse
end
return 1            -- allow

Worked example: a token bucket in Redis

-- KEYS[1] = "tb:consumer:partner-42"
-- ARGV: capacity, refill per second, cost of this request
local cap, rate, cost =
  tonumber(ARGV[1]), tonumber(ARGV[2]), tonumber(ARGV[3])
local t = redis.call("TIME")              -- Redis's clock
local now = t[1] + t[2] / 1e6
local s = redis.call("HMGET", KEYS[1], "tokens", "ts")
local tokens = tonumber(s[1]) or cap
local ts = tonumber(s[2]) or now
tokens = math.min(cap, tokens + (now - ts) * rate)
local ok = tokens >= cost
if ok then tokens = tokens - cost end
redis.call("HSET", KEYS[1], "tokens", tokens, "ts", now)
redis.call("EXPIRE", KEYS[1], math.ceil(cap / rate) * 2)
return ok and 1 or 0

Three design points: it uses Redis's TIME rather than each pod's clock, so clock skew between nodes doesn't mint tokens (calling TIME before writes in a script needs Redis 5 or later, where scripts replicate their effects). cost makes it a cost-based limiter. And the expiry cleans up idle keys, so memory doesn't grow with every consumer you've ever seen.

Then decide the failure mode deliberately. If Redis is unreachable, a login or OTP limiter should probably fail closed, while a coarse traffic limit might fail open (L04's deep dive). On EKS, run Redis with replication and keep the limiter's timeout short, so a slow Redis doesn't become your outage.

🧠 Extra takeaways

• "Read, check, write" across pods is a race; attackers trigger it on purpose with parallel requests.
• Use INCR-then-check or a Lua script so the check and the update are one atomic step.
• Never leave a counter without an expiry; use a script or EXPIRE ... NX (Redis 7+).
• Use the server's clock, add a cost per request, and choose fail-open or fail-closed per limiter.

Rate limiting has two sides. Your services and scheduled jobs (Airflow DAGs in the examples) also call APIs: internal ones behind the gateway, and vendors outside. A client that handles 429s badly turns a short blip into an outage, and can lock you out of a partner's API.

Retry storms

A service slows down. Every client times out and retries immediately, so the load doubles, then triples as retries retry. If all clients use the same fixed delay, they all come back at the same moment, in waves. The service never recovers until the clients stop.

 load
  │      ▲ first failure
  │ ███  │  ███        ███        ███     fixed delay: retries
  │ ███  │  ███        ███        ███     arrive in waves
  │ ███████████        ███        ███
  └───────────────────────────────────▶ time
  │
  │ ███  │                                full jitter: the same
  │ ███████▄▄▄▄▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂            retries, spread out
  └───────────────────────────────────▶ time

The rules

1. Honor Retry-After. RFC 9110 allows a number of seconds or an HTTP date. Kong's 429 sends seconds. Never retry sooner than it says.
2. Otherwise, exponential backoff with jitter. Double a delay cap each attempt and sleep a random time up to it ("full jitter", as described in AWS's Architecture Blog). Randomness is what breaks up the waves.
3. Cap attempts and total time. Give up after a few tries and surface the failure.
4. Retry only what's safe. GETs are idempotent. A POST that moves money is only safe to retry with an idempotency key the server deduplicates on (an IETF draft standardizes an Idempotency-Key header; several payment APIs already use one).
5. Retry only retryable errors: 429, 503, timeouts. A 400 or 403 will fail the same way again.

Worked example: an Airflow task that calls a rate-limited API

import random, time
import requests

def get_with_backoff(url, headers, max_tries=5, base=1, cap=60):
    for attempt in range(max_tries):
        r = requests.get(url, headers=headers, timeout=10)
        if r.status_code not in (429, 503):
            r.raise_for_status()
            return r.json()
        ra = r.headers.get("Retry-After", "")
        if ra.isdigit():
            delay = int(ra)                  # the server knows best
        else:
            delay = random.uniform(0, min(cap, base * 2 ** attempt))
        time.sleep(delay)
    raise RuntimeError(f"gave up on {url} after {max_tries} tries")

At the DAG level, two Airflow features do the rest. Task arguments retries, retry_delay, retry_exponential_backoff=True and max_retry_delay space out whole-task retries. And a pool (for example pool="partner_api" with 4 slots) caps how many tasks call that API at once, however many DAGs run in parallel. A pool is a concurrency limit you apply to yourself, so a backfill of 300 dates doesn't become 300 simultaneous callers.

Watch the response headers too. If RateLimit-Remaining is near zero, slow down before you get the 429. (RateLimit-* headers come from an IETF draft that's still changing, so names and formats vary between providers.)

🧠 Extra takeaways

• Immediate or fixed-delay retries synchronize clients into waves; jitter spreads them out.
• Honor Retry-After, cap attempts, and retry only idempotent requests or ones with an idempotency key.
• Retry 429, 503 and timeouts; don't retry 400 or 403.
• Airflow pools limit your own concurrency against a partner's API; task retries add backoff between attempts.