L12 · Firewalls, Segmentation, Proxies, and WAFs

Module 2: Networking for Defenders · ~20 min · Lesson 12 of 71

🎯 What you'll learn

• The difference between north-south and east-west traffic, and why the second is the one attackers love
• How stateful and stateless filters differ, using AWS security groups and NACLs
• How Kubernetes NetworkPolicy segments a cluster, and the one setting on EKS that decides whether it does anything at all
• What a reverse proxy, an API gateway, and a WAF each protect, and what none of them can see

🔥 Why this matters

In L06, the attackers got into one Equifax web app and then queried 51 databases, because nothing inside the network said "this app has no business talking to that database." The front door had controls; the hallways didn't. Today you'll look at the controls on both: the ones at the edge of a typical API stack, and the ones between the services inside it.

1. Two directions of traffic

Picture a customer's request for their policy. It comes in from the internet, crosses AWS's edge, reaches Kong, then policy-service, then MongoDB. Some of those hops cross the edge of your network; others happen entirely inside it.

North-south traffic is traffic that crosses the edge: internet to your VPC, or your VPC out to the internet. East-west traffic is traffic between systems inside your environment: Kong to a service, a service to MongoDB, one pod to another.

             [ Customer: mobile app ]
                        │ HTTPS
   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─│─ ─ ─ internet │ AWS ─ ─ ─ ─ ─ ─ ─
                        ▼                         ▲
        ┌──────────────────────────────┐          │
        │ ALB + AWS WAF                │    NORTH-SOUTH
        └──────────────┬───────────────┘    (crosses the
                       │ HTTPS                 edge)
   ┌─ EKS cluster ─────▼──────────────────────────────────┐
   │   ( Kong ) ──HTTP :8080──▶ ( policy-service )        │
   │                                    │                 │
   │   ( batch-pod ) ─ ─ ─ ─ ─ ?─ ─ ─ ─ ┤  EAST-WEST       │
   │   ( any other pod ) ─ ─ ─ ?─ ─ ─ ─ ┤  (inside: who    │
   │                                    │   may call whom?)│
   └────────────────────────────────────│─────────────────┘
                                        │ TLS :27017
                                        ▼
                           ═══ MongoDB (private) ═══

Most teams guard north-south well: a WAF, TLS, Kong's auth plugins. East-west is often wide open, and it's the path an attacker takes after the first foothold (lateral movement, L03). Segmentation means dividing a network into zones and allowing only the flows each zone needs, so one compromised pod can't reach everything.

2. Firewalls: stateful vs stateless

A firewall is a control that allows or blocks network traffic based on rules, usually about source, destination, port, and protocol (the L07 layers 3 and 4). The big design choice is whether it remembers connections.

When policy-service opens a connection to MongoDB, it sends from a random high ephemeral port (say 50432) to port 27017. The reply comes back from 27017 to 50432.

  policy-service            filter               MongoDB
  10.0.4.17:50432 ── SYN to :27017 ───▶ ✔ rule: allow 27017 in
                                          │
  10.0.4.17:50432 ◀── reply from :27017 ──┤
                                          │
     STATEFUL: "that's the reply to a connection I allowed"
               → passes automatically
     STATELESS: "packet to port 50432? I have no rule for it"
               → dropped, unless you allow 1024-65535 out

A stateful firewall tracks each connection, so replies to allowed traffic pass without a separate rule. A stateless firewall judges every packet on its own, so you must write rules for both directions.

AWS gives you one of each:

AWS recommends security groups as your main control and NACLs as coarse subnet guard rails. A security group rule can also reference another group instead of an IP range ("allow 27017 from the eks-nodes group"), which survives IP changes.

3. Segmentation inside Kubernetes

Many pods share one node, so security groups can't tell them apart. To segment pods, Kubernetes has NetworkPolicy: an object that says which pods may talk to which others, by label, namespace, IP range, and port.

Three facts trip people up:

1. Default allow. A pod no policy selects is "non-isolated": every pod in the cluster can reach it, and it can reach anything. Isolation starts only when a policy selects the pod.
2. Allow-only and additive. Standard NetworkPolicy has no deny rules. Once a pod is selected, only what some policy allows gets through; several policies combine as a union.
3. Something must enforce it. The Kubernetes docs say it plainly: a NetworkPolicy without a network plugin that implements it "will have no effect." The object is stored and kubectl get shows it, but packets flow as before.

On EKS, the Amazon VPC CNI can enforce policies (added in VPC CNI v1.14, 2023), but you have to switch it on with "enableNetworkPolicy": "true" in the add-on's configuration. Current AWS docs also list VPC CNI 1.21+, Linux EC2 nodes (not Fargate or Windows), and kernel 5.10+. Calico or Cilium can enforce instead. And a connection needs both sides to agree: the source pod's egress policy and the destination pod's ingress policy.

4. Proxies, gateways, and WAFs

A proxy is a server that sits in the middle and makes requests on someone else's behalf. Which side it works for is the whole difference:

 FORWARD PROXY: works for the clients (outbound)
   [ your pods ] ──▶ ( egress proxy ) ──▶ api.partner.example
   it decides which outside sites YOUR side may reach

 REVERSE PROXY: works for the servers (inbound)
   [ the internet ] ──▶ ( ALB / Kong ) ──▶ policy-service
   clients think they're talking to the proxy; it decides
   what reaches YOUR servers

An API gateway like Kong is a reverse proxy that understands APIs. For security it can: authenticate callers (JWT, OAuth, mTLS, key-auth plugins), rate-limit per consumer, cap request size (Kong's request-size-limiting plugin defaults to a generous 128 MB, so set your own), allow or deny by IP, and log every call. What it can't do is decide whether this customer may read this policy. Kong knows the caller is C-20417; only policy-service knows which policies belong to C-20417. That's object-level authorization, and it lives in your code (L40, L43).

A web application firewall (WAF) inspects HTTP requests for known attack patterns: SQL injection strings, ../../ path traversal, ${jndi: from Log4Shell. AWS WAF attaches to an ALB, CloudFront, or API Gateway, and managed rule groups like AWSManagedRulesCommonRuleSet and AWSManagedRulesKnownBadInputsRuleSet give you a maintained set of signatures.

A WAF is useful, and limited in four ways:

• Signatures only match what they know. A new trick, or an old one re-encoded, slips by.
• It sees only part of the request. Behind an ALB, AWS WAF inspects only the first 8 KB of the body (deep dive 2).
• It can't see business logic. A valid token reading someone else's policy looks normal.
• False positives. Strict rules block real customers, so teams set rules to "count only" and forget them.

So: the WAF filters known junk at the edge, Kong checks who is calling, and the service checks what they may do. That's L04's defense in depth, layer by layer.

🧪 Worked example: only Kong may call policy-service

Our threat model in L05 found that any pod could call policy-service directly and fake the X-Consumer-ID header. Here's the network fix (with the service still verifying the token, since this is one layer, not the only one). First, deny all ingress in the namespace:

# 1. default-deny: selects every pod in "policy", allows no
#    ingress. Egress is untouched (deep dive 1 covers that).
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: policy
spec:
  podSelector: {}
  policyTypes: ["Ingress"]

Then allow the one flow you want:

# 2. allow: only Kong pods, in the kong namespace, on 8080
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-kong-to-policy-service
  namespace: policy
spec:
  podSelector:
    matchLabels:
      app: policy-service
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kong
          podSelector:          # same list item = AND
            matchLabels:
              app.kubernetes.io/name: kong
      ports:
        - protocol: TCP
          port: 8080

Watch the indentation under from. With namespaceSelector and podSelector in one list item, both must match: Kong pods in the kong namespace. Put a - before podSelector and they become two items joined by OR, which also lets in any pod labelled app.kubernetes.io/name: kong in the policy namespace.

Check what Kubernetes understood (made-up cluster):

$ kubectl describe networkpolicy allow-kong-to-policy-service \
    -n policy
Name:         allow-kong-to-policy-service
Namespace:    policy
Spec:
  PodSelector:     app=policy-service
  Allowing ingress traffic:
    To Port: 8080/TCP
    From:
      NamespaceSelector: kubernetes.io/metadata.name=kong
      PodSelector: app.kubernetes.io/name=kong
  Not affecting egress traffic
  Policy Types: Ingress

Then test the deny path from a throwaway pod in another namespace:

$ kubectl run np-test -n default --rm -it --restart=Never \
    --image=curlimages/curl -- \
    curl -sS -m 3 http://policy-service.policy:8080/health
curl: (28) Connection timed out after 3002 milliseconds
pod "np-test" deleted

A timeout (packets dropped), not a 403, is what a network block looks like. If you get {"status":"UP"}, nothing is enforcing the policy: check the CNI setting. One more EKS catch from AWS's docs: with the VPC CNI, a Service's port must equal the container port.

🛠 Try it (optional, ~5 min)

Part 1: your Mac's firewall (read-only, no sudo needed).

$ /usr/libexec/ApplicationFirewall/socketfilterfw --getglobalstate
Firewall is disabled. (State = 0)
$ /usr/libexec/ApplicationFirewall/socketfilterfw --getstealthmode
Firewall stealth mode is off
$ /usr/libexec/ApplicationFirewall/socketfilterfw --listapps
Total number of apps = 8
1 : /usr/local/libexec/remotepairingdeviced
             (Allow incoming connections)
3 : /usr/bin/python3
             (Allow incoming connections)
...

That's real output from a macOS 26 laptop. Is it on, and which apps may accept connections? Notice it only talks about incoming connections: the built-in macOS firewall filters inbound traffic per app, not outbound. A company laptop may show "disabled" because an endpoint security agent handles this instead; ask before changing it. L07's lsof -i -P | grep LISTEN shows what those apps are listening on.

Part 2: sketch your service's allowed flows. For one service you own, fill in a table like this, then compare it to what's actually enforced (kubectl get networkpolicy -n <ns>, and the security groups on your nodes and database):

If the honest answer for the last row is "allowed," you've found a segmentation gap.

🧠 Key takeaways

• North-south crosses your edge; east-west moves inside it, and is where attackers spread.
• Stateful filters (security groups) allow replies automatically; stateless ones (NACLs) need rules both ways.
• Kubernetes pods are reachable from every pod by default. NetworkPolicy fixes that only when the CNI enforces it: on EKS, enable it in the VPC CNI add-on.
• A gateway checks who is calling and how much; only the service can check what they may touch.
• A WAF blocks known patterns at the edge. It's bypassable, size-limited, and blind to business logic: one layer, not the defense.

📖 Words to know

⏭ Next time

L13: the Target breach of 2013, where stolen credentials from a heating and air-conditioning vendor, a network with no real segmentation, and security alerts nobody acted on led to 40 million stolen payment cards.

Why outbound matters

Teams write careful inbound rules and leave outbound wide open. AWS makes this easy: a new security group comes with an outbound rule allowing all traffic, and an unselected pod can reach anything. But almost every attack needs to go out at some point:

• Pulling a second stage. The classic Log4Shell exploit (CVE-2021-44228) made the vulnerable server connect out to the attacker's LDAP server to fetch code. Servers with strict egress filtering broke that step. Not every variant needed it (a DNS lookup could still leak data), so egress shrank the damage rather than removing it.
• Command and control. Malware calls home for instructions (L03's kill chain).
• Exfiltration. Stolen data has to leave somehow. In L06, nobody alerted on Equifax's unusual outbound volume.
• SSRF. A server tricked into fetching a URL for an attacker, often the cloud metadata service at 169.254.169.254 (Capital One, L47).

Egress control means limiting where your workloads may send traffic, so that a compromised pod can reach its database and its known partners, and nothing else.

Layers of egress control on AWS and EKS

 ( policy-service pod )
        │
        │ 1. NetworkPolicy egress: which pods, CIDRs, ports
        ▼
 [ node / pod security group ]
        │ 2. outbound SG rules (replace the default
        │    "allow all" rule)
        ├──────────────▶ VPC endpoints ──▶ S3, Secrets Manager,
        │                3. AWS APIs without leaving the VPC
        ▼
 [ AWS Network Firewall or egress proxy ]
        │ 4. domain allow list: api.partner.example only
        ▼
 [ NAT gateway ] ──▶ internet

1. NetworkPolicy egress works per pod and is the finest-grained layer. It works by IP and port, not domain name.
2. Security group outbound rules limit a whole node or (with security groups for pods) a pod.
3. VPC endpoints let pods reach AWS services privately, so "talk to S3" doesn't require "talk to the internet."
4. Domain filtering for the internet. AWS Network Firewall's domain list rules read the TLS SNI or HTTP Host header, so a client can claim a name it isn't really connecting to. AWS's docs suggest also matching on the IP the domain actually resolved to. A forward proxy is the alternative.

🧪 Worked example: default-deny egress for policy-service

# Deny all egress for every pod in "policy" ...
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: policy
spec:
  podSelector: {}
  policyTypes: ["Egress"]
---
# ... then allow DNS and MongoDB for policy-service only
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: policy-service-egress
  namespace: policy
spec:
  podSelector:
    matchLabels:
      app: policy-service
  policyTypes: ["Egress"]
  egress:
    - to:                     # CoreDNS in kube-system
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }
    - to:                     # MongoDB subnet (made up)
        - ipBlock:
            cidr: 10.20.8.0/24
      ports:
        - { protocol: TCP, port: 27017 }

The first thing that breaks with default-deny egress is DNS, which the Kubernetes docs call out for exactly this reason. Allow it first, or every call fails with a name-resolution error that looks like a networking outage. The second is the EC2 metadata endpoint: pods using IRSA or EKS Pod Identity don't need it, so leaving it blocked is a bonus against SSRF.

Roll it out in a dev namespace, watch for failures, and add only what's genuinely needed. The list of allowed destinations doubles as documentation of what the service depends on.

🛠 Try it (~3 min)

List the outbound destinations of one service you own: databases, AWS APIs, partner URLs, DNS, telemetry (an APM such as New Relic, logging). For each one, write which layer above could allow exactly that and nothing more. Anything you can't name is a destination you can't yet restrict.

🧠 Extra takeaways

• Most attacks need an outbound connection: second stages, command and control, exfiltration, SSRF.
• Defaults allow all egress, in AWS security groups and in Kubernetes.
• Combine NetworkPolicy egress, outbound SG rules, VPC endpoints, and a domain allow list.
• Default-deny egress breaks DNS first; allow it explicitly.

Anatomy of a rule

Every WAF rule, in AWS WAF or ModSecurity, has the same parts. Knowing them tells you where the gaps are:

 ┌ which part? ┐   ┌ cleaned how? ┐   ┌ match how? ┐   ┌ then? ┐
 │ URI path    │──▶│ URL-decode,  │──▶│ contains   │──▶│ Block │
 │ query string│   │ lowercase,   │   │ "${jndi:"  │   │ Allow │
 │ a header    │   │ HTML-decode  │   │ regex, SQLi│   │ Count │
 │ body (first │   │ ...          │   │ or XSS     │   └───────┘
 │ 8-64 KB)    │   └──────────────┘   │ detector   │
 └─────────────┘                      └────────────┘
   + oversize handling: what to do with bytes past the limit

Text transformations normalize the input before matching, because attackers encode payloads. %24%7Bjndi is ${jndi URL-encoded; without a URL-decode step, a rule looking for ${jndi misses it. Each gap below lives in one of these boxes.

🧪 Worked example: four ways past one signature

A rule blocks bodies containing ${jndi: (lowercased first). All payloads are harmless strings pointed at attacker.example:

Real managed rule groups are smarter than this one rule and were updated quickly for Log4Shell obfuscations. But the pattern holds: the WAF must normalize and parse the request exactly as your app does, and every difference is a gap. That's why the fix for Log4Shell was patching Log4j; the WAF bought time.

Closing the size gap

AWS's own guidance: add a rule before the body-inspecting rules that blocks bodies over the inspection limit (size greater than 8,192 bytes on an ALB, with oversize handling set to Match). The Core rule set's SizeRestrictions_BODY does roughly this at 8 KB. That rule also causes the most common false positive: a legitimate 20 KB JSON request gets blocked, someone sets the rule to Count, and gap 4 is open again. Better: allow large bodies only on the specific routes that need them, and enforce real limits in the gateway (Kong's request-size-limiting) and the service.

What no WAF rule can see

 GET /v1/policies/PL-100246
 Authorization: Bearer eyJ...   (valid token for C-20417)

 WAF:     no SQLi, no XSS, no traversal, normal size  → ALLOW
 Kong:    valid JWT, under rate limit                 → ALLOW
 Service: PL-100246 belongs to C-88310, not C-20417   → 404

Broken object-level authorization (L40) uses perfectly clean requests. So do credential stuffing with real passwords and a scraper paging slowly through your API. Rate-based rules and bot controls help with the last two; only your code fixes the first.

Running a WAF well

• Start new rules in Count mode, read the logs for a week, then switch to Block.
• Review what's still in Count every quarter; "temporary" Count is how protection quietly vanishes.
• Log WAF decisions and alert when the block rate jumps (an attack) or drops to zero (a broken rule).

🧠 Extra takeaways

• A rule = which part of the request + transformations + match + action + oversize handling.
• Encodings, parser differences, and size limits are the classic bypasses.
• Block oversize bodies explicitly, and scope exceptions to routes instead of disabling rules.
• A WAF buys time against known exploits; patching and authorization checks fix the problem.