L07 · How Packets Travel

Module 2: Networking for Defenders · ~20 min · Lesson 7 of 71

🎯 What you'll learn

• How one API call gets wrapped in layers, and which defenses can see which layer
• What ports are, and listening sockets vs established connections
• How the TCP 3-way handshake works, and how a SYN flood abuses it
• TCP vs UDP, and why binding to 0.0.0.0 instead of 127.0.0.1 matters

🔥 Why this matters

A teammate asks, "Can we block /admin with a security group?" Another says, "The pod's port is open but curl just hangs." A third left a Java debug port in a Dockerfile because "it's inside the cluster." All three are networking questions with security answers. Every control in this module works at a particular layer, and it can't protect what it can't see.

1. Layers: an envelope inside an envelope

A customer's app calls POST https://api.example.com/v1/payments. Your code thinks in HTTP: method, path, headers, body. But HTTP can't travel by itself. The operating system wraps it in layers, each adding a header for one job, like a letter in an envelope, in a mailbag, on a truck.

This is encapsulation: each layer treats everything from the layer above as opaque payload and puts its own header in front. The TCP/IP model has four layers:

 Layer (TCP/IP)    OSI #  Adds                   Example value
 ───────────────────────────────────────────────────────────────
 Application       L7     HTTP method, path,     POST /v1/payments
                          headers, body          Authorization: ...
   (TLS wraps the HTTP here; covered in L10)
 Transport         L4     ports, sequence nums,  src 52814 → dst 443
                          flags (SYN, ACK...)    TCP
 Internet          L3     source and dest IP     203.0.113.50 →
                                                 198.51.100.10
 Link              L2     MAC addresses (for     Wi-Fi / Ethernet
                          the next hop only)     frame

 On the wire, outermost first:
 ┌───────┬──────────┬───────────┬──────────────────────────────┐
 │ Frame │ IP hdr   │ TCP hdr   │ TLS record (encrypted HTTP)  │
 │ hdr   │ who→who  │ port→port │ POST /v1/payments {...}      │
 └───────┴──────────┴───────────┴──────────────────────────────┘

"L4" and "L7" come from the older seven-layer OSI model. In daily use, L3 means IP, L4 means TCP/UDP ports, and L7 means the application protocol, usually HTTP. Each hop unwraps only as far as it needs: a router reads the IP header, an L4 load balancer reads IP and port, and only something that terminates TLS and parses HTTP, like Kong, can read the path and the token.

2. What defenders see at each layer

Follow the payment request into a platform like the one in these examples. Each box can only enforce rules about what it reads:

 [ mobile app ]
      │ 1. TCP SYN to 198.51.100.10:443
      ▼
 ┌─ NACL (subnet) ─────────── sees L3/L4: IPs, ports. Stateless.
 │    ▼
 │  NLB ───────────────────── L4: forwards TCP to Kong pods
 │    │ 2. same TCP stream (TLS still encrypted)
 │    ▼
 │  Security group ────────── sees L3/L4. Stateful: replies are
 │    │                       allowed automatically.
 │    ▼
 │  ( Kong pod ) ──────────── L7: decrypts TLS, reads path,
 │    │                       headers, JWT; rate limits per user
 │    │ 3. new TCP connection: GET http://payments-svc:8080
 │    ▼
 │  ( payments-service pod ) ─ L7: business rules, authorization
 └────────────────────────────────────────────────────────────

A security group is a stateful allow-list on a network interface: if a connection was allowed in, its replies go out without a rule. A network ACL (NACL) is a stateless list on a subnet, evaluated in number order, with allow and deny rules; return traffic needs its own rule.

So, "Can we block /admin with a security group?" No. The path is inside encrypted HTTP; the security group sees only TCP 443. Blocking a path is Kong's job (or a WAF's, L12). The reverse holds too: Kong can't stop a packet that never reaches it, so "only Kong may reach payments-service on 8080" is a job for security groups or Kubernetes network policies. That's defense in depth (L04): each layer catches what the others can't see.

3. Ports and sockets

An IP address gets a packet to a machine; a port (a 16-bit number, 0 to 65535) gets it to the right program. RFC 6335 defines well-known ports (0-1023, like 443), registered ports (1024-49151, like MongoDB's 27017), and dynamic ports (49152-65535). Clients use temporary ephemeral ports: macOS picks from 49152-65535, Linux by default from 32768-60999.

A socket is an endpoint a program has open. Two states matter most:

• LISTEN: a server is waiting for new connections on a port. Every listening socket is an entry point on your attack surface (L03).
• ESTABLISHED: a live connection between two IP-and-port pairs.

The address a server binds to decides who can reach it. 127.0.0.1 (loopback) means "only this machine." 0.0.0.0 (shown by lsof as *) means "every network interface," so a mock server on 0.0.0.0:8080 is reachable from the whole coffee-shop Wi-Fi unless a firewall stops it.

Good tools default to loopback. Since MongoDB 3.6, mongod binds to localhost unless you set net.bindIp. Kong's admin API defaults to 127.0.0.1:8001, while its proxy listens on 0.0.0.0:8000 and :8443. One catch: in a container, a service usually must bind to 0.0.0.0, because Kubernetes traffic arrives on the pod's IP. There, network policies and security groups decide who can reach it.

4. The TCP 3-way handshake

TCP gives a reliable, ordered byte stream. Before data moves, the two sides agree on starting sequence numbers, used to number every byte so lost data can be resent and reordered. That agreement is the 3-way handshake (RFC 9293):

  client 10.0.4.17:52814              server 10.0.9.30:8080
  (Kong pod)                          (payments-service pod)
        │                                     │
        │ 1. SYN, seq=1000                    │
        │────────────────────────────────────▶│ allocates state
        │                                     │ (SYN-RECEIVED)
        │ 2. SYN-ACK, seq=5000, ack=1001      │
        │◀────────────────────────────────────│
        │ 3. ACK, ack=5001                    │
        │────────────────────────────────────▶│ ESTABLISHED
        │                                     │
        │ 4. data: GET /v1/payments/PMT-1042  │
        │────────────────────────────────────▶│
        │ ...                                 │
        │ FIN / ACK (close)                   │
        │◀───────────────────────────────────▶│

Real sequence numbers are random 32-bit values. Two failure modes help when debugging. If nothing listens on the port, the server answers the SYN with RST (reset) and the client fails instantly ("connection refused"). If a firewall silently drops the SYN, the client waits until it times out. Instant failure means "reached the host, nothing there"; a hang usually means "something in between is dropping it."

5. SYN floods: attacking availability

In step 1, the server stores state for a half-open connection before the client has proven anything. A SYN flood sends huge numbers of SYNs, often from spoofed IPs, and never sends step 3. The half-open table fills and real customers get turned away. Nothing is read or changed: it's an attack on availability, the "A" in L01's CIA triad. Publicized in 1996, it's documented in RFC 4987.

The classic defense is SYN cookies: the server encodes the connection details in its own sequence number instead of storing them, and keeps no state until a valid ACK returns. Linux turns them on by default when the queue overflows. In AWS, Shield Standard (free for every account) mitigates common L3/L4 floods in front of services like load balancers. Kong's rate limiting is L7: it only sees connections that already finished the handshake.

6. TCP vs UDP

UDP skips all of that: no handshake, no ordering, no resending. It's fast and simple, and fits cases where a late answer is worthless or the application handles loss itself.

With no handshake, nobody checks the source IP. An attacker sends a small DNS query "from" a victim's address and the server sends a large reply to the victim: amplification, back in L08. And since HTTP/3 runs over UDP, "web traffic = TCP 443" is no longer always true.

🧪 Worked example: the forgotten debug port

A pod in payments behaves oddly, so you check what it listens on. Slim images often lack ss and netstat, but /proc/net/tcp is always there (made-up names):

$ kubectl -n payments exec payments-svc-7d9f-x2kq -- \
    cat /proc/net/tcp
  sl  local_address rem_address   st ...
   0: 00000000:1F90 00000000:0000 0A ...
   1: 00000000:138D 00000000:0000 0A ...
   2: 1E09000A:1F90 1104000A:CE4E 01 ...

Addresses and ports are hex; st 0A is LISTEN and 01 is ESTABLISHED. On x86 and most ARM nodes the IP's bytes print in reverse order, so read them right to left.

• Line 0: 0.0.0.0:8080 (1F90), the API. Expected.
• Line 1: 0.0.0.0:5005 (138D), on every interface. 5005 is the usual Java remote-debugging (JDWP) port, and anyone who can connect to JDWP can run code in the JVM.
• Line 2: a connection to 8080 from Kong's pod (1104000A backwards is 10.0.4.17, port 52814).

The cause is one line in the Dockerfile:

# Vulnerable: debug agent baked into the production image
ENTRYPOINT ["java",
  "-agentlib:jdwp=transport=dt_socket,server=y,address=*:5005",
  "-jar", "/app/payments.jar"]

The fix works at several layers. Remove the agent from production images (attack surface). Add a network policy that lets only Kong's pods reach port 8080, and nothing else in (least privilege, L04). Add a CI check that fails the build if jdwp appears in a production Dockerfile.

🛠 Try it (optional, ~5 min)

On your Mac, list what's listening:

$ lsof -i -P -n 2>/dev/null | grep LISTEN

-P -n print numbers instead of names (faster); 2>/dev/null hides harmless macOS warnings. Trimmed sample (DEVICE and SIZE columns removed):

COMMAND    PID USER  TYPE  NAME
ARDAgent  1421 you   IPv6  TCP *:3283 (LISTEN)
OneDrive  2368 you   IPv6  TCP [::1]:42050 (LISTEN)
idea     74834 you   IPv6  TCP 127.0.0.1:63342 (LISTEN)

* means all interfaces: ARDAgent (Apple Remote Desktop) is reachable from the network; the others are loopback only. Then try netstat -an -p tcp | grep LISTEN. It shows more, because lsof without sudo lists only your processes, and services like SSH (*.22, if Remote Login is on) run as root.

Now see binding for yourself. Start each server in its own tab, then check from a third:

$ python3 -m http.server 8000 --bind 127.0.0.1
$ python3 -m http.server 8001 --bind 0.0.0.0
$ lsof -i -P -n 2>/dev/null | grep ':800[01]'
Python  15480 you ... TCP 127.0.0.1:8000 (LISTEN)
Python  15482 you ... TCP *:8001 (LISTEN)

Port 8001 is open to anyone on your network who can reach your Mac. Stop both with Ctrl-C.

🧠 Key takeaways

• Each layer wraps the one above: HTTP inside TLS inside TCP inside IP inside a frame.
• Controls filter only what they see: security groups and NACLs see IPs and ports (L3/L4); only something that decrypts and parses HTTP, like Kong, sees paths and tokens (L7).
• Every LISTEN socket is attack surface; 127.0.0.1 keeps it local, 0.0.0.0 opens it to all.
• The handshake makes the server keep state first, which SYN floods exploit; SYN cookies fix it.
• "Connection refused" means you reached the host; a timeout usually means a filter dropped you.

📖 Words to know

⏭ Next time

L08: DNS, which turns api.example.com into an IP address, and how attackers poison it, tunnel data through it, and take over subdomains that point at nothing.

Why look at raw packets?

Logs tell you what an application thinks happened. A packet capture shows what actually crossed the wire: whether the SYN arrived, who sent the RST, whether the TLS handshake failed before any HTTP was sent. When Kong logs a 502 and the service logs nothing, a capture settles the argument.

tcpdump captures packets on the command line; Wireshark is the graphical tool that opens the same capture files (.pcap). Both are free. You'll capture only on your own loopback interface, which is safe and contains only your own test traffic.

Capture a handshake on your Mac

macOS ships tcpdump in /usr/sbin, but capturing needs sudo. The loopback interface is lo0 on macOS (lo on Linux). Use three terminal tabs:

# tab 1: a throwaway web server on loopback only
$ python3 -m http.server 8000 --bind 127.0.0.1

# tab 2: capture TCP on port 8000; -n no DNS, -S full seq nums
$ sudo tcpdump -i lo0 -n -S tcp port 8000

# tab 3: make one request
$ curl -s -o /dev/null http://127.0.0.1:8000/

Annotated output

Representative output, trimmed: timestamps shortened, 127.0.0.1. dropped from the addresses, and TCP options removed. Your ports, sequence numbers, and lengths will differ.

 time   from    to       flags  details
 .101  62998 > 8000   [S]   seq 1830410954           (1)
 .101  8000 > 62998   [S.]  seq 2972511200,          (2)
                            ack 1830410955
 .101  62998 > 8000   [.]   ack 2972511201           (3)
 .102  62998 > 8000   [P.]  length 78  GET / ...     (4)
 .102  8000 > 62998   [.]   ack 1830411033
 .105  8000 > 62998   [P.]  length 1024  HTTP/1.0 200
 .106  8000 > 62998   [F.]  (server closes)          (5)
 .106  62998 > 8000   [F.]  (client closes)
 .106  8000 > 62998   [.]   ack (done)

1. SYN ([S]) from the client's ephemeral port to 8000, with a random starting sequence number.
2. SYN-ACK ([S.]; the dot means ACK). The server's own random sequence number, and ack = client's seq + 1.
3. ACK ([.]). Handshake done; both sides are ESTABLISHED.
4. PSH-ACK ([P.]) carries data: 78 bytes of HTTP request. Because this is plain HTTP on loopback, you can read it. On port 443 you'd see only TLS records, plus the server name in the unencrypted TLS ClientHello (the SNI, in L10).
5. FIN ([F.]) from each side closes the connection. Python's server speaks HTTP/1.0 and closes after one response.

Other flags to recognize: [R] is a reset. Stop the server and run curl again, and you'll see a [S] answered immediately by [R.], which is what curl reports on macOS as "Couldn't connect to server." Afterward, netstat -an -p tcp | grep 8000 shows the closed connection in TIME_WAIT for a short while, which is normal.

What defenders look for

In Wireshark, the display filters tcp.flags.syn == 1 && tcp.flags.ack == 0 (new connection attempts) and tcp.flags.reset == 1 (resets) find these quickly.

Captures are sensitive data

A capture records every byte, including tokens, cookies, and any plain-text personal data. On shared or production systems (including kubectl debug with a tcpdump image on EKS), capture only with approval, filter to the narrowest host and port, keep files short-lived, and treat the .pcap like any other secret. Never capture on networks you don't own.

🧠 Extra takeaways

• tcpdump flags: [S] SYN, [S.] SYN-ACK, [.] ACK, [P.] data, [F.] FIN, [R] reset.
• Captures answer "did the packet arrive, and who hung up?" better than any app log.
• With TLS you see the handshake and the SNI, not the HTTP.
• A capture is as sensitive as the data in it.

Metadata, not content

You can't packet-capture a whole AWS VPC, but you can turn on VPC Flow Logs: AWS records a summary of the IP traffic through each network interface (ENI) and sends it to CloudWatch Logs or S3. Think of it as a phone bill rather than a recording: who called whom, when, for how long, and whether the call was connected. No words.

Reading one record

Each record summarizes one flow, identified by a 5-tuple (source and destination IP and port, plus protocol), during an aggregation interval. The default format has 14 fields. Made-up record, split across lines:

2 111122223333 eni-0a1b2c3d4e5f60789
203.0.113.77 10.20.9.30 51544 27017 6
3 180 1762430400 1762430401 REJECT OK

Story: someone on the internet tried to open a TCP connection to MongoDB, and your filters dropped it. That's good news and also a signal: is 27017 published somewhere it shouldn't be?

Custom formats can add fields such as tcp-flags (FIN = 1, SYN = 2, RST = 4, SYN-ACK = 18, OR-ed together over the interval), flow-direction, vpc-id, and pkt-srcaddr / pkt-dstaddr.

Questions it answers well

• "Did anything reach this port?" Filter dstport = 27017 and look at srcaddr.
• "Who is scanning us?" One source with REJECTs across many destination ports.
• "Where is data going?" Large bytes to an unfamiliar external IP can mean exfiltration.
• "Is my security group change working?" REJECTs on the denied port, ACCEPTs on the allowed one.

In CloudWatch Logs Insights, a first query might be:

filter action = "REJECT"
| stats count(*) as rejects by srcAddr, dstPort
| sort rejects desc
| limit 20

What it doesn't tell you

 Flow log sees                 Flow log does NOT see
 ─────────────────────────     ──────────────────────────────
 IPs, ports, protocol          HTTP path, headers, JWT, user
 packet and byte counts        the payload itself
 ACCEPT / REJECT               why the app returned 403 or 500
 per-interface, per-interval   individual packets, exact timing

AWS also lists traffic that flow logs don't record at all, including: DNS queries to the Amazon DNS resolver, traffic to and from the instance metadata service at 169.254.169.254, the Amazon Time Sync Service at 169.254.169.123, DHCP, and ARP. Two consequences matter to defenders. DNS tunneling (L08) through the AWS resolver won't show up here; you need Route 53 Resolver query logs. And an SSRF attack that steals credentials from 169.254.169.254 (L35) leaves no flow log trace; you'd look at CloudTrail for the stolen credentials being used.

EKS gotchas

With the AWS VPC CNI, pods get IPs from the VPC, usually secondary addresses on the node's ENI. AWS documents that when traffic goes to a secondary IP, the default dstaddr shows the ENI's primary IP, meaning the node, not the pod. Add pkt-srcaddr and pkt-dstaddr to a custom format to see pod IPs. Traffic between two pods on the same node generally never leaves the node, so it doesn't pass through an ENI and won't appear at all. For pod-to-pod visibility you need cluster-level tools such as network policy logging.

Flow logs are also not real time. On Nitro instances the aggregation interval is 1 minute or less; AWS says delivery typically takes about 5 minutes to CloudWatch Logs and about 10 to S3, on a best effort basis.

🧠 Extra takeaways

• Flow logs are L3/L4 metadata: who talked to whom, on which port, accepted or rejected.
• They never contain payloads, paths, or user identities; Kong logs and app logs cover L7.
• DNS to the Amazon resolver and metadata service traffic are not logged.
• On EKS, add pkt-srcaddr / pkt-dstaddr to see pod IPs; same-node pod traffic is invisible.