L49 · Container security

Module 7: Cloud, Containers, and Kubernetes · ~20 min · Lesson 49 of 71

🎯 What you'll learn

• How image layers work, and why a file you deleted in a Dockerfile is still in the image
• Why containers should run as a non-root user, and what minimal and distroless images remove
• What image scanners (Trivy, Grype, ECR scanning) find, and what they can't
• How pinning by digest and signing an image prove which bytes run and who built them

🔥 Why this matters

A developer COPYs a settings.xml holding a private Maven repository token into the statements-svc image and rms it two lines later. The image goes to ECR. Anyone who can pull it (a vendor scanner, an attacker with a leaked read-only role) can recover that token in a minute. This isn't rare: a 2023 RWTH Aachen study of about 345,000 images, mostly from Docker Hub, found secrets (largely private keys and API keys) in about 8.5% of them.

1. An image is a stack of layers, and nothing is ever deleted

A container image is a set of read-only layers plus a small config file. Each Dockerfile instruction that changes files (RUN, COPY, ADD) adds a layer: a tar file of what changed. At start, the runtime stacks them into one view, with a thin writable layer on top.

 Dockerfile line                          layer contents
 FROM maven:3.9-eclipse-temurin-21  ───▶  L1-L6  OS, JDK, Maven
 COPY settings.xml /root/.m2/       ───▶  L7     settings.xml (TOKEN)
 RUN mvn package                    ───▶  L8     target/*.jar
 RUN rm /root/.m2/settings.xml      ───▶  L9     "whiteout" marker

 what `ls` shows in the container:   settings.xml is gone
 what `docker save` / a registry
 pull gives anyone:                  L7 still holds the token

A deletion is just a whiteout: a marker in a later layer that hides the file from the merged view. The bytes stay in the earlier layer, which ships with the image. The same goes for build arguments: Docker's docs warn that ARG values used in a RUN are visible in docker history and in provenance attestations, so ARG NEXUS_TOKEN leaks too.

Keep secrets out of layers with BuildKit secret mounts (RUN --mount=type=secret,... exposes a secret to one command without writing it to a layer), multi-stage builds (only the final stage ships), and a .dockerignore (COPY . . otherwise copies .git/ and .env). A secret that reached a pushed image is leaked: rotate it (L19).

2. Root in the container is root on the kernel

A container isn't a small VM. It's an ordinary Linux process that the kernel fences in with namespaces (what it can see) and cgroups (how much it can use). Every container on a node shares that node's kernel (first deep dive).

Unless the Dockerfile says otherwise, the process runs as root (UID 0): without user namespaces, the same UID 0 the host kernel knows. The runtime removes some of root's powers (capabilities), but if a kernel or runtime bug lets the process out, it lands on the node as root.

 container as root (default)        container as UID 10001
 ┌────────────────────────────┐     ┌────────────────────────────┐
 │ RCE in statements-svc      │     │ RCE in statements-svc      │
 │ uid=0: rewrite the app,    │     │ uid=10001: can't modify    │
 │ install tools (if a pkg    │     │ app files or install       │
 │ manager exists)            │     │ packages                   │
 └─────────────┬──────────────┘     └─────────────┬──────────────┘
               │ runtime bug (e.g. CVE-2019-5736)  │
               ▼                                   ▼
   escape lands as host root          escape lands as an
   on the EKS node: every pod's       unprivileged user
   secrets, the node's IAM role       (still bad, much less)

CVE-2019-5736 did exactly this: root inside a container could overwrite the host's runc binary and gain host root. The fix is one line, USER 10001. Use a numeric UID: Kubernetes' runAsNonRoot: true (L51) can only verify a number, and refuses an image whose user is a name like appuser. Listen on a port above 1024, since low ports need extra privileges.

3. Ship less: minimal and distroless base images

Everything in the image is something to patch and something an attacker can use. A full OS image brings a shell, curl, a package manager and hundreds of packages your service never calls.

• Slim images (eclipse-temurin:21-jre, python:3.12-slim) drop build tools and docs.
• Alpine is tiny but uses musl instead of glibc, which occasionally breaks native libraries.
• Distroless images (gcr.io/distroless/*) hold only the runtime and its libraries: no shell, no package manager. Their :nonroot tags run as UID 65532. Chainguard offers similar images.

Be precise about what this buys you. "No shell" doesn't stop exploitation: Log4Shell (L36) ran attacker code inside the JVM. It makes the next steps harder (no sh, no curl), and it cuts the scanner's CVE list sharply. The cost is debugging: use kubectl debug instead of shelling in.

4. Scan the image, and keep scanning it

An image scanner lists the packages in every layer (OS packages and language packages such as jars and wheels) and matches them against CVE databases.

Scan in CI (fail on fixable CRITICAL/HIGH), on push, and continuously: an image clean on Monday gains CVEs on Tuesday without changing a byte. Scan what's running, not just the repo (L36's deep dive).

Know the limits. A scanner only finds known CVEs in packages it can identify. It doesn't find bugs in your code, a root user, or (unless secret scanning is on) a leaked token.

5. Prove which bytes run, and who built them

A tag like statements-svc:1.4.2 is a movable label. Pinning by digest (@sha256:...) names the exact bytes; L14's deep dive covers why and how. But a digest only proves sameness. If an attacker pushes their own image, its digest is perfectly valid too.

Image signing answers "who built this?". The build pipeline signs the image digest with a private key; the cluster verifies the signature with the public key before running it (L17).

 CI pipeline                  registry (ECR)          cluster
 ┌──────────────┐ 1. push    ┌──────────────┐        ┌──────────────┐
 │ build image  │ ─────────▶ │ image        │ 3.pull │ admission    │
 │ get digest   │            │ sha256:9f74  │ ─────▶ │ check (L51): │
 │ sign digest  │ 2. push    │ signature    │        │ signed by    │
 │ (KMS key or  │ ─────────▶ │ for 9f74     │        │ OUR CI?      │
 │  keyless)    │            └──────────────┘        │ yes: run     │
 └──────────────┘                                    │ no: reject   │
                                                     └──────────────┘

Two common toolchains: cosign (from Sigstore) signs with a key, which can live in AWS KMS, or keyless with the CI job's OIDC identity (second deep dive). Notation (the CNCF Notary Project) signs with X.509 certificates, and AWS Signer has a Notation plugin for ECR.

A signature nobody checks protects nothing. Verify it at admission (L51), and name the expected signer, not just "any valid signature".

🧪 Worked example: hardening the statements-svc Dockerfile

Before:

FROM maven:3.9-eclipse-temurin-21          # (4) moving tag
WORKDIR /app
COPY . .                                   # (3) .git, .env too
COPY settings.xml /root/.m2/settings.xml   # (1) token in a layer
RUN mvn -B package -DskipTests
RUN rm /root/.m2/settings.xml              # (1) whiteout only
EXPOSE 8080
CMD ["java", "-jar", "target/statements-svc.jar"]
                                           # (2) runs as root, with
                                           # JDK, Maven, shell, apt

After:

# syntax=docker/dockerfile:1
FROM maven:3.9.9-eclipse-temurin-21@sha256:<digest> AS build
WORKDIR /src
COPY pom.xml .
COPY src ./src
RUN --mount=type=secret,id=mvn,target=/root/.m2/settings.xml \
    --mount=type=cache,target=/root/.m2/repository \
    mvn -B package -DskipTests

FROM gcr.io/distroless/java21-debian12:nonroot@sha256:<digest>
WORKDIR /app
COPY --from=build /src/target/statements-svc.jar app.jar
USER 65532:65532
EXPOSE 8080
CMD ["app.jar"]

Built with docker build --secret id=mvn,src=$HOME/.m2/settings.xml .. What changed:

1. The token is a secret mount: present while mvn runs, in no layer or history.
2. The runtime stage is distroless; only pom.xml, src/ and the jar are copied.
3. Both bases are pinned by digest (placeholder shown); Renovate keeps pins fresh.
4. USER is numeric (the :nonroot tag sets it too; explicit is better).
5. CMD ["app.jar"]: the distroless Java image's entrypoint is already java -jar.

CI then scans the image and signs its digest before anything deploys it.

🛠 Try it (optional, ~5 min)

Prove to yourself that rm doesn't remove anything. Local Docker (or Podman) only, fake data.

mkdir -p ~/tmp/l49 && cd ~/tmp/l49
cat > Dockerfile <<'EOF'
FROM alpine:3.22
RUN echo "token=FAKE-not-a-real-secret" > /tmp/creds.txt
RUN rm /tmp/creds.txt
EOF
docker build -q -t layer-demo .
docker run --rm layer-demo cat /tmp/creds.txt   # No such file
docker save layer-demo -o demo.tar
mkdir -p x && tar -xf demo.tar -C x
find x -type f | while read -r f; do
  tar -tf "$f" 2>/dev/null | grep -q 'creds.txt' && echo "$f" \
    && tar -tf "$f" | grep creds
done

You should see two layers: one with tmp/creds.txt, and a later one with tmp/.wh.creds.txt, the whiteout. Print the "deleted" file with tar -xOf <first-layer-path> tmp/creds.txt. (Newer Docker saves layers under blobs/sha256/, older under <id>/layer.tar; the loop finds either.) Clean up with rm -rf ~/tmp/l49.

🧠 Key takeaways

• Layers are append-only: a file deleted in a later layer still ships in the earlier one.
• Use BuildKit secret mounts and multi-stage builds; never COPY or ARG a secret.
• Run as a numeric non-root USER; root in a container is UID 0 on a shared kernel.
• Minimal or distroless images cut both the attacker's toolkit and the CVE noise.
• Scan in CI, on push, and continuously; pin by digest; sign the digest and verify at deploy.

📖 Words to know

⏭ Next time

L50 opens up the cluster itself: who gets through the Kubernetes API server's front door, and how a pod proves who it is to Kubernetes and to AWS.

The main lesson said a container is "just a process". This is what that means, and why it decides how much a container escape matters.

Three kernel features doing all the work

• Namespaces give a process its own view of something global: process IDs (pid), network interfaces (net), mounted filesystems (mnt), hostname (uts), shared memory (ipc), and user IDs (user). In its own pid namespace, your Java process sees itself as PID 1.
• cgroups limit and account for resources: CPU, memory, number of processes. Kubernetes resources.limits become cgroup settings, and the OOM killer enforces the memory one.
• Capabilities split root's power into about 40 pieces (CAP_NET_ADMIN, CAP_SYS_ADMIN, CAP_NET_RAW...). Container runtimes start root with a reduced default set. A seccomp profile can also block dangerous system calls.

  virtual machines                  containers
 ┌────────┐ ┌────────┐            ┌────────┐ ┌────────┐
 │ app    │ │ app    │            │ app    │ │ app    │
 │ libs   │ │ libs   │            │ libs   │ │ libs   │
 │ KERNEL │ │ KERNEL │            └───┬────┘ └───┬────┘
 └───┬────┘ └───┬────┘   namespaces, │          │
 ┌───┴──────────┴────┐   cgroups,    ▼          ▼
 │ hypervisor        │   caps    ┌───────────────────┐
 └───────────────────┘   ───────▶│ ONE shared kernel │
 boundary: hardware              └───────────────────┘
 virtualization                  boundary: kernel rules

A VM escape needs a hypervisor bug. A container escape needs a bug in the one kernel, or the runtime (runc, containerd), that every pod on the node shares, or a misconfiguration that hands the container host access (privileged mode, a mounted host path, the Docker socket).

Escapes that made the news

The pattern: the escape gives you whatever the container process was. As root without user namespaces, that's host root. That's why USER, dropped capabilities and seccomp (L51 shows the pod settings) matter even though they don't stop the first exploit. And it's why nodes need patching (L45: on managed node groups that's your job).

Two Kubernetes facts worth knowing, both version-dependent: seccomp is unconfined by default unless the pod asks for RuntimeDefault (or the kubelet's --seccomp-default is on), and user namespaces for pods (hostUsers: false), which map container root to an unprivileged host UID, are newer and need recent Kubernetes, kernel and runtime versions. Check your cluster before relying on either.

🛠 Try it: look at your own containers (~3 min, local only)

docker run --rm alpine:3.22 sh -c 'id; grep CapEff /proc/self/status'
docker run --rm --user 10001 alpine:3.22 \
  sh -c 'id; grep CapEff /proc/self/status'
docker run --rm alpine:3.22 ps        # PID 1 is ps itself
docker run --rm alpine:3.22 uname -r  # the host's kernel version

Typical output with Docker's defaults:

uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),...
CapEff: 00000000a80425fb
uid=10001 gid=0(root) groups=0(root)
CapEff: 0000000000000000

a80425fb is a bitmask of Docker's default capabilities for root; the non-root process has none. On macOS, uname -r shows the kernel of Docker Desktop's Linux VM: your containers share that kernel, the same way pods share an EKS node's.

🧠 Extra takeaways

• Namespaces control what a container sees, cgroups what it uses, capabilities and seccomp what it may do.
• Every container on a node shares one kernel; that's the real isolation boundary.
• An escape inherits the container's user, so non-root turns a disaster into an incident.
• Kubernetes doesn't apply seccomp by default; L51 shows how to turn it on.

Signing with a long-lived key brings back L19's problem: the key must be stored, rotated and protected, and if it leaks, anyone can sign "as CI". Keyless signing replaces the stored key with the identity the CI job already has.

The three parts of Sigstore

• Fulcio, a certificate authority. It accepts an OIDC ID token (L24), such as the one GitHub Actions gives a job, and issues a certificate valid for about 10 minutes that binds a fresh, throwaway key pair to that identity.
• Rekor, a public, append-only transparency log. Every signing event is recorded there, with a timestamp, so a signature can be checked long after the short-lived certificate expires.
• cosign, the client that drives both and stores the signature next to the image in the registry.

 GitHub Actions job (release.yml on main)
   │ 1. request OIDC token: iss=token.actions.githubusercontent.com
   ▼
 Fulcio ── 2. verify token, issue 10-min cert for an ephemeral
   │          key, identity = repo + workflow + ref
   ▼
 cosign ── 3. sign image DIGEST with ephemeral key; key discarded
   │
   ├── 4. signature + cert ──▶ registry (next to the image)
   └── 5. log entry ─────────▶ Rekor (public, timestamped)

 verifier: cert chains to Fulcio? identity = expected workflow?
           logged in Rekor while the cert was valid? ──▶ trust

Verifying: name the signer, exactly

Verification only means something if it pins who signed. For a made-up repository:

cosign verify \
  --certificate-oidc-issuer \
    https://token.actions.githubusercontent.com \
  --certificate-identity \
    "https://github.com/example-org/statements-svc/\
.github/workflows/release.yml@refs/heads/main" \
  registry.example.com/statements-svc@sha256:<digest>

The identity is the workflow file and branch that ran. The classic mistake is a loose check, such as --certificate-identity-regexp '.*' with only the GitHub issuer. Then any GitHub workflow, in any public repo, can produce an image that passes. It's L24's IRSA lesson again: checking the issuer without the subject trusts everyone that issuer serves.

Pin the branch too. A workflow on a feature branch, or one triggered from a fork's pull request, shouldn't be able to produce a "release" signature.

Choosing an option in a regulated company

The public-log point matters in regulated environments. Many teams sign with a KMS-held key, AWS Signer, or a private Sigstore deployment instead of the public instance.

What a signature still doesn't tell you

A valid signature says "our release pipeline produced this digest". It doesn't say the code was reviewed, the dependencies were safe, or the CI runner wasn't compromised (L56's SolarWinds case is exactly that). Signed attestations (an SBOM, build provenance under SLSA) carry more facts about how the image was built; L55 picks those up.

🧠 Extra takeaways

• Keyless signing swaps a stored signing key for CI's short-lived OIDC identity plus a transparency log.
• Always verify the exact identity (repo, workflow, branch) and issuer, never just "signed".
• The public Rekor log reveals signer identities; KMS keys, AWS Signer or private Sigstore avoid that.
• Signatures prove origin, not safety; pair them with scanning, review and provenance.