L51 · Kubernetes security, part 2

Module 7: Cloud, Containers, and Kubernetes · ~20 min · Lesson 51 of 71

🎯 What you'll learn

• How to lay out NetworkPolicies for a whole namespace, what they can't do, and what enforces them on EKS
• The three Pod Security Standards, and how Pod Security Admission labels enforce them
• How admission control works: webhooks (Kyverno, Gatekeeper) and the built-in ValidatingAdmissionPolicy
• What runtime security sees that every other layer misses (Falco, GuardDuty Runtime Monitoring)

🔥 Why this matters

An attacker finds a deserialization bug (L36) in quotes-api and gets a shell inside its pod. L50 limited what the pod's ServiceAccount and IAM role can do through APIs. Now the questions change. Is the process root, and can it reach the node? Can it connect to another team's MongoDB? Will anyone notice the shell? Each answer depends on a layer you configure today.

1. Four layers, four questions

L50 secured the API server. This lesson secures the workloads behind it.

 CI / kubectl apply
       │ 1. "May this object exist?"
       ▼
 ┌────────────────────────┐  Pod Security Admission, Kyverno,
 │ Admission control      │  Gatekeeper, ValidatingAdmission-
 └───────────┬────────────┘  Policy
             │ 2. stored, scheduled, container starts
             ▼
 ┌────────────────────────┐  "What may it do to its node?"
 │ Pod securityContext    │  (Pod Security Standards)
 └───────────┬────────────┘
             │ 3. it opens connections
             ▼
 ┌────────────────────────┐  "Who may it talk to?"
 │ NetworkPolicy (CNI)    │
 └───────────┬────────────┘
             │ 4. it runs for weeks
             ▼
 ┌────────────────────────┐  "Is it doing something odd now?"
 │ Runtime detection      │  Falco, GuardDuty
 └────────────────────────┘

The first three prevent; the last one detects what got past them.

2. NetworkPolicy, namespace by namespace

L12 introduced NetworkPolicy, pod-level firewall rules, and its traps: pods are open until a policy selects them, and nothing happens unless the network plugin (the CNI) enforces it.

The pattern that scales is the same in every namespace: deny everything, then allow each flow by name.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: default-deny-all, namespace: payments}
spec:
  podSelector: {}                      # every pod here
  policyTypes: ["Ingress", "Egress"]   # no rules = none allowed

Then one small policy per arrow:

 kong ns ──8080──▶ quotes-api ──27017──▶ 10.20.8.0/24 (Mongo)
                       │
                       ├──53 UDP/TCP──▶ kube-system/kube-dns
                       └──443───────▶ VPC endpoints (STS, S3)
 monitoring ns ──8081──▶ quotes-api (metrics only, L38)
 anything else ──X──▶ dropped

Who enforces it on EKS? The Amazon VPC CNI does, once enableNetworkPolicy is set in the add-on's configuration (L12 lists the version and node requirements). Calico or Cilium can enforce instead. Without one of these, kubectl get networkpolicy shows your rules and packets ignore them.

Standard NetworkPolicy has limits: no deny rules or cluster-wide policy (a namespace owner can always add an allow), no DNS names (api.partner.example must be a CIDR), no HTTP paths, and no identity or encryption. Cilium (CiliumNetworkPolicy: HTTP rules, toFQDNs, deny) and Calico (GlobalNetworkPolicy with ordered deny rules) fill the first three. An upstream cluster-wide admin policy API is still alpha and changing. For identity, a service mesh such as Istio or Linkerd runs mTLS between pods (L10's deep dive): NetworkPolicy says "this IP may reach that port", mTLS says "this caller is provably quotes-api".

3. Pod Security Standards

Here is a pod spec that looks like a harmless debugging aid:

securityContext: {privileged: true}
volumes:
- name: host
  hostPath: {path: /}

That's root on the node, and every other pod's secrets on it (deep dive 1). Kubernetes defines three Pod Security Standards, which are named levels of what a pod may ask for:

Restricted additionally requires:

• runAsNonRoot: true (and runAsUser not 0)
• allowPrivilegeEscalation: false
• capabilities.drop: ["ALL"] (only NET_BIND_SERVICE may be added back)
• seccompProfile.type: RuntimeDefault or Localhost
• volumes only of safe types: configMap, secret, emptyDir, projected, persistentVolumeClaim, downwardAPI, csi, ephemeral

Restricted doesn't require readOnlyRootFilesystem; set it anyway where the app allows.

4. Pod Security Admission: labels on the namespace

Pod Security Admission (PSA) is the built-in admission controller that enforces these levels. It replaced PodSecurityPolicy, which was removed in Kubernetes 1.25. You turn it on per namespace with labels:

apiVersion: v1
kind: Namespace
metadata:
  name: payments
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.33
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/audit: restricted

One trap catches nearly everyone: enforce checks Pods, not Deployments. warn and audit also check workload templates. So a non-compliant Deployment is created with a warning, and then its ReplicaSet fails to create any pods (the worked example).

On EKS, PSA is on, but a namespace with no labels runs at privileged.

5. Admission control beyond PSA

Rules like "images only from our ECR registry" need a policy engine. All of them plug into the same pipeline:

 request ─▶ authn ─▶ authz (RBAC, L50)
              │
              ▼
   mutating admission   (may change it: add defaults, sidecars)
              │
              ▼
   schema validation
              │
              ▼
   validating admission (may only say yes/no: PSA, VAP,
              │          webhooks)
              ▼
   persisted to etcd ─▶ scheduler ─▶ kubelet

You have three common choices:

• Validating (or mutating) admission webhooks: the API server sends the object to your HTTPS service and waits for an answer. Kyverno (policies in YAML) and OPA Gatekeeper (policies in Rego, L27) both run this way.
• ValidatingAdmissionPolicy (VAP): built in, GA since Kubernetes 1.30. Rules are CEL (Common Expression Language) expressions evaluated inside the API server. There's no extra service to run or keep healthy.
• PSA, for the pod-security basics.

A VAP that only allows images from your registry (made-up account ID):

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata: {name: ecr-images-only}
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups: [""]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["pods"]
  variables:
  - name: reg
    expression: "'111122223333.dkr.ecr.us-east-1.amazonaws.com/'"
  validations:
  - expression: >-
      object.spec.containers.all(c,
        c.image.startsWith(variables.reg)) &&
      (!has(object.spec.initContainers) ||
        object.spec.initContainers.all(c,
          c.image.startsWith(variables.reg)))
    message: "images must come from the team ECR registry"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata: {name: ecr-images-only}
spec:
  policyName: ecr-images-only
  validationActions: ["Deny"]
  matchResources:
    namespaceSelector: {matchLabels: {team: payments}}

Note the initContainers clause. Forgetting it is the classic bypass for image rules.

6. Runtime security: watching what actually runs

Admission checks the manifest once. It can't see bash spawning in a running pod after an RCE.

Falco (a CNCF graduated project) watches system calls on each node through eBPF and matches them against rules. A custom rule:

- rule: Shell in payments container
  desc: An interactive shell started in a payments pod
  condition: >
    spawned_process and container
    and proc.name in (bash, sh, zsh)
    and k8s.ns.name = "payments"
  output: >
    Shell in payments (user=%user.name
    pod=%k8s.pod.name cmd=%proc.cmdline)
  priority: WARNING

GuardDuty Runtime Monitoring is AWS's managed equivalent. A GuardDuty agent runs as an EKS add-on (a DaemonSet) and reports findings such as Execution:Runtime/ReverseShell, PrivilegeEscalation:Runtime/ContainerMountsHostDirectory and CryptoCurrency:Runtime/BitcoinTool.B. It's separate from GuardDuty's EKS Protection, which reads the cluster's audit log (L52).

Detection only helps if the alert reaches an on-call with a runbook (L13's alert fatigue).

🧪 Worked example: from "created" to running restricted

The payments namespace has the labels from section 4. CI applies an old manifest:

$ kubectl apply -f quotes-api.yaml -n payments
Warning: would violate PodSecurity "restricted:v1.33":
allowPrivilegeEscalation != false (container "app" must
set securityContext.allowPrivilegeEscalation=false), ...
deployment.apps/quotes-api created

CI is green. But no pods appear:

$ kubectl get events -n payments \
    --field-selector reason=FailedCreate
... replicaset/quotes-api-6f7c9d  Error creating: pods
"quotes-api-6f7c9d-" is forbidden: violates PodSecurity
"restricted:v1.33": runAsNonRoot != true, seccompProfile ...

(Output wrapped and shortened.) Fixed pod template:

spec:
  automountServiceAccountToken: false  # no k8s API calls
  securityContext:                     # pod level
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile: {type: RuntimeDefault}
  containers:
  - name: app
    image: 111122223333.dkr.ecr.us-east-1.amazonaws.com/q@sha256:...
    securityContext:                   # container level
      allowPrivilegeEscalation: false
      capabilities: {drop: ["ALL"]}
      readOnlyRootFilesystem: true     # not required; do it
    volumeMounts:
    - {name: tmp, mountPath: /tmp}     # app still needs /tmp
  volumes:
  - {name: tmp, emptyDir: {}}

Two gotchas. If the image's USER is a name like app, the kubelet can't prove it's non-root and fails with CreateContainerConfigError; set runAsUser or a numeric USER (L49). And skip the token only if the pod never calls the Kubernetes API (IRSA uses its own projected token).

Make CI wait on kubectl rollout status so "created, no pods" fails the build.

🛠 Try it (optional, ~5 min)

Read-only, if you have access to a dev namespace on a cluster (EKS for the last command; replace <ns> and <cluster>):

kubectl get ns <ns> --show-labels          # any pod-security labels?
kubectl get networkpolicy -n <ns>          # any default-deny?
kubectl get validatingadmissionpolicies    # cluster-scoped
kubectl get validatingwebhookconfigurations
J='{range .items[*]}{.metadata.name}{"\t"}'
J+='{.spec.containers[*].securityContext}{"\n"}{end}'
kubectl get pods -n <ns> -o jsonpath="$J"
aws eks describe-addon --cluster-name <cluster> \
  --addon-name vpc-cni --query addon.configurationValues

No enforce label means the namespace is privileged. An empty securityContext means defaults, often root. Policies without enableNetworkPolicy in the add-on config (and no Calico or Cilium) aren't enforced. If RBAC denies a command, just note which.

🧠 Key takeaways

• Default-deny ingress and egress per namespace, then one allow per flow; on EKS, confirm something enforces it.
• Baseline blocks host access; Restricted adds non-root, no privilege escalation, drop ALL and seccomp.
• PSA is per-namespace labels; enforce checks Pods, so non-compliant Deployments "succeed" and never start.
• VAP (CEL, built in) or Kyverno/Gatekeeper (webhooks) enforce your own rules at admission.
• Runtime tools (Falco, GuardDuty Runtime Monitoring) catch what passed every check, if someone answers.

📖 Words to know

⏭ Next time

L52 moves up to the cloud account: CloudTrail, VPC Flow Logs and GuardDuty, and the first five things an attacker does after landing in AWS.

The Baseline level can look like box-ticking. It isn't. Each field it blocks is a known way out of the container and onto the node, and the node is the prize.

Why the node matters

A node holds far more than one pod's secrets:

• every mounted Secret and ServiceAccount token of every pod scheduled there, under /var/lib/kubelet/pods/
• the kubelet's own credentials, which can read objects for the pods on that node
• the container runtime socket (containerd.sock), which can start any container
• the node's IAM role, through the instance metadata service

 1. RCE in quotes-api (L36)
        │
        ▼
 2. pod is privileged, or mounts hostPath /
        │ mount or read the node's filesystem
        ▼
 3. read tokens of every pod on the node
        │ /var/lib/kubelet/pods/*/volumes/...
        ▼
 4. one of them is a CI or operator ServiceAccount
        │ with wide RBAC (L27's "secretly admin" verbs)
        ▼
 5. cluster-wide access; node IAM role on the side

What each field gives away

GuardDuty Runtime Monitoring has findings for several of these, for example PrivilegeEscalation:Runtime/ContainerMountsHostDirectory, PrivilegeEscalation:Runtime/DockerSocketAccessed and PrivilegeEscalation:Runtime/CGroupsReleaseAgentModified. Detection is the backup; not granting the field is the fix. (Don't try these on a shared cluster. Reading about them is enough.)

Worked example: reviewing a vendor's DaemonSet

A monitoring vendor's Helm chart renders this (trimmed):

kind: DaemonSet
spec:
  template:
    spec:
      hostNetwork: true
      hostPID: true
      containers:
      - name: agent
        securityContext: {privileged: true}
        volumeMounts:
        - {name: root, mountPath: /host}
      volumes:
      - {name: root, hostPath: {path: /}}

Ask what it needs, not what it asks for. A log shipper needs to read /var/log/pods; it doesn't need to write /. A reasonable negotiation:

 asks for               needs                  change
 hostPath /             read /var/log/pods     hostPath /var/log/pods,
                                               readOnly: true
 privileged: true       none (logs only)       remove
 hostPID, hostNetwork   none for logs          remove

Some agents genuinely need more (CNI plugins, Falco itself, GuardDuty's agent). Give them their own namespace labelled enforce: privileged, with tight RBAC so app teams can't deploy there, and write down why. Every app namespace stays at Restricted, and the exception list is short enough to audit.

🧠 Extra takeaways

• The node holds every pod's tokens, the runtime socket and an IAM role; escaping one pod reaches all of them.
• privileged, hostPath, hostPID, hostNetwork and SYS_ADMIN are each a known escape path.
• hostNetwork pods bypass NetworkPolicy and the IMDSv2 hop-limit protection.
• Isolate genuinely privileged agents in their own namespace with a documented exception.

Switching a busy namespace straight to enforce: restricted can stop the next deployment cold. Every admission tool has a "tell me, don't block me" mode. Use it first.

Step 1: measure with a dry run

Server-side dry run evaluates a label change against the pods already running, without saving it. It needs permission to patch namespaces, so a platform engineer may have to run it:

$ kubectl label --dry-run=server --overwrite ns payments \
    pod-security.kubernetes.io/enforce=restricted
Warning: existing pods in namespace "payments" violate the
new PodSecurity enforce level "restricted:latest"
Warning: quotes-api-6f7c9d-x2k4p: allowPrivilegeEscalation
!= false, unrestricted capabilities, runAsNonRoot != true
namespace/payments labeled (server dry run)

(Output wrapped; exact wording varies by version.)

Step 2: warn and audit, then enforce

 week 1        week 2-3                week 4
 warn+audit ─▶ fix manifests, watch ─▶ enforce, pinned
 (nothing      warnings in CI logs     enforce-version
  blocked)     and audit log           (e.g. v1.33)

Pin enforce-version rather than latest, so a cluster upgrade can't tighten the rules under you without a reviewed change. The same pattern exists everywhere:

Webhooks can take the cluster down

A webhook is a network call in the path of every matching request. Two settings decide what happens when it's slow or down:

• failurePolicy: Fail rejects the request if the webhook can't answer (fail closed); Ignore lets it through (fail open). The default in the v1 API is Fail.
• timeoutSeconds: default 10, maximum 30. A slow webhook makes every kubectl apply slow.

The classic outage: a policy engine's webhook matches every namespace with failurePolicy: Fail. Its pods crash. Now nothing can be created, including the engine's own replacement pods. That's L04's fail-closed trade-off at cluster scale. Fail closed is right for security rules, so make the webhook hard to break instead:

kind: ValidatingWebhookConfiguration
webhooks:
- name: policy.example.com
  failurePolicy: Fail
  timeoutSeconds: 5
  namespaceSelector:
    matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: NotIn
      values: ["kube-system", "policy-engine"]

Exclude the system namespaces and the engine's own namespace, run several replicas with a PodDisruptionBudget, and alert when the webhook's error rate or latency climbs. Those excluded namespaces are now unguarded, so lock down who can deploy into them with RBAC (L50). For simple rules, a VAP avoids the problem: it runs inside the API server, so there's no extra service to fail.

🧠 Extra takeaways

• Dry-run a PSA label first; it lists the running pods that would violate it.
• Go warn/audit → fix → enforce, and pin enforce-version.
• Webhooks default to fail closed; exclude system namespaces and keep the engine highly available.
• Prefer VAP for simple rules: no webhook to go down.