L50 · Kubernetes security, part 1

Module 7: Cloud, Containers, and Kubernetes · ~20 min · Lesson 50 of 71

🎯 What you'll learn

• What the Kubernetes API server checks on every request, and why it's the cluster's front door
• How your AWS identity becomes a Kubernetes user on EKS: the aws-auth ConfigMap vs access entries
• What a service account token is, why projected tokens are safer, and when not to mount one at all
• How pods get AWS credentials with IRSA or EKS Pod Identity, and how the two differ

🔥 Why this matters

In 2018 researchers at RedLock found Tesla's Kubernetes console open to the internet with no password. One pod held AWS credentials, and attackers were using the cluster to mine cryptocurrency. Today's lesson answers the two questions nobody had answered there: who is talking to the cluster, and what identity does each pod carry into AWS?

1. The API server is the front door

Every read or change of cluster state goes through kube-apiserver: kubectl, CI, controllers, kubelets and pods. Nothing else should touch etcd, the database behind it. Every request passes three gates in order:

 callers                    kube-apiserver
 you (kubectl) ──┐  ┌─────────────────────────────────────┐
 CI pipeline   ──┤  │ 1. AUTHENTICATION  who are you?     │
 pods (SA)     ──┼─▶│    IAM token, SA token, cert ✗ 401  │
 kubelets      ──┘  │ 2. AUTHORIZATION   may you do it?   │
                    │    Node, EKS access policies,       │
                    │    RBAC (L27)               ✗ 403   │
                    │ 3. ADMISSION   is this object OK?   │
                    │    mutate, validate (L51)   ✗ deny  │
                    └──────────────────┬──────────────────┘
                                       │ 4. store / read
                                       ▼
                                     etcd

Two facts shape everything else. Kubernetes has no user database: a "user" is just a name, plus groups, that an authenticator vouches for. And members of the system:masters group skip authorization entirely, so no RBAC change can remove their access. Never bind anything to system:anonymous or system:unauthenticated.

On EKS, the API endpoint is public by default (L03). Restrict it to known CIDRs or make it private, so a stolen credential also needs network access.

2. Authentication on EKS: from IAM role to Kubernetes user

When you run kubectl against EKS, your kubeconfig runs aws eks get-token. That builds a pre-signed STS GetCallerIdentity request with your AWS credentials (from IAM Identity Center, if your organization uses it; SigV4, L17's deep dive) and sends it as a bearer token. The API server replays it to STS to learn who you are, then maps that ARN to a Kubernetes user and groups.

 laptop                       EKS control plane           AWS STS
 kubectl ─1─▶ aws eks get-token
   │ 2. Bearer k8s-aws-v1.<pre-signed GetCallerIdentity>
   ▼
 API server ───── 3. replay the signed request ─────────▶ STS
            ◀──── 4. arn:...:assumed-role/
                     AWSReservedSSO_PaymentsDev_ab12/
                     dev@example.com
   │ 5. map ARN ─▶ k8s username + groups
   │    (access entry, or aws-auth ConfigMap)
   ▼
 authorization: EKS access policies + RBAC

Step 5 is where security lives. EKS has two ways to do it:

A cluster's authentication mode is CONFIG_MAP, API_AND_CONFIG_MAP, or API. Per AWS's docs, once access entries are on they can't be turned off, and AWS recommends migrating ConfigMap entries to access entries. If you can reach an EKS cluster, check its mode (read-only): aws eks describe-cluster --name <cluster> --query cluster.accessConfig.

Watch for this line in any aws-auth you inherit:

mapRoles: |
  - rolearn: arn:aws:iam::111122223333:role/ci-deployer
    username: ci-deployer
    groups: ["system:masters"]   # admin that RBAC can't revoke

IAM decides who you are; RBAC and access policies decide what you can do. Both only ever grant, so they add up (L27's deep dive covers the RBAC verbs that are secretly admin).

3. Service accounts: every pod has an identity

Every pod runs as a ServiceAccount (SA); if the spec names none, it gets default. Kubernetes mounts that SA's token at /var/run/secrets/kubernetes.io/serviceaccount/token, where any process in the pod, including an attacker with code execution (L36), can read it and use it.

 legacy SA token (Secret)          projected token (today)
 ─────────────────────────         ─────────────────────────
 stored as a Secret object         requested by the kubelet,
                                   lives only in the pod
 never expires                     ~1 hour, kubelet refreshes it
 no audience binding               bound audience (aud claim)
 outlives the pod                  invalid once the pod is gone

Projected (or bound) tokens became the default in Kubernetes 1.22, and since 1.24 Kubernetes no longer auto-creates the legacy Secret tokens. One EKS caveat, from AWS's docs: so that old client libraries keep working, EKS accepts these tokens for up to 90 days, and flags tokens older than an hour in the audit log (authentication.k8s.io/stale-token). A stolen token can therefore outlive its hour.

Most services (a Spring Boot API on MongoDB, an Airflow task) never call the Kubernetes API. For them the token is pure risk. Turn it off:

apiVersion: v1
kind: ServiceAccount
metadata: {name: payments-svc, namespace: payments}
automountServiceAccountToken: false

You can also set it on the pod spec, which wins over the SA. Give each workload its own SA, and never grant RBAC to default.

4. AWS credentials for pods: IRSA vs EKS Pod Identity

Pods shouldn't use the node's IAM role (shared by every pod; L45's IMDS hop limit blocks it) or access keys in a Secret (L19). EKS offers two ways to give a pod its own short-lived role:

 IRSA (2019)                         EKS Pod Identity (late 2023)
 SA annotated with a role ARN        association: ns + SA ▶ role
   │ webhook mounts a token,           │ webhook mounts a token,
   │ aud = sts.amazonaws.com           │ aud = pods.eks.amazonaws.com
   ▼                                   ▼
 SDK ▶ STS AssumeRoleWith-           SDK ▶ Pod Identity Agent (node)
       WebIdentity                         ▶ EKS Auth API
   │ trust policy: this cluster's      │ trust policy: service
   │ OIDC issuer + sub = ns:sa         │ pods.eks.amazonaws.com
   ▼                                   ▼
 temporary credentials               temporary credentials + tags

L24's deep dive walks through IRSA's trust policy, including the classic bug: a missing or wildcard sub condition lets any SA in the cluster assume the role. Note that the SA annotation grants nothing by itself; the trust policy decides.

AWS now recommends Pod Identity where possible. Either way: one role per workload (L46).

🧪 Worked example: what an attacker inherits from one pod

payments-svc has a remote code execution bug. What the attacker gets depends on the pod's identity.

Before:

spec:
  # no serviceAccountName ▶ runs as "default", token mounted
  containers:
  - name: app
    image: registry.example.com/payments-svc@sha256:<digest>
    env:
    - name: AWS_ACCESS_KEY_ID      # long-lived key from a Secret
      valueFrom: {secretKeyRef: {name: aws-keys, key: id}}
    # ...and AWS_SECRET_ACCESS_KEY the same way
# Someone once bound "default" to a Role with list on secrets,
# for a config reloader that no longer exists.

The attacker gets a never-expiring AWS key, plus a Kubernetes token that can list every Secret in payments (including other teams' credentials if they share the namespace).

After:

spec:
  serviceAccountName: payments-svc   # own SA, no RBAC bindings
  automountServiceAccountToken: false
  containers:
  - name: app
    image: registry.example.com/payments-svc@sha256:<digest>
    # no AWS keys: the SDK picks up Pod Identity credentials

aws eks create-pod-identity-association \
  --cluster-name dev-payments --namespace payments \
  --service-account payments-svc \
  --role-arn arn:aws:iam::111122223333:role/app/payments/payments-svc

Now the attacker gets no Kubernetes token, and AWS credentials that expire within hours and can do only what payments-svc needs. Delete the aws-keys Secret and deactivate the key in IAM. (IRSA and Pod Identity mount their own token, so automount off shouldn't break them; confirm in dev.)

🛠 Try it (optional, ~5 min)

Read-only commands, if you have a dev cluster and a namespace you can use there. Replace dev-team with that namespace.

kubectl auth whoami                       # needs Kubernetes 1.28+
kubectl auth can-i --list -n dev-team
kubectl get sa -n dev-team -o \
  custom-columns=SA:.metadata.name,AUTO:.automountServiceAccountToken
kubectl get pods -n dev-team -o \
  custom-columns=POD:.metadata.name,SA:.spec.serviceAccountName

Made-up output, trimmed:

ATTRIBUTE   VALUE
Username    arn:aws:sts::111122223333:assumed-role/
            AWSReservedSSO_PaymentsDev_ab12/dev@example.com
Groups      [system:authenticated]

SA             AUTO
default        <none>
payments-svc   false

What to look for:

1. whoami: the IAM role you came in as, and the groups it maps to.
2. can-i --list: any *, secrets, pods/exec, bind or impersonate (L27's deep dive)? The list can be incomplete when other authorizers, such as EKS access policies, are involved, so also ask directly: kubectl auth can-i list secrets -n dev-team.
3. SAs: <none> means unset, so tokens are mounted. Which pods run as default?

🧠 Key takeaways

• Every request passes authentication, authorization and admission at the API server; restrict who can even reach it.
• On EKS, IAM proves who you are; an access entry (preferred) or aws-auth maps that to a Kubernetes user and groups.
• system:masters bypasses RBAC; don't map anyone to it.
• Projected SA tokens are short-lived and pod-bound, but turn them off (automountServiceAccountToken: false) where unneeded.
• Give pods their own AWS role through Pod Identity or IRSA, never the node role or static keys.

📖 Words to know

⏭ Next time

L51 covers what happens inside the door: network policies, Pod Security Standards, admission control, and spotting an attacker at runtime.

Assume the worst: an attacker has code execution in a pod. The SA token is usually the first thing they look for, because it's a ready-made credential for the API server.

The attack path

 1. RCE in payments-svc (L36)
      │ read /var/run/secrets/kubernetes.io/serviceaccount/token
      ▼
 2. call https://kubernetes.default.svc from inside the pod
    (or from outside, if the EKS endpoint is public)
      │
      ▼
 3. "what can I do?"  ── SelfSubjectRulesReview (can-i --list)
      │
      ├─ list secrets ──────▶ DB passwords, API keys
      ├─ create pods ───────▶ mount any Secret, run as any SA
      └─ pods/exec ─────────▶ hop into other pods (L27)
      ▼
 4. copy the token out; reuse it later from anywhere

How each control cuts the path

That last row is your incident lever. A projected token is bound to the pod's UID, so deleting the pod (or a kubectl rollout restart) kills the stolen token, even though EKS would otherwise accept it for up to 90 days. AWS credentials the pod obtained through IRSA or Pod Identity are different: STS credentials stay valid until they expire, so revoke them on the IAM side as well (L24's deep dive mentions the aws:TokenIssueTime deny).

Seeing it in the audit log

Every API call is in the EKS audit log, if it's turned on (L45's deep dive). An application SA asking "what can I do?", or reading Secrets, is a strong signal. A CloudWatch Logs Insights query over the audit log stream:

fields @timestamp, user.username, verb,
       objectRef.resource, sourceIPs.0, userAgent
| filter @logStream like /kube-apiserver-audit/
| filter user.username like /^system:serviceaccount:payments:/
| filter objectRef.resource in
    ["selfsubjectrulesreviews", "secrets", "pods/exec"]
| sort @timestamp desc

Two more clues: a sourceIPs value that isn't a pod IP in your VPC means the token left the cluster, and a userAgent of kubectl for an SA that's normally a Java client is odd. Recent Kubernetes versions also put the pod and node into the token and record a credential ID in audit events, which makes tracing a specific stolen token easier (check what your EKS version supports).

🧠 Extra takeaways

• An SA token is a bearer credential for the API server; whoever holds it is that SA.
• No token (automount off) and no permissions (no RBAC on app SAs) defeat most of the path.
• Delete the pod to invalidate a stolen projected token; revoke AWS sessions separately.
• Watch the audit log for app SAs listing secrets, asking for their rules, or calling from outside the VPC.

The problem

Thirty teams each run an export job in their own namespace on the same EKS cluster. Each may write only to its own prefix in the bucket ex-exports. With IRSA that's thirty roles, thirty policies, and thirty trust policies: L27's role explosion, in IAM.

The tags Pod Identity adds

When EKS Pod Identity assumes a role, it attaches session tags describing the pod. Per the EKS docs, they are eks-cluster-arn, eks-cluster-name, kubernetes-namespace, kubernetes-service-account, kubernetes-pod-name and kubernetes-pod-uid. Policies can read them as ${aws:PrincipalTag/<key>}. EKS sets the values from the association, not from anything the pod says, so a pod can't forge them.

Worked example: one policy, thirty namespaces

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource":
  "arn:aws:s3:::ex-exports/${aws:PrincipalTag/kubernetes-namespace}/*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::ex-exports",
      "Condition": {"StringLike": {"s3:prefix":
        "${aws:PrincipalTag/kubernetes-namespace}/*"}}
    }
  ]
}

One role, export-job, carries this policy. Each team gets an association from its own namespace's SA to that role.

The same trick works with resource tags. For Secrets Manager, tag each secret namespace=claims and allow secretsmanager:GetSecretValue only when secretsmanager:ResourceTag/namespace equals ${aws:PrincipalTag/kubernetes-namespace}, just like L27's team-tag example.

Where this can bite you

• The namespace is the boundary. Anyone who can run a pod with that SA in claims gets the claims tag. Keep namespaces per team (L19's deep dive makes the same point for Secrets).
• Namespace names repeat across clusters. claims in a dev cluster and claims in prod get the same tag value. If one role is shared across clusters, also check eks-cluster-name, or keep separate roles per environment.
• Tags are transitive. Per the EKS docs, they carry into role chaining, so a role in another account can check them too. Useful, but know they're there.
• Harder to audit. "Who can read payments/?" now depends on associations and namespaces, not on one policy (L27's ABAC trade-off). Keep the association list in code review.
• Size limits. If STS returns PackedPolicyTooLarge, EKS lets you disable session tags on an association, which also disables this pattern.

🧠 Extra takeaways

• Pod Identity tags each session with cluster, namespace, SA and pod; IRSA doesn't.
• ${aws:PrincipalTag/kubernetes-namespace} turns one role into per-namespace permissions.
• The tag is trustworthy, but the namespace becomes your security boundary.
• Add the cluster name to conditions when a role is shared across environments.