L48 · Secrets management

Module 7: Cloud, Containers, and Kubernetes · ~20 min · Lesson 48 of 71

🎯 What you'll learn

• Why a Kubernetes Secret is only base64, and what EKS etcd encryption with KMS does and doesn't fix
• Two ways to get a secret from Secrets Manager or Vault into a pod: External Secrets Operator and the Secrets Store CSI driver
• When to pick Vault or AWS Secrets Manager, and how Airflow should find its connections
• How a rotated value reaches a running pod, and why a secret in git history must be rotated, not just deleted

🔥 Why this matters

A pull request adds values-dev.yaml with mongoPassword: "...". A reviewer spots it, and the next commit blanks the line. The file looks clean, so everyone moves on. Months later a pen tester runs one git log command and logs in to MongoDB with it. L19 covered what a good secret store is (KMS, Secrets Manager, rotation labels). This lesson is the plumbing: getting secrets from that store into pods and DAGs without leaving copies everywhere.

1. One source of truth, a few delivery paths

Every secret should have exactly one source of truth, the store where it's created and rotated, and as few copies as possible. In a stack like the one in these examples, the paths look like this:

 SOURCE OF TRUTH      DELIVERY                   CONSUMER
 Secrets Manager      External Secrets Operator
 or Vault ──IRSA────▶ copies it into a ─────────▶ k8s Secret ─▶ pod
  │                   k8s Secret                  (base64, etcd)
  │
  ├──pod's IRSA─────▶ Secrets Store CSI driver ─▶ file in the pod
  │                   mounts it at pod start      (no k8s Secret)
  │
  └──MWAA role──────▶ Airflow secrets backend ──▶ hook in a DAG
                      looks it up at run time

Each arrow authenticates with a workload identity (an IAM role, or for Vault the pod's service account), not a stored password. Each box on the right is a copy you now have to protect.

2. Kubernetes Secrets: base64, etcd and RBAC

A Kubernetes Secret stores values as base64, an encoding with no key: anyone who can read the object can decode it in one command (Try it below). L19's second deep dive drew who can read one: RBAC get/list on secrets, anyone who can create a pod in the namespace, node access, and anyone with etcd or its backups.

Encryption fixes only the last of those. Per AWS's docs, EKS 1.28+ already envelope-encrypts all Kubernetes API data in etcd with an AWS owned KMS key. You can add your own customer managed key, so decryption shows up in your CloudTrail and your key policy decides who can use it:

aws eks describe-cluster --name payments-dev \
  --query 'cluster.encryptionConfig[0].provider.keyArn'
# "arn:aws:kms:us-east-1:111122223333:key/1a2b3c4d-EXAMPLE"
# null = no customer managed key (AWS owned key still used)

It's enabled with aws eks associate-encryption-config, and per AWS's docs it can't be turned off afterwards. If that KMS key is disabled or deleted, the cluster can't read its Secrets. None of this stops a user or pod that RBAC lets in: the API server decrypts for them.

3. Getting secrets into pods: ESO or the CSI driver

External Secrets Operator (ESO) is a controller. You create an ExternalSecret that says "copy prod/recon/mongo from Secrets Manager into a Secret called recon-mongo, every 15 minutes". Pods use it like any Secret. Simple, but the value now also lives in etcd as base64.

The Secrets Store CSI driver (with the AWS provider) mounts the secret as a file when the pod starts, using the pod's own IRSA or Pod Identity role. No Kubernetes Secret is created unless you turn on syncing. The catches: the pod can't start if the store is unreachable, and you get files; env vars need that optional sync.

Rule of thumb: CSI when per-pod identity and "never in etcd" matter (card or customer data); ESO when you need a normal Secret, for example for an ingress TLS cert or a third-party Helm chart.

4. Vault or AWS Secrets Manager?

On an AWS-only stack, Secrets Manager is the sensible default: nothing to run, and access is plain IAM. Vault earns its operating cost with multiple clouds or data centers, dynamic database credentials, or an internal PKI. One licensing note: Vault moved to the Business Source License in 2023; OpenBao is the open-source fork under the Linux Foundation.

5. Airflow: connections, variables and secrets backends

Airflow looks up a connection or variable in a fixed order, and stops at the first hit:

 MongoHook(mongo_conn_id="recon_mongo")
   │ 1. secrets backend:  airflow/connections/recon_mongo
   │    (Secrets Manager, via the MWAA execution role)
   │ 2. env var:          AIRFLOW_CONN_RECON_MONGO
   │ 3. metadata DB:      connections created in the UI
   ▼
 first match wins

On MWAA you set secrets.backend to airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend and secrets.backend_kwargs to prefixes such as {"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables"} (L28 showed the DAG side). Three habits:

• Scope the execution role to secret:airflow/*, not every secret in the account.
• No Variable.get() at the top of a DAG file. That code runs on every parse, roughly every 30 seconds, and each run is a Secrets Manager call. Read variables inside tasks or templates.
• Don't count on masking. Airflow hides connection passwords and variables named like password, secret or api_key in task logs, but not a value you've base64-encoded or split.

6. Rotation and git history

Rotation (L19) changes the value in the store. Whether your pod notices depends on the path:

 t=0    Secrets Manager rotates: AWSCURRENT = v2
 ≤15m   ESO refreshInterval: k8s Secret now holds v2
 +~1m   kubelet updates mounted Secret files (not subPath mounts)
        env vars: still v1 until the pod restarts
 app    re-reads the file on an auth error (L19) and gets v2
 later  v1 retired: anything still holding it fails

The CSI driver re-reads only if rotation is enabled (enableSecretRotation, polling every 2 minutes by default). Env vars never change; that's one more reason to prefer files.

Git history is where deleted secrets live forever. Removing a line adds a commit; the old one still holds the value, and so does every clone, fork, CI cache and pull request ref. Rewriting history with git filter-repo cleans your copy only. GitHub's own guidance: treat the secret as compromised and revoke it first. Scan with gitleaks or trufflehog (L28's second deep dive), and turn on push protection so it doesn't happen again.

 delete the line        ──▶ fixes the latest commit only
 rewrite history        ──▶ fixes your remote, not clones/forks
 rotate the secret      ──▶ fixes every copy, everywhere

🧪 Worked example: recon-loader's password, out of Helm values

Before: the password sits in values-prod.yaml and the chart renders it into a Secret.

mongo:
  user: recon_loader
  password: "Ex4mple-NOT-real"     # in git, in every clone

After: store it once (prod/recon/mongo in Secrets Manager), and let ESO copy it. The SecretStore authenticates with an IRSA service account in the recon namespace:

apiVersion: external-secrets.io/v1   # v1beta1 on older ESO
kind: ExternalSecret
metadata:
  name: recon-mongo
  namespace: recon
spec:
  refreshInterval: 15m
  secretStoreRef: {name: aws-sm, kind: SecretStore}
  target: {name: recon-mongo}       # the Secret ESO writes
  data:
  - secretKey: password
    remoteRef: {key: prod/recon/mongo, property: password}

The IRSA role may read that one secret and nothing else. Secrets Manager adds six random characters to every secret ARN, hence the -* (one string, wrapped to fit):

{
  "Effect": "Allow",
  "Action": ["secretsmanager:GetSecretValue",
             "secretsmanager:DescribeSecret"],
  "Resource": "arn:aws:secretsmanager:us-east-1:111122223333:
               secret:prod/recon/mongo-*"
}

The Deployment mounts recon-mongo as a volume at /var/run/secrets/recon instead of an env var, so a rotation arrives without a restart. Then rotate the old password, because it's still in git.

🛠 Try it (optional, ~5 min)

All local, all fake. First, decode a Secret's values the way an attacker would:

echo 'cmVjb25fbG9hZGVy' | base64 -d; echo
# recon_loader
echo 'RXg0bXBsZS1OT1QtcmVhbA==' | base64 -d; echo
# Ex4mple-NOT-real

Now the classic outage bug. Compare:

echo 'Ex4mple-NOT-real' | base64           # RXg0...cmVhbAo=
printf '%s' 'Ex4mple-NOT-real' | base64    # RXg0...cmVhbA==

echo adds a newline, so the first Secret's password ends in \n and MongoDB says "Authentication failed". (Older macOS base64 used -D to decode.)

Finally, watch a "deleted" secret survive, in a throwaway repo:

cd "$(mktemp -d)" && git init -q
echo 'mongoPassword: "Ex4mple-NOT-real"' > values-dev.yaml
git add . && git commit -qm "add dev values"
echo 'mongoPassword: ""' > values-dev.yaml
git commit -qam "remove password"
grep -c Ex4mple values-dev.yaml       # 0: looks clean
git log -p -S Ex4mple --oneline       # both commits, value shown

-S finds commits that add or remove a string, which is why rotation, not deletion, is the fix.

🧠 Key takeaways

• base64 is not protection; RBAC on secrets and pod creation is.
• EKS etcd encryption with KMS protects etcd, not what the API server hands out.
• ESO copies secrets into Kubernetes; the CSI driver mounts them as files with the pod's own role.
• Secrets Manager is the default on AWS; Vault pays off with dynamic secrets or many platforms.
• Mount secrets as files and re-read on auth failure; env vars never see a rotation.
• A secret that reached git is compromised: rotate first, rewrite history second.

📖 Words to know

⏭ Next time

L49: container security: what's inside an image's layers, why running as root matters, distroless base images, and how scanning and signing images fit into deploys to a cluster such as EKS.

A password that exists for one hour

Every secret so far was static: created once, stored, and copied to whoever needs it, until someone rotates it. A dynamic secret is created on demand for one caller, with an expiry. When recon-loader starts, Vault creates a brand-new MongoDB user just for that pod, valid for an hour. Nobody else ever sees it, and Vault deletes the user when the time runs out.

 recon-loader pod            Vault                      MongoDB
   │ 1. login (k8s auth)       │                           │
   │──────────────────────────▶│                           │
   │ 2. read database/creds/   │ 3. createUser             │
   │    recon-read             │    v-k8s-recon-read-x7Qk  │
   │──────────────────────────▶│──────────────────────────▶│
   │◀── 4. user + password ────│   role: read on "recon"   │
   │    lease: 1h, renewable   │                           │
   │                           │ 5. lease expires:         │
   │                           │    dropUser ─────────────▶│

Leaked credentials now have a short shelf life, every pod has its own DB user (so MongoDB's audit log says which pod did something), and there's no shared password to rotate.

Worked example: MongoDB with the database secrets engine

An admin configures it once (made-up names; the admin password is replaced right away):

vault secrets enable database
URL='mongodb://{{username}}:{{password}}@mongo.example.com'
vault write database/config/recon-mongo \
  plugin_name=mongodb-database-plugin \
  allowed_roles=recon-read \
  connection_url="$URL:27017/admin?tls=true" \
  username=vault-admin password=EXAMPLE-ONLY
vault write -f database/rotate-root/recon-mongo

rotate-root changes vault-admin's password to one only Vault knows, so the bootstrap password in your shell history stops working. Then a role says what each generated user may do:

vault write database/roles/recon-read \
  db_name=recon-mongo \
  creation_statements='{"db":"admin",
    "roles":[{"role":"read","db":"recon"}]}' \
  default_ttl=1h max_ttl=24h

A caller allowed to use the role asks for credentials:

$ vault read database/creds/recon-read
Key                Value
---                -----
lease_id           database/creds/recon-read/2f6a614c-EXAMPLE
lease_duration     1h
lease_renewable    true
password           A1a-EXAMPLE-not-a-real-password
username           v-k8s-recon-read-x7Qk2EXAMPLE

The pod renews the lease while it runs (up to max_ttl) and stops when it's done. The exact username format depends on the plugin's username template. MongoDB Atlas uses a separate plugin, mongodbatlas-database-plugin, which manages users through Atlas's API.

When something leaks

Revoke by prefix, and every user issued from that role is dropped at once:

vault lease revoke -prefix database/creds/recon-read

Compare that with a static password: find every consumer, rotate, redeploy, hope nothing was missed. The trade-off is availability: if Vault is down, new pods can't get database credentials, so Vault itself needs high availability, backups and monitoring.

The AWS-native alternative: no password at all

If you run MongoDB on Atlas, there's another option: AWS IAM authentication (authMechanism=MONGODB-AWS). The pod signs a request with its IRSA role credentials and Atlas maps that role to a database user. There's no database password to store, rotate or leak. Per MongoDB's driver docs, this mechanism is for Atlas clusters. On AWS-only stacks, it's often a simpler path than running Vault.

🧠 Extra takeaways

• Dynamic secrets are made per caller, expire on their own and need no shared rotation.
• rotate-root makes the bootstrap admin password useless; do it right after setup.
• Revoking a lease prefix kills every credential from a role in one command.
• For MongoDB Atlas on AWS, IAM authentication removes the password altogether.

The convenient setup

A platform team installs ESO once for a shared EKS cluster. To keep it simple, they create one ClusterSecretStore, a store any namespace can reference, backed by an IAM role allowed to read secretsmanager:GetSecretValue on *. Every team just writes ExternalSecrets. It works on day one.

Now look at it from the marketing-batch namespace:

 dev in marketing-batch (may create ExternalSecrets)
   │ 1. ExternalSecret: key = prod/payments/ledger-db,
   │    store = ClusterSecretStore "aws-all"
   ▼
 ESO controller (role: GetSecretValue on *)
   │ 2. "ESO is allowed, so I fetch it"
   ▼
 Secret "ledger-db" in marketing-batch
     3. dev reads it with kubectl: payments' DB password

ESO is a confused deputy (L46's first deep dive): a trusted service using its broad access on behalf of whoever asks. Being allowed to create an ExternalSecret has become being allowed to read every secret the controller's role can reach.

Fix 1: one store per namespace, with that namespace's identity

Use a namespaced SecretStore that authenticates with a service account in that namespace, whose IRSA role reads only that team's prefix:

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: aws-sm
  namespace: payments
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: payments-secrets   # IRSA: payments/* only

Now a marketing-batch ExternalSecret can only use marketing's store, and AWS refuses anything outside marketing's prefix, whatever ESO is asked to do.

Fix 2: if you keep a ClusterSecretStore, fence it

A ClusterSecretStore can list which namespaces may use it in spec.conditions, by name or label selector (newer releases also accept regexes). Pair that with an IAM policy that matches on secret names or tags, for example secretsmanager:ResourceTag/team equal to the caller's team, so the role itself can't reach other teams' secrets. Check your ESO version's docs for the exact fields.

Fix 3: control who can write ExternalSecrets

RBAC create on externalsecrets is now a secrets permission. Treat it like get secrets: grant it to the deploy pipeline's service account, not to every developer. Also watch PushSecret, an ESO resource that writes to the external store; most clusters don't need it, and it's safest to deny it by default.

Worked example: auditing a shared cluster (read-only, a cluster you're allowed to inspect)

kubectl get clustersecretstores \
  -o custom-columns=NAME:.metadata.name,COND:.spec.conditions
# NAME      COND
# aws-all   <none>          <- any namespace can use it
kubectl get externalsecrets -A -o json | jq -r '.items[]
  | "\(.metadata.namespace)  \(.spec.data[]?.remoteRef.key)"'
# payments         prod/payments/ledger-db
# marketing-batch  prod/payments/ledger-db   <- wrong team

Any key outside the namespace's own prefix deserves a question, and a ClusterSecretStore with no conditions and a * role is the finding to fix first.

How the CSI driver compares

The CSI driver authenticates as the pod's own service account, so each workload's IRSA role already limits what it can mount. Its risk is the same as for any pod: whoever can create pods with a service account can use that account's role. Either way the rule is the same: the identity that fetches a secret should belong to the team that owns it.

🧠 Extra takeaways

• A ClusterSecretStore with a broad role turns "create ExternalSecret" into "read any secret".
• Prefer namespaced SecretStores, each using its own namespace's IRSA role and prefix.
• Fence shared stores with conditions and tag- or name-scoped IAM policies.
• Treat RBAC on externalsecrets and pushsecrets as secrets permissions.