L19 · Key Management and Secrets

Module 3: Cryptography Essentials · ~20 min · Lesson 19 of 71

🎯 What you'll learn

• Why a strong cipher is worthless if the key sits next to the data
• How AWS KMS, HSMs and envelope encryption keep keys out of your hands, and what rotation rotates
• What "encrypted at rest" and "encrypted in transit" protect against, and what they don't
• Where app secrets belong, and why the best secret is one you never create

🔥 Why this matters

In 2016, attackers got into a private GitHub repository used by Uber engineers and found an AWS access key written into the code. With it, they downloaded data on about 57 million riders and drivers from S3. Strong encryption was available the whole time; it didn't matter, because the key that opened everything sat in a repo. L14 to L18 were about algorithms. This lesson is about what attackers actually go after: keys and secrets.

1. The key is the secret

Picture a vault with a perfect door and the combination taped beside it. Kerckhoffs's principle says a system should stay secure even if everything except the key is public. AES-256-GCM (L15) is public; your key isn't. So key management, handling a key safely through its whole life, is the real work:

 1 generate ──▶ 2 store ──▶ 3 use ──▶ 4 rotate ──▶ 5 destroy
 (strong        (never       (only by    (new key     (only when no
  randomness,    beside the   allowed     for new      data needs
  L20)           data)        callers)    data)        it, L01)

Most real failures happen at step 2: keys in application.yml, a ConfigMap, an image layer, a CI log, or a chat message.

2. KMS and HSMs: keys you can use but never see

A hardware security module (HSM) is tamper-resistant hardware that generates and stores keys and does the crypto inside itself, so key material never leaves in plaintext. Think of a notary who stamps your documents but never lends you the stamp: you send data in and get only the result back.

AWS KMS is a managed service built on HSMs that AWS says are validated under FIPS 140-3, Security Level 3 (the US standard for crypto modules; Level 3 adds tamper resistance). A KMS key is the logical key you create and call by ARN or alias through APIs like Encrypt and Decrypt. You can never download it. Who may call those APIs is set by the key policy (every KMS key has one, and IAM policies only count if it allows them), IAM policies (L46), and grants, narrow delegated permissions that AWS services such as EBS use on your behalf. Every call is logged in CloudTrail.

CloudHSM is the alternative: a single-tenant HSM cluster where you, not AWS, manage the keys and HSM users. It's for specific compliance or protocol needs; for services on EKS, KMS is the default.

3. Envelope encryption

KMS Encrypt takes at most 4 KB, and calling KMS for every MongoDB document would be slow. So almost everything uses envelope encryption: encrypt the data with a fresh data key (DEK), then encrypt the data key with the KMS key, which acts as the key encryption key (KEK).

 policy-svc                    AWS KMS (HSMs)           storage
     │ 1 GenerateDataKey(          │                        │
     │   key=alias/policy-data,    │                        │
     │   context={policyId:..})    │                        │
     │────────────────────────────▶│                        │
     │◀── 2 plaintext data key ────│                        │
     │      + encrypted data key   │                        │
     │                             │                        │
     │ 3 AES-GCM encrypt record locally with plaintext key  │
     │ 4 forget the plaintext key                           │
     │ 5 store ciphertext + encrypted data key ────────────▶│
     │                                                      │
     │ later: 6 Decrypt(encrypted data key, same context)   │
     │────────────────────────────▶│                        │
     │◀── 7 plaintext data key ────│  (logged in CloudTrail)│

You store two locked boxes: the data, locked with the data key, and the data key, locked with the KMS key, which never leaves KMS. Reading the data needs kms:Decrypt permission. S3, EBS and EKS all do this under the hood.

The encryption context (step 1) is a set of non-secret key-value pairs that KMS binds to the ciphertext as additional authenticated data (AAD), the AEAD feature from L01. Decrypt fails unless it gets the same context. That's an integrity control: nobody can move policy PL-100246's encrypted data key onto another record. It hides nothing, and it appears in plaintext in CloudTrail.

4. Rotation: what actually changes

With automatic rotation on a customer managed key, KMS creates new key material every 365 days by default (configurable from 90 to 2,560), and you can trigger on-demand rotation up to 25 times per key. The key ID, ARN and policy stay the same, and KMS keeps old material so old ciphertext still decrypts. No code changes.

Per AWS's docs, rotation does not re-encrypt existing data or rotate data keys already issued. If a data key leaks, re-encrypt that data under a new one. Asymmetric and HMAC keys need manual rotation: create a new key and move the alias. And as L01's deep dive showed, deleting a KMS key makes its data unreadable forever, so KMS enforces a 7 to 30 day waiting period. Alert on ScheduleKeyDeletion and DisableKey.

5. At rest, in transit, in use

"We encrypt everything" means less than it sounds:

  in transit          at rest                 in use
 ─────────────▶   ═══════════════════    ┌───────────────┐
 Kong ──TLS──▶    EBS / S3 / Atlas disk   │ app memory:   │
 policy-svc       (KMS-backed volume      │ PLAINTEXT     │
 ──TLS──▶ Mongo    or object keys)        │ while working │
                                          └───────────────┘
 stops: sniffing  stops: stolen disk,     stops: almost
 and tampering    copied snapshot,        nothing, unless
 on the wire      backup in the wrong     the app encrypts
 (L10)            hands without key       fields itself
                  access

Encryption at rest on EBS, S3 (SSE-S3 is the default for new objects) or a MongoDB Atlas cluster protects against someone getting the storage without the key. It does nothing against someone who gets the application: abuse the service's role or DB user and the storage decrypts automatically. That's why L05's Q3 answer wasn't "encrypt at rest". Encryption in transit (TLS, L10) protects each hop. For data in use, the practical tool is field-level encryption: the app encrypts taxId before it reaches MongoDB, so a stolen dump shows ciphertext (L53).

6. Secrets: store them well, or don't have them

A secret is any value that grants access: a DB password, an API key, a webhook HMAC key (L17). AWS has two managed stores, both encrypting with KMS:

Better still is no static secret. IRSA and EKS Pod Identity give a pod short-lived AWS credentials tied to its service account (L50). GitHub Actions OIDC does the same for CI (L57), shown next. Kubernetes Secrets are only base64 (second deep dive); L48 covers vaults in depth.

🧪 Worked example: the access key in the repo

A developer commits this to policy-service (AWS's documented fake example key):

# deploy/config/dev.yml
aws:
  accessKeyId: AKIAIOSFODNN7EXAMPLE
  secretAccessKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Detection. With secret scanning and push protection on, GitHub blocks the push and names the line. For keys pushed to public repos, GitHub also tells AWS, which attaches a quarantine policy (AWSCompromisedKeyQuarantineV3) to the IAM user. That limits damage; it doesn't revoke the key.

Response. Treat the key as stolen, even in a private repo: deactivate it, check CloudTrail for everything it called since the commit, then delete it. Rewriting git history doesn't un-leak it.

The fix: no key at all. Register GitHub's OIDC provider in IAM and give the deploy role this trust policy:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Federated": "<GITHUB_OIDC_PROVIDER_ARN>" },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "token.actions.githubusercontent.com:aud":
          "sts.amazonaws.com",
        "token.actions.githubusercontent.com:sub":
          "repo:example-org/policy-service:environment:prod"
      }
    }
  }]
}

The provider ARN looks like arn:aws:iam::111122223333:oidc-provider/token.actions.githubusercontent.com. In the workflow, the job gets permissions: id-token: write, declares environment: prod, and calls aws-actions/configure-aws-credentials with role-to-assume. Each run gets credentials that expire in about an hour.

The sub condition matters most. Without it, any GitHub repo could assume your role; with StringLike on repo:example-org/*, every repo in the org could deploy to prod. One more check: per GitHub's docs, repos created after July 15, 2026 use an immutable sub with numeric IDs (repo:example-org@123456/policy-service@456789:...), so read a real token's claims first.

🛠 Try it (optional, ~5 min)

Build envelope encryption locally. An AES key plays the KMS key, so nothing touches AWS. You need Python 3 and cryptography (pip3 install cryptography if the import fails). Save as envelope_demo.py in a scratch folder:

import base64, os
from cryptography.hazmat.primitives.ciphers.aead import AESGCM

# 1. The "KMS key". In real KMS it never leaves the HSM.
kms = AESGCM(AESGCM.generate_key(bit_length=256))

def generate_data_key(context):
    """Like GenerateDataKey: plaintext + wrapped copy."""
    data_key = AESGCM.generate_key(bit_length=256)
    n = os.urandom(12)
    return data_key, n + kms.encrypt(n, data_key, context)

def decrypt_data_key(wrapped, context):
    """Like Decrypt: needs the same encryption context."""
    return kms.decrypt(wrapped[:12], wrapped[12:], context)

ctx = b"service=policy-svc;policyId=PL-100246"
record = b'{"policyId":"PL-100246","taxId":"000-00-0000"}'

# 2. Encrypt locally with a fresh data key, then drop it.
data_key, wrapped = generate_data_key(ctx)
nonce = os.urandom(12)
ct = AESGCM(data_key).encrypt(nonce, record, None)
del data_key
print("stored key:", base64.b64encode(wrapped).decode())
print("stored data:", base64.b64encode(ct).decode())

# 3. Read it back: unwrap the data key, then decrypt.
key = decrypt_data_key(wrapped, ctx)
print("decrypted:", AESGCM(key).decrypt(nonce, ct, None).decode())

# 4. Same wrapped key, wrong context: refused.
try:
    decrypt_data_key(wrapped, b"service=reporting-job")
except Exception as e:
    print("wrong context:", type(e).__name__)

Run python3 envelope_demo.py. Sample output on macOS (trimmed; yours differs every run):

stored key: V4vokDS4IFWnQUTKwVF0...kR7WX5U
stored data: 8Wr/0uP+IS2sqpWHLV+r...ous4TI=
decrypted: {"policyId":"PL-100246","taxId":"000-00-0000"}
wrong context: InvalidTag

Notice: what you store holds only a wrapped key, and the wrong context fails with InvalidTag, the same AEAD check as L01. Also, del only drops a reference; Python can't reliably wipe memory, one reason real code uses the AWS Encryption SDK.

Optional, in your own sandbox AWS account only (read-only; not run for this lesson): aws kms list-aliases, then aws kms get-key-rotation-status --key-id <your-key-id>.

🧠 Key takeaways

• The algorithm is public; the key is the secret. Most failures are keys stored in the wrong place.
• KMS keeps keys inside HSMs; you get permission to use them, and every use is logged.
• Envelope encryption: data keys encrypt data, the KMS key encrypts data keys, and the encryption context binds them (integrity, not secrecy).
• KMS rotation adds new material under the same key ID; it doesn't re-encrypt data or data keys.
• Encryption at rest protects the storage, not the app. Anyone your app lets in sees plaintext.
• Prefer no static secret: IRSA or Pod Identity for pods, OIDC for GitHub Actions.

📖 Words to know

⏭ Next time

L20: crypto mistakes in the wild: homemade crypto, hard-coded keys, Math.random(), nonce reuse, and how Adobe's 2013 leak of about 150 million "encrypted" passwords became a giant crossword puzzle.

Why rotation breaks things

Teams often avoid rotating secrets because the last time they did, something fell over. The usual story: someone changes the MongoDB password for policy-svc, updates the secret, and for the next ten minutes every pod still holding the old password gets Authentication failed. Some pods cached it at startup and never reload it. A nightly Airflow job fails at 2 a.m. because its connection still has the old value.

The root cause is a hard cutover: there's one valid secret, and you swap it. The fix is an overlap window, where both the old and the new secret work at the same time:

 time ─────────────────────────────────────────────────────▶
 old secret  ███████████████████████████████░░░░░░░ revoked
 new secret              ░░░░░░████████████████████████████
                         │     │                 │
                     1 create  2 consumers     3 retire old
                       new       switch over     once nobody
                       (valid    (any order,     uses it
                       but not   at their own    (check logs)
                       in use)   pace)

Every safe rotation, whatever the secret, follows these three steps.

Secrets Manager versions and staging labels

Secrets Manager builds the overlap in. A secret holds several versions, and staging labels mark which is which: AWSCURRENT is what callers get by default, AWSPREVIOUS is the one before it, and AWSPENDING is the new one during a rotation. A rotation Lambda runs four steps:

Secrets Manager offers two strategies for databases. Single user changes one user's password; it's simple, but AWS's docs note a short window where the database may reject the old password before callers pick up the new one, so you need retries. Alternating users keeps two DB users (policy_svc and policy_svc_clone) and changes the password of the one not in use. Callers on the old version keep working until they refresh, which is the overlap window from the diagram.

🧪 Worked example: making policy-service rotation-safe

The secret side is half the job. The app has to cooperate:

1. Fetch, don't bake. Read the secret from Secrets Manager (or a mounted file) at runtime, with a short cache (AWS publishes caching clients for Java and Python). Never bake it into the image or an env var set once at deploy.
2. Refresh on failure. On an authentication error, invalidate the cache, fetch AWSCURRENT again, reconnect once, then fail.

try {
    return repo.find(policyId);
} catch (MongoSecurityException e) {
    secrets.invalidate("prod/policy-svc/mongo");
    mongo.reconnect(secrets.get("prod/policy-svc/mongo"));
    return repo.find(policyId);        // one retry only
}

3. Know your consumers. List everything using the secret: pods, Airflow connections, a reporting job. An Airflow connection stored as a plain value in its metadata DB won't follow a rotation; point Airflow at a secrets backend (it supports AWS Secrets Manager) instead.
4. Watch before you retire. With alternating users, check the DB logs that the old user has gone quiet before the next rotation changes its password again.

The same pattern elsewhere

• API keys at Kong: issue a second key to the consumer, let them switch, then delete the first. Kong lets one consumer hold several credentials at once, which is exactly the overlap.
• JWT signing keys: publish the new public key in the JWKS with a new kid before signing with it, and keep the old one until every token it signed has expired (L25).
• TLS certificates: deploy the new cert before the old one expires (L11's second deep dive).
• KMS keys: already overlap-safe, because old key material is kept for decryption. A leaked data key is the exception: re-encrypt the data under a new one.

🛠 Try it (~5 min)

Pick one secret your service uses. Write down: where it's stored, every consumer, how each one picks up a new value (restart? cache expiry? never?), and how you'd know the old value is unused. Any "never" is your next outage.

🧠 Extra takeaways

• Rotation fails at hard cutovers; design for two valid secrets at once.
• Secrets Manager's staging labels (AWSPENDING, AWSCURRENT, AWSPREVIOUS) give you the overlap.
• Apps must fetch at runtime and refresh on auth failure, or rotation becomes an outage.
• The same create, switch, retire pattern works for API keys, JWT keys and certificates.

base64 is not encryption

Here's a Secret someone pasted into a pull request:

apiVersion: v1
kind: Secret
metadata:
  name: policy-svc-mongo
type: Opaque
data:
  username: cG9saWN5X3N2Yw==
  password: czNjcjN0LUV4YW1wbGU=

It looks scrambled. It isn't:

$ echo 'czNjcjN0LUV4YW1wbGU=' | base64 -d
s3cr3t-Example

base64 is an encoding: a reversible way to write bytes as text so they fit in YAML. There's no key. The Kubernetes docs say it plainly: Secrets are stored unencrypted in etcd by default, anyone with API access can read them, and anyone who can create a Pod in a namespace can read every Secret in it (even indirectly, by creating a Deployment). (On macOS, base64 -d works on current versions; older ones used -D.)

Who can see a Secret

                 ┌──────────────────────────────┐
 kubectl get ───▶│ API server  (RBAC decides)   │◀── create Pod
 secret          └──────────────┬───────────────┘    that mounts it
                                │ store / load
                 ┌──────────────▼───────────────┐
                 │ etcd   (encrypted at rest?)  │◀── etcd backup
                 └──────────────────────────────┘
 node running the pod: Secret in tmpfs / env vars ◀── node access,
                                                      kubectl exec

There are four ways in: RBAC permission to get or list secrets, permission to create pods, access to etcd or its backups, and access to a node or a running container.

What EKS encrypts for you

Per AWS's EKS docs, clusters on Kubernetes 1.28 or later get default envelope encryption of all Kubernetes API data in etcd, Secrets included, using KMS v2 with an AWS owned key. You can bring your own customer managed KMS key instead, which lets you control and audit it in CloudTrail. It's the envelope pattern from the main lesson: the API server uses data keys, wrapped by a KMS key.

But look at the diagram again. That only protects the etcd box. Anyone who passes RBAC gets the decrypted Secret from the API server, as AWS's own FAQ notes for backup tools. And a customer managed key brings a new risk: AWS's docs warn that disabling or deleting it can leave the cluster degraded or unrecoverable. Same lesson as L01's deleted KMS key.

What actually protects them

1. RBAC: almost nobody needs get or list on secrets. Check what a service account or user can do:

   $ kubectl auth can-i list secrets -n policy \
       --as=system:serviceaccount:policy:policy-svc
   no
   

2. Namespaces as boundaries: anyone who can create pods in a namespace can read its Secrets, so don't share namespaces between teams with different access.

3. Keep the real secret outside the cluster. Two common tools:

   	External Secrets Operator	Secrets Store CSI Driver (AWS provider)
   How	Syncs from Secrets Manager into a k8s Secret	Mounts the secret as a file into the pod
   Creates a k8s Secret?	Yes	No, unless you turn on syncing
   Rotation	Re-syncs on an interval	Optional rotation polling; app must re-read the file

   Both authenticate to AWS with IRSA or Pod Identity, so the cluster holds no AWS keys.

4. Prefer files to env vars. Env vars leak into crash dumps, debug endpoints and child processes, and they never change without a restart.

5. Skip the secret entirely for AWS access: IRSA or Pod Identity, as in the main lesson.

🛠 Try it (~3 min, local only)

Decode the made-up Secret above with base64 -d on both values. Then, in a dev namespace you're allowed to use, run kubectl auth can-i list secrets for yourself. If the answer is yes, ask whether you really need it.

🧠 Extra takeaways

• base64 is encoding, not encryption; anyone who can read the object can read the secret.
• EKS 1.28+ envelope-encrypts Secrets in etcd by default, but that doesn't stop anyone RBAC lets in.
• Create-pod permission in a namespace means read access to its Secrets.
• Keep the source of truth in Secrets Manager; deliver it with ESO or the CSI driver; prefer files.