L53 · Database security

Module 7: Cloud, Containers, and Kubernetes · ~20 min · Lesson 53 of 71

🎯 What you'll learn

• The four layers that guard a MongoDB database (network, authentication, authorization, data) and the key setting in each
• How SCRAM, x.509 and AWS IAM authentication differ, and when to use each
• What TLS, encryption at rest, Client-Side Field Level Encryption and Queryable Encryption each hide, and from whom
• How to give an Airflow DAG a read-only custom role, and what auditing can tell you afterwards

🔥 Why this matters

In late December 2016, an attacker using the name Harak1r1 started connecting to MongoDB servers that anyone on the internet could reach without a password. They copied or deleted the data and left a note in a new collection asking for 0.2 bitcoin. Copycats joined within days, and by mid-January 2017 trackers counted roughly 27,000 to 32,000 wiped databases (counts differ by source). Victims who paid often got nothing back. In 2020 one script wiped about 22,900 more. No exploit was needed.

1. Four layers, and the ransom victims failed two at once

Think of a bank vault: a locked door, a guard checking badges, badges that open only some boxes, and cash in sealed bags. A database has the same layers:

 app pod / Airflow DAG / attacker
        │ 1. "can I reach port 27017?"
 ┌──────▼──────────────────────────────────────────────┐
 │ NETWORK    bindIp, security group, Atlas IP access  │
 │            list, private endpoint (PrivateLink)     │
 ├─────────────────────────────────────────────────────┤
 │ AUTHN      who are you? SCRAM, x.509, MONGODB-AWS   │
 ├─────────────────────────────────────────────────────┤
 │ AUTHZ      what may you do? roles, custom roles     │
 ├─────────────────────────────────────────────────────┤
 │ DATA       what if 1-3 fail? TLS, at rest, CSFLE/QE │
 └─────────────────────────────────────────────────────┘
   across all layers: AUDIT (who did what, when)

The 2017 victims were reachable from the internet (layer 1) and had no authentication (layer 2). Self-managed MongoDB still ships with access control off by default (L45): until you set security.authorization: enabled and create users, anyone who can connect is effectively root.

2. Network: who can even reach the port

net.bindIp is the self-managed setting for which network interfaces mongod listens on. Since MongoDB 3.6 it defaults to localhost only; older installs often listened on every interface. The classic mistake is bindIp: 0.0.0.0 "to make Docker work", plus a security group open to the world. Bind to the interfaces you need, and let a security group allow 27017 only from the app's node or pod security group (EKS in the examples; L12).

On Atlas the network starts closed. The IP access list is Atlas's allow-list of source addresses, and 0.0.0.0/0 means "the whole internet" (a password is then your only door). Better is a private endpoint: an interface endpoint in your VPC that reaches Atlas over AWS PrivateLink, so traffic never touches the internet. It needs a dedicated (M10+) cluster.

  BEFORE                          AFTER
  EKS pods (10.20.0.0/16)         EKS pods (10.20.0.0/16)
     │ via NAT gateway               │ private IP
     ▼                               ▼
  public internet                 VPC interface endpoint
     │                               │ AWS PrivateLink
     ▼                               ▼
  Atlas cluster                   Atlas cluster
  access list: 0.0.0.0/0 (!)      access list: empty, so
  anyone can try passwords        no public path at all

Private-endpoint connections don't need access-list entries. Atlas resource policies can also forbid 0.0.0.0/0 organization-wide (check the current docs).

3. Authentication: proving who's connecting

MongoDB supports several mechanisms. The three you'll meet:

SCRAM (Salted Challenge Response Authentication Mechanism) is the default since 4.0: the password never crosses the wire and the server stores a salted hash (L18). It's still a password, so it needs a secret store and rotation (L48).

MONGODB-AWS removes the password. The driver signs an sts:GetCallerIdentity request with the pod's temporary role credentials (L50), Atlas asks AWS who signed it, and the database user is the IAM role ARN. A connection string looks like:

mongodb+srv://ledger-prod.abcde.mongodb.example.net/
  ?authSource=%24external&authMechanism=MONGODB-AWS

4. Authorization: least privilege in the database

MongoDB uses RBAC (L27). Built-in roles such as read, readWrite, dbAdmin, readAnyDatabase and root are convenient and usually far too broad. A custom role lists exact privileges: an action (find, insert, update, remove...) on a resource (a database, or one collection).

Three habits:

1. One database user per workload, so a leak is contained and the audit log names the culprit.
2. Scope to the collections and actions needed. A reporting job needs find, not readWrite.
3. Scope to one cluster. Atlas users can be restricted to specific clusters, so a dev user can't touch prod in the same project.

On Atlas you can't run db.createRole() from mongosh; roles and users are managed through the Atlas UI, Admin API, CLI or Terraform. (L31's deep dive showed the self-managed shell version.)

5. Encryption: in transit, at rest, and in use

In transit: Atlas always requires TLS. Self-managed, set net.tls.mode: requireTLS and never ship tlsAllowInvalidCertificates=true (L10).

At rest: Atlas encrypts storage by default; with customer key management (M10+) your own AWS KMS key wraps the database's keys, so you can audit and revoke them (L19). Self-managed, use EBS encryption or Enterprise's encrypted storage engine. This protects stolen disks and snapshots, not data from anyone who logs in: the database decrypts for every authorized query.

In use: Client-Side Field Level Encryption (CSFLE), added in 4.2, encrypts chosen fields in the driver before they leave your service. The server, its logs, backups and a DBA with root see only ciphertext. Keys follow L19's envelope pattern: a KMS key wraps data encryption keys kept in a key vault collection. CSFLE offers two algorithms:

• Random: equal values give different ciphertexts. Safest, but you can't query the field.
• Deterministic: equal values give equal ciphertexts, so equality queries work, but it leaks which records share a value and how often (L15's deep dive).

Queryable Encryption (QE) keeps every ciphertext randomized and stores encrypted index data alongside, so the server can match queries it can't read. Equality queries went GA in 7.0 and range queries in 8.0; prefix, suffix and substring are a public preview in 8.2, not for production. QE is set when a collection is created, adds storage overhead and supports only those query types. Automatic encryption for either needs Atlas or Enterprise; Community supports explicit encryption.

 taxId "123-45-6789"   driver   wire   mongod    disk and
                                       memory    backups
 TLS only              plain    enc    PLAIN     PLAIN
 + at rest             plain    enc    PLAIN     enc
 + CSFLE or QE         plain    enc    enc       enc
                       └ only the app with KMS access sees it

6. Auditing: evidence for later

Auditing records who authenticated, who changed users and roles, and (if you ask) who ran which command on which collection. Atlas has database auditing (M10+); self-managed, it's an Enterprise feature; Community has none. Ship it to your SIEM (L62); the deep dive shows what to capture.

🧪 Worked example: a read-only user for an Airflow DAG

The positions_daily_report DAG reads ledger.positions and ledger.trades and writes a CSV to S3. Today it uses svc_airflow, a shared SCRAM user with readWriteAnyDatabase that four DAGs and one script use. If that password leaks, an attacker can rewrite trades anywhere.

After, in Terraform (made-up project and account IDs):

resource "mongodbatlas_custom_db_role" "report_reader" {
  project_id = var.project_id
  role_name  = "positionsReportReader"
  actions {
    action = "FIND"
    resources {
      database_name   = "ledger"
      collection_name = "positions"
    }
    resources {
      database_name   = "ledger"
      collection_name = "trades"
    }
  }
}

resource "mongodbatlas_database_user" "report_dag" {
  project_id         = var.project_id
  auth_database_name = "$external"
  aws_iam_type       = "ROLE"
  username = "arn:aws:iam::111122223333:role/af-positions-report"
  roles {
    role_name     = "positionsReportReader"
    database_name = "admin"           # custom roles live in admin
  }
  scopes {
    name = "ledger-prod"              # this cluster only
    type = "CLUSTER"
  }
}

No password, find on two collections, one cluster, and a user name that tells the audit log which DAG acted. If the DAG tries to write:

MongoServerError: not authorized on ledger to execute command
  { insert: "trades", ... }    code: 13 (Unauthorized)

On MWAA one execution role is shared by every DAG in the environment, so a per-DAG identity needs separate environments or a task that assumes a per-DAG role.

🛠 Try it (optional, ~5 min)

See "no auth by default" on your own laptop, bound to localhost only. Needs Docker.

docker run -d --name l53 -p 127.0.0.1:27017:27017 mongo:8.0
sleep 5
docker exec l53 mongosh --quiet --eval \
  'db.adminCommand({listDatabases:1}).databases.map(d=>d.name)'
# [ 'admin', 'config', 'local' ]    <- no credentials needed
docker rm -f l53

docker run -d --name l53 -p 127.0.0.1:27017:27017 \
  -e MONGO_INITDB_ROOT_USERNAME=admin \
  -e MONGO_INITDB_ROOT_PASSWORD=Ex4mple-NOT-real mongo:8.0
sleep 5
docker exec l53 mongosh --quiet --eval \
  'db.adminCommand({listDatabases:1})'
# MongoServerError: Command listDatabases requires authentication
docker rm -f l53

Notice 127.0.0.1: in -p. The image listens on all interfaces inside the container, so the mapping is your only network control. Plain -p 27017:27017 publishes it on every interface.

🧠 Key takeaways

• Four layers: network, authentication, authorization, data. The ransom waves needed only the first two to fail.
• Keep databases off the internet: tight bindIp and security groups, or an Atlas private endpoint with an empty IP access list.
• Prefer MONGODB-AWS or x.509 to passwords; give every workload its own user with a custom role scoped to collections, actions and cluster.
• Encryption at rest protects disks, not data from logged-in users; CSFLE and QE hide fields even from the server.
• Turn on auditing before you need it.

📖 Words to know

⏭ Next time

L54 starts Module 8: what SAST, DAST, dependency and IaC scanners each find in your pipeline, what they all miss, and how to handle their false positives without ignoring real bugs.

A client of a wealth-management platform closes their account and asks for their personal data to be erased. The team deletes their documents. But they're also in 35 days of Atlas snapshots, last quarter's S3 export from an Airflow DAG, and a restored copy in a QA environment. You can't edit backups.

The idea

Crypto-shredding means encrypting each customer's sensitive fields with a key used only for that customer, then erasing the data by destroying the key. Every copy of the ciphertext, wherever it is, becomes unreadable at once.

 AWS KMS key (CMK)
     │ wraps
     ▼
 key vault collection (encryption.__keyVault)
  ├─ DEK  keyAltName "cust-C-10442" ──▶ fields of C-10442
  ├─ DEK  keyAltName "cust-C-10443" ──▶ fields of C-10443
  └─ ...
                 delete DEK C-10442
                         │
                         ▼
 live data, snapshots, S3 exports: C-10442's fields are
 ciphertext that no remaining key can open

CSFLE's explicit encryption lets you choose the data key per value, so one key per customer is possible. Automatic encryption uses one key per field from its schema, so it doesn't fit this pattern.

Worked example: per-customer keys with PyMongo

A sketch with PyMongo's ClientEncryption (4.x; made-up names, and the KMS key ARN is fake):

from pymongo.encryption import ClientEncryption, Algorithm

ce = ClientEncryption(kms_providers, "encryption.__keyVault",
                      client, codec_options)
RANDOM = Algorithm.AEAD_AES_256_CBC_HMAC_SHA_512_Random
CMK = {"region": "us-east-1",
       "key": "arn:aws:kms:us-east-1:111122223333:key/ex-1"}

def key_for(cid):
    name = f"cust-{cid}"
    if ce.get_key_by_alt_name(name) is None:
        ce.create_data_key("aws", master_key=CMK,
                           key_alt_names=[name])
    return name

# write: the server only ever sees ciphertext
doc = {"_id": "C-10442",
       "taxId": ce.encrypt("123-45-6789", RANDOM,
                           key_alt_name=key_for("C-10442"))}

# erase: one delete, every copy of taxId goes dark
k = ce.get_key_by_alt_name("cust-C-10442")
ce.delete_key(k["_id"])

After delete_key, decrypting that taxId, from the live collection or a two-week-old snapshot, fails because the key can't be found. Drivers cache decrypted keys briefly (about a minute by default in current drivers), so the effect isn't instant inside a running process.

The catches

• The key vault is backed up too. If snapshots include encryption.__keyVault, an old snapshot holds the deleted key. Keep the key vault in its own cluster with short backup retention, or accept that erasure completes when the oldest backup containing the key expires.
• Random algorithm only, ideally. Deterministic ciphertexts can't be read either, but they still show which records shared a value.
• Legal acceptance varies. Whether destroying keys counts as "erasure" is a legal judgment, and financial record-keeping rules may require you to keep some data for years. Get compliance sign-off on the design before building it.
• Lose a key, lose the data. Deleting the wrong DEK is an irreversible outage for that customer. Protect key deletion like a production delete: approvals, logging and a delay.

🧠 Extra takeaways

• One data key per customer, plus explicit encryption, lets you erase a customer by deleting a key.
• It reaches backups and exports you can't edit, but not old backups of the key vault itself.
• Use the random algorithm, and guard key deletion with approvals and logs.
• Agree the design with legal and compliance; retention rules may override erasure for some fields.

Auditing is easy to switch on and easy to get wrong: too little and the log can't answer "who read the customers collection on Tuesday?"; too much and it hurts performance.

What gets recorded

Each audit event has an atype (action type). The useful groups:

Reads and writes (find, insert, update) appear only as authCheck events. Logging every successful one requires auditAuthorizationSuccess: true, which MongoDB's docs warn degrades performance more than logging failures only. So filter narrowly: successful reads of sensitive collections by humans, all failures by everyone.

On a self-managed Enterprise server the configuration looks like this (Atlas takes the same JSON filter in its Database Auditing settings):

auditLog:
  destination: file
  format: JSON
  path: /var/log/mongodb/audit.json
  filter: '{ "$or": [
    { "atype": { "$ne": "authCheck" } },
    { "atype": "authCheck", "result": { "$ne": 0 } },
    { "atype": "authCheck",
      "param.ns": { "$in": ["ledger.customers",
                            "ledger.accounts"] } } ] }'
setParameter:
  auditAuthorizationSuccess: true

Worked example: a DAG that wandered

A failed authCheck (made-up values, trimmed):

{ "atype": "authCheck",
  "ts": { "$date": "2026-11-06T02:14:09.511Z" },
  "remote": { "ip": "10.20.7.88", "port": 51544 },
  "users": [{ "user":
    "arn:aws:iam::111122223333:role/af-positions-report",
    "db": "$external" }],
  "param": { "command": "find", "ns": "ledger.customers" },
  "result": 13 }

result: 13 is "Unauthorized". The positions DAG has no reason to read customers: either a code change wandered, or someone is using its credentials to explore. Either way, look. To summarize a day of failures (on an audit log downloaded from a dev cluster you have access to):

jq -r 'select(.atype=="authCheck" and .result!=0)
  | [.users[0].user, .param.command, .param.ns] | @tsv' \
  audit.json | sort | uniq -c | sort -rn | head
#  212  arn:aws:iam::111122223333:role/af-positions-report
#       find  ledger.customers
#    3  jdoe  insert  ledger.trades

What auditing can't tell you

The audit log says who ran which command on which collection, not how many documents came back. For volume, the slow query log and Atlas Query Profiler record nreturned and docsExamined for slow operations, but not for everything. A bulk export by a legitimate user, done in small fast pages, can look normal in both. Pair auditing with least privilege: the best audit event is the one that says result: 13.

Good alerts to start with: failed authentication bursts, any authCheck failure by a service user, user or role changes made outside your Terraform pipeline, and human reads of sensitive collections outside working hours.

🧠 Extra takeaways

• Log authentication, user and role changes, and schema changes always; log all authorization failures.
• Successful reads need auditAuthorizationSuccess, which costs performance, so filter by collection.
• A service user failing an authCheck is a strong signal: code drift or stolen credentials.
• Audit logs show who and what, not how much; least privilege limits what an insider can pull.