L52 · Cloud logging and threat detection

Module 7: Cloud, Containers, and Kubernetes · ~20 min · Lesson 52 of 71

🎯 What you'll learn

• What CloudTrail records, the difference between management and data events, and how organization trails and log file validation make it trustworthy
• Where VPC Flow Logs, EKS audit logs and GuardDuty fit next to it
• The first things an attacker does after landing in AWS, and the trail each step leaves
• Which events deserve an alert on day one

🔥 Why this matters

A developer pushes a branch with an .env file containing an access key for svc-report-export (L19). Bots scan public repositories constantly, and leaked keys are often tried within minutes. The attacker's first call is aws sts get-caller-identity: "whose key is this?" If you know what that and the next few calls look like in your logs, you can stop them before the data walks out.

1. Four sources, four questions

 AWS API calls   ──▶ CloudTrail ─────┐
 ENI traffic     ──▶ VPC Flow Logs ──┤
 DNS queries     ──▶ Route 53 logs ──┼──▶ GuardDuty ──▶ findings
 kubectl, pods   ──▶ EKS audit log ──┤                     │
 node syscalls   ──▶ runtime agent ──┘                     ▼
                                              EventBridge ──▶ on-call

CloudTrail answers "who called which AWS API?", flow logs "which IPs talked?", the EKS audit log "who did what in Kubernetes?", and GuardDuty "does any of this look malicious?". GuardDuty pulls its own copies of CloudTrail, flow log, DNS and EKS audit data. It works even if you never enabled those logs, but then you have a finding and no evidence to investigate it with (L45's deep dive). You need both.

2. CloudTrail: the flight recorder for AWS APIs

Every AWS API call, from the console, CLI, an SDK in your pod, or another AWS service, becomes a CloudTrail event: who called (userIdentity), what (eventSource, eventName), when, from where (sourceIPAddress, userAgent), with which parameters, and whether it failed (errorCode).

Out of the box you get event history: 90 days of management events per region, searchable but not exportable as evidence. For anything more you create a trail, which delivers log files to S3 (and optionally CloudWatch Logs) for as long as you keep them.

The kind of event decides what you can investigate later:

The trap: without S3 data events, "which objects did the attacker download from example-statements?" has no answer. You don't need them everywhere. Turn them on with advanced event selectors for the buckets holding customer or financial data.

Three settings make a trail trustworthy:

• Multi-region. Attackers like regions you don't use. Global services such as IAM log to us-east-1.
• Organization trail. Created in the management account (or by a delegated administrator), it logs every account in the organization. Member accounts can see it but can't stop or delete it, so an attacker with admin in one account can't switch it off.
• Log file integrity validation. Every hour CloudTrail writes a digest file with a SHA-256 hash of each log file delivered, signed with RSA, and chained to the previous digest. aws cloudtrail validate-logs then tells you whether any file was changed or deleted. It detects tampering; preventing it is deep dive 1.

CloudTrail isn't real time. AWS says delivery typically averages about 5 minutes, without a guarantee.

3. The other three sources

VPC Flow Logs are L3/L4 metadata per network interface (L07's deep dive): useful for spotting exfiltration volume or scanning, blind to payloads and to DNS and metadata-service traffic.

EKS audit logs answer what CloudTrail can't. CloudTrail sees eks:CreateCluster; it doesn't see kubectl get secrets. Turn on the audit (and authenticator) control plane log types and EKS sends them to CloudWatch Logs under /aws/eks/<cluster>/cluster. They're off by default. An event, trimmed:

{
  "verb": "list",
  "user": {
    "username": "system:serviceaccount:payments:quotes-api"
  },
  "sourceIPs": ["10.20.3.41"],
  "objectRef": {"resource": "secrets", "namespace": "payments"},
  "responseStatus": {"code": 200},
  "annotations": {"authorization.k8s.io/decision": "allow"}
}

An API's service account listing every Secret in its namespace deserves a question. PSA's audit mode (L51) writes here too.

GuardDuty is AWS's managed threat detection. It's regional, so enable it in every region, organization-wide. On top of its foundational sources you add protection plans: S3 Protection, EKS Protection (audit logs), Runtime Monitoring (L51) and others. Findings are named ThreatPurpose:ResourceType/ThreatName:

Note the third one's default severity: Low. Don't page only on High, or you'll miss someone switching off your logs.

4. What an attacker does first

Whether the key came from a repo, a laptop, or an SSRF bug (L35), the opening moves are similar:

 stolen credentials
   │
   ▼ 1. Who am I?      sts:GetCallerIdentity
   │
   ▼ 2. What can I     iam:List*/Get*, ec2:Describe*,
   │    do?            s3:ListBuckets... across regions,
   │                   with bursts of AccessDenied
   │
   ▼ 3. Blind you      cloudtrail:StopLogging, DeleteTrail,
   │                   PutEventSelectors,
   │                   guardduty:DeleteDetector,
   │                   ec2:DeleteFlowLogs
   │
   ▼ 4. Stay in        iam:CreateAccessKey, CreateUser,
   │                   CreateLoginProfile, CreateRole,
   │                   UpdateAssumeRolePolicy,
   │                   eks:CreateAccessEntry
   │
   ▼ 5. Act            mass s3:GetObject, snapshots
                       shared out, crypto mining

1. Who am I. GetCallerIdentity needs no permission and can't be denied (L46), but it's logged. Some tools learn the ARN from other services' error messages instead, so this event is a clue, not a guarantee.
2. Enumerate. Applications call the same few APIs every day. A person or tool exploring calls dozens of different ones, and many fail. A burst of AccessDenied from one identity is one of the best cheap signals you have.
3. Disable logging. Org trails and SCPs (deep dive 1) make this fail; the attempt is logged.
4. Persist. A second access key on the same user survives you rotating the first. A role whose trust policy now trusts the attacker's account survives every key rotation. An EKS access entry with an admin access policy gives their identity the cluster, recorded in CloudTrail rather than the audit log.

🧪 Worked example: reading the leaked key's trail

Four events from svc-report-export, trimmed. The first in full:

{
  "eventTime": "2026-11-05T02:14:07Z",
  "eventSource": "sts.amazonaws.com",
  "eventName": "GetCallerIdentity",
  "awsRegion": "us-east-1",
  "sourceIPAddress": "203.0.113.50",
  "userAgent": "aws-cli/2.17.0 Python/3.11.8 Linux/6.5.0",
  "userIdentity": {
    "type": "IAMUser",
    "userName": "svc-report-export",
    "accessKeyId": "AKIAEXAMPLEEXPORT001",
    "accountId": "111122223333"
  },
  "readOnly": true
}

Response: deactivate both keys, then trace every call by either key ID. And ask why a report job may call iam:CreateAccessKey at all (L46).

What should alert on day one, mostly through EventBridge rules on CloudTrail write events:

An EventBridge pattern for the first row (it needs an enabled trail in the account):

{
  "source": ["aws.cloudtrail"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventSource": ["cloudtrail.amazonaws.com"],
    "eventName": ["StopLogging", "DeleteTrail",
                  "UpdateTrail", "PutEventSelectors"]
  }
}

🛠 Try it (optional, ~5 min)

Read-only, if you have AWS CLI access to a dev account (use the profile you'd normally use there). First make a call, then find it:

aws sts get-caller-identity
# wait ~5-10 minutes, then:
aws cloudtrail lookup-events --max-results 3 \
  --lookup-attributes \
    AttributeKey=EventName,AttributeValue=GetCallerIdentity \
  --query 'Events[].CloudTrailEvent' --output text \
  | jq '{eventTime, sourceIPAddress, userAgent,
         arn: .userIdentity.arn}'
aws cloudtrail describe-trails --query 'trailList[].{
  name:Name, org:IsOrganizationTrail,
  multiRegion:IsMultiRegionTrail,
  validation:LogFileValidationEnabled}'

Find your own call: your public IP, CLI version and (if you sign in through IAM Identity Center) your SSO session name, exactly what a defender sees of an attacker. In describe-trails, look for an org trail with validation on. A denied command just means your role or permission set lacks it.

🧠 Key takeaways

• CloudTrail logs management events by default; data events like S3 GetObject are opt-in, so enable them for sensitive buckets.
• An organization trail, multi-region, with log file validation, is the baseline; member accounts can't turn it off.
• CloudTrail doesn't see Kubernetes API calls; turn on EKS audit logs.
• Attackers go: who am I, enumerate, blind you, persist. Each step leaves a CloudTrail event.
• Alert on logging changes and new credentials at any severity; CloudTrailLoggingDisabled defaults to Low.

📖 Words to know

⏭ Next time

L53 closes the module with database security: MongoDB authentication, network exposure, field-level encryption, and the ransom wave against databases left open to the internet.

An attacker who can't stop logging will try to delete what was logged. Digest files tell you that happened; they don't bring the files back. The fix is to put the logs where the attacker can't reach.

The layout

 Workload accounts (dev, prod)          Log archive account
 ┌───────────────────────────┐          ┌───────────────────────┐
 │ org trail (can't modify)  │──logs──▶ │ S3: example-org-trail │
 │ SCP: deny StopLogging,    │          │  Object Lock (compl.) │
 │  DeleteDetector, ...      │          │  bucket policy: only  │
 │ GuardDuty member          │          │  CloudTrail may write │
 └───────────────────────────┘          │ KMS key, own policy   │
                                        │ ~5 people, read-only  │
 Management account: creates the        └───────────────────────┘
 org trail; used for little else

• A separate account. Admin in prod gives no rights in the log archive account. Few people have access there, and mostly read-only.
• S3 Object Lock in compliance mode. Objects can't be overwritten or deleted by anyone, including the root user, until the retention period ends. Governance mode is the softer option: users with a special permission can bypass it. Compliance mode is also a commitment: you pay to store the data for the whole period, so choose the retention deliberately.
• Encryption with a KMS key in the archive account, whose key policy lets CloudTrail encrypt and only investigators decrypt (L19).
• Log file validation on, so you can prove files are untouched.

SCPs for what the org trail doesn't cover

The org trail is protected for you. Other sources aren't: GuardDuty, flow logs, EKS logging and CloudWatch log groups can all be turned off by an admin in the account. An SCP closes that gap. SCPs only limit (L27's deep dive), and they never apply to the management account.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "ProtectSecurityLogging",
    "Effect": "Deny",
    "Action": [
      "cloudtrail:StopLogging",
      "cloudtrail:DeleteTrail",
      "cloudtrail:UpdateTrail",
      "cloudtrail:PutEventSelectors",
      "guardduty:DeleteDetector",
      "guardduty:UpdateDetector",
      "guardduty:DisassociateFromAdministratorAccount",
      "ec2:DeleteFlowLogs",
      "logs:DeleteLogGroup"
    ],
    "Resource": "*",
    "Condition": {"ArnNotLike": {"aws:PrincipalArn":
      "arn:aws:iam::*:role/example-security-breakglass"}}
  }]
}

Two refinements in real life. logs:DeleteLogGroup on every group will annoy teams; scope it to /aws/eks/* and your security log groups. And the break-glass exception should itself alert whenever that role is used. To stop someone turning off EKS audit logging, the call is eks:UpdateClusterConfig, which also changes other settings, so many teams alert on it instead of denying it.

Worked example: proving nothing is missing

After the 02:14 incident, an auditor asks: "How do you know the attacker didn't delete log files from that night?"

$ aws cloudtrail validate-logs \
    --trail-arn arn:aws:cloudtrail:us-east-1:111122223333:\
trail/example-org-trail \
    --start-time 2026-11-05T00:00:00Z \
    --end-time 2026-11-05T06:00:00Z
Validating log files for trail ... between
2026-11-05T00:00:00Z and 2026-11-05T06:00:00Z

Results requested for 2026-11-05T00:00:00Z to ...
Results found for 2026-11-05T00:12:41Z to ...:

6/6 digest files valid
212/212 log files valid

(Illustrative output; the exact wording varies by CLI version.)

Because each digest is signed and references the previous one, a gap or an edited file shows up as invalid or missing. Combined with Object Lock, you can say "these files couldn't have been deleted, and these are unmodified" rather than "we don't think anyone touched them".

🧠 Extra takeaways

• Send logs to a separate log archive account the workload admins can't touch.
• Object Lock compliance mode prevents deletion, even by root, until retention ends; choose the period with care.
• SCPs protect GuardDuty, flow logs and log groups, which the org trail doesn't cover.
• Validation proves integrity after the fact; Object Lock prevents the damage.

In L35, SSRF let an attacker fetch credentials from a server. On EKS, the prize is usually the pod's role session (IRSA or Pod Identity, L50), or the node's role if the metadata service is reachable. Those are temporary credentials, but they last hours, which is plenty.

What GuardDuty covers, and what it doesn't say it covers

GuardDuty's InstanceCredentialExfiltration.OutsideAWS and .InsideAWS findings fire when an EC2 instance's role credentials are used from outside AWS or from another AWS account. That catches a pod that stole the node's credentials. ResourceCredentialExfiltration covers ECS task and Lambda credentials. The current finding descriptions don't mention IRSA or Pod Identity sessions, so don't assume they're covered. Check the docs for your setup, and build your own check for the roles you care about most.

What "normal" looks like

Pods in a typical EKS setup reach AWS through one of two paths, and each shows up differently in CloudTrail:

 pod ──▶ NAT gateway ──▶ public AWS endpoint
         CloudTrail: sourceIPAddress = NAT IP
                     (198.51.100.10 or .11)

 pod ──▶ VPC endpoint ──▶ AWS service
         CloudTrail: sourceIPAddress = pod's private IP
                     vpcEndpointId   = vpce-0abc...

 attacker laptop ──▶ public AWS endpoint
         CloudTrail: sourceIPAddress = 203.0.113.50
                     no vpcEndpointId          ◀── suspicious

When an AWS service calls on your behalf, sourceIPAddress holds the service's name (such as s3.amazonaws.com) instead of an IP.

Worked example: an Athena hunt

With the trail's S3 logs set up as an Athena table (AWS documents the standard table definition), look for the quotes-api role used from anywhere unexpected:

SELECT eventtime, eventname, sourceipaddress, useragent
FROM cloudtrail_logs
WHERE useridentity.arn LIKE
      '%:assumed-role/quotes-api-role/%'
  AND vpcendpointid IS NULL
  AND sourceipaddress NOT IN
      ('198.51.100.10', '198.51.100.11')
  AND sourceipaddress NOT LIKE '%amazonaws.com'
  AND eventtime > '2026-11-01'
ORDER BY eventtime;

Zero rows is the goal. One row from 203.0.113.50 with a different useragent than your SDK is an incident. Once it runs clean for a week, turn it into a scheduled detection (L62 covers detection-as-code).

Prevention: make stolen credentials useless elsewhere

Detection tells you after the fact. A data perimeter condition in the role's policy makes the credentials fail anywhere but your network:

{
  "Sid": "DenyOutsideOurNetwork",
  "Effect": "Deny",
  "Action": "*",
  "Resource": "*",
  "Condition": {
    "NotIpAddressIfExists": {"aws:SourceIp": [
      "198.51.100.10/32", "198.51.100.11/32"]},
    "StringNotEqualsIfExists": {
      "aws:SourceVpc": "vpc-0abc1234example"},
    "BoolIfExists": {"aws:ViaAWSService": "false"}
  }
}

All three conditions must hold for the deny to apply. A call from your NAT IPs fails the first. A call through your VPC endpoint fails the second. A call that an AWS service makes for you fails the third. The attacker's laptop matches all three and is denied. These keys have subtleties (for example, aws:SourceIp isn't useful for calls that arrive through a VPC endpoint), so use AWS's published data perimeter examples as the reference and test in dev. AWS also lets you apply the same idea organization-wide with SCPs and resource control policies.

🧠 Extra takeaways

• GuardDuty's credential exfiltration findings name EC2, ECS and Lambda credentials; don't assume IRSA or Pod Identity coverage.
• Normal pod traffic comes from your NAT IPs or through a VPC endpoint; anything else is worth a look.
• A scheduled query on sourceIPAddress and vpcEndpointId is a cheap, precise detection.
• Data perimeter conditions make stolen credentials fail outside your network.