L59 · Testing your code for security

Module 8: Secure SDLC and Supply Chain · ~20 min · Lesson 59 of 71

🎯 What you'll learn

• How to turn a threat or a past bug into an abuse-case test that fails if the attack works
• How coverage-guided fuzzing (Jazzer, Atheris) finds the inputs you'd never think to type
• How property-based testing (Hypothesis, jqwik) checks a rule against hundreds of generated inputs
• Where each one fits next to the scanners from L54

🔥 Why this matters

A pen test finds that any customer can download any statement (L34). The team fixes it in a day. Eight months later a refactor swaps findByIdAndCustomerId back to findById, every test stays green, and the bug ships again. Nothing failed because no test ever said "Bob gets a 404". Security fixes without tests drift back. In 2014, Heartbleed (a missing length check in OpenSSL) was found independently by a Google engineer reading the code and by Codenomicon engineers testing their fuzzer.

1. Happy-path tests don't test security

Most unit tests check that the feature works: valid input, right user, expected output. Attackers send everything else.

 every possible request to GET /v1/statements/{id}
 ┌────────────────────────────────────────────────────────┐
 │  ┌──────────────────┐                                  │
 │  │ happy-path tests │   x Bob asks for Alice's STM-A1  │
 │  │ valid id, owner  │   x id = "../../etc/passwd"      │
 │  │ asks, 200 OK     │   x id = {"$ne": null}           │
 │  └──────────────────┘   x 10 MB id, NUL bytes, emoji   │
 │   you test here         attackers live out here        │
 └────────────────────────────────────────────────────────┘

Three techniques push into that outer area. You write abuse-case tests by hand for attacks you know about. Fuzzers generate huge numbers of strange inputs looking for crashes. Property-based tests generate inputs too, but check a rule you state, such as "the path never leaves the folder".

2. Abuse-case tests: assert the attack fails

An abuse case is a user story told from the attacker's side: "As Bob, I want Alice's statement." An abuse-case test runs that story and asserts it fails. Good sources: rows of your threat model (L05), the OWASP API Top 10 (L39), and every security bug you've ever fixed.

Here is the IDOR from the hook, as a Spring MockMvc test. jwt() from spring-security-test fakes a validated token, so no identity provider is needed:

@WebMvcTest(StatementController.class)
class StatementAbuseTest {
  @Autowired MockMvc mvc;
  @MockitoBean StatementRepository repo; // @MockBean before Boot 3.4

  @Test
  void bobCannotReadAlicesStatement() throws Exception {
    var alices = new Statement("STM-A1", "C-ALICE");
    // make the object EXIST, or a buggy lookup also returns 404
    when(repo.findById("STM-A1")).thenReturn(Optional.of(alices));
    when(repo.findByIdAndCustomerId("STM-A1", "C-ALICE"))
        .thenReturn(Optional.of(alices));

    mvc.perform(get("/v1/statements/STM-A1")
            .with(jwt().jwt(j -> j.claim("customer_id", "C-BOB"))))
       .andExpect(status().isNotFound());
  }
}

Two habits make these tests trustworthy. First, set up data so the attack would succeed against vulnerable code; otherwise the test passes for the wrong reason. Second, pair each deny test with an allow test (Alice gets 200), or an endpoint that always returns 404 passes too. L34's second deep dive scales this up to a full user-by-endpoint matrix.

The same idea in Python, for NoSQL operator injection (L31). One parametrized test holds every payload you know about:

ATTACKS = [
    {"status": {"$ne": None}},             # match everything
    {"status": {"$regex": "^"}},           # same, via regex
    {"currency": {"$where": "sleep(5000)"}},
    {"status": ["OPEN", "CLOSED"]},        # arrays match too
    {"customerId": "C-ALICE"},             # swap the owner
]

@pytest.mark.parametrize("params", ATTACKS)
def test_query_attack_is_neutralized(params):
    try:
        f = build_filter("C-BOB", params)
    except BadRequest:
        return                              # rejected: good
    assert f["customerId"] == "C-BOB"
    assert all(type(v) is str for v in f.values())

Against a build_filter that didn't check value types, four of the five cases failed; adding a type(v) is str check fixed all of them. A new bypass is one more line in ATTACKS.

3. Fuzzing: let the computer invent the inputs

You can only write abuse cases for attacks you can imagine. Fuzzing feeds a function huge numbers of generated inputs and watches for crashes, uncaught exceptions, hangs or memory blowups. Modern fuzzers are coverage-guided: they watch which code paths each input reaches and keep the inputs that reach new ones.

      ┌──────── corpus: inputs that reached new code ◀───────┐
      │                                                      │
      ▼                                                      │
 1. pick input ─▶ 2. mutate ─▶ 3. run target ─▶ new branch? ─┘
                   flip bits,    (instrumented    yes: keep it
                   splice,        for coverage)
                   insert           │
                   "magic" values   ▼
                         4. crash, timeout, OOM?
                            save crash-<hash> file
                            → becomes a regression test

Coverage feedback lets a fuzzer reach deep into a parser: when code checks for "USD", an input that matches one more byte is kept and mutated again.

In Python, Atheris (Google, built on LLVM's libFuzzer) fuzzes a function like this payment amount parser:

import sys, atheris
with atheris.instrument_imports():
    from payments.amounts import parse_amount

def TestOneInput(data: bytes):
    fdp = atheris.FuzzedDataProvider(data)
    raw = fdp.ConsumeUnicodeNoSurrogates(64)
    try:
        parse_amount(raw)        # "1,250.00" -> Decimal
    except ValueError:
        pass                     # documented rejection: fine

atheris.Setup(sys.argv, TestOneInput)
atheris.Fuzz()

If parse_amount uses Decimal, the fuzzer quickly finds inputs like x, NaN or 1E+30 that raise decimal.InvalidOperation. That isn't a ValueError (it's an ArithmeticError), so it escapes your handler and becomes a 500 with a stack trace (L38). Atheris prints the traceback and saves the input as a crash-<hash> file.

In Java, Jazzer (Code Intelligence, Apache-2.0) plugs into JUnit 5:

class AmountParserFuzzTest {
  @FuzzTest(maxDuration = "60s")
  void parse(FuzzedDataProvider data) {
    try {
      AmountParser.parse(data.consumeRemainingAsString());
    } catch (InvalidAmountException expected) { }
  }
}

A normal mvn test replays only the saved crash inputs, so it's fast and deterministic. With JAZZER_FUZZ=1 set, it actually fuzzes. Jazzer also has bug detectors that flag OS command injection, path traversal, SSRF and other dangerous calls even when nothing crashes.

Version notes: current Atheris source supports Python 3.11 to 3.14, and on macOS you may need a clang with libFuzzer (Apple's doesn't ship it), so a Linux container is often easier. Fuzz in short bursts (about a minute) on pull requests and for hours nightly.

4. Property-based testing: state the rule, let it hunt

A unit test checks examples you chose. A property-based test states a rule that must hold for every input, generates hundreds of inputs, and shrinks any failure to the smallest example. Security rules make good properties:

The third row is subtle: float("nan") > 25000 is False, so if amount > limit: reject lets NaN through. A property finds that kind of bug; an example test only does if you think of it.

In Python the tool is Hypothesis (see the worked example). In Java, jqwik plugs into JUnit 5 (its README says it's now in maintenance mode, getting bug fixes only):

@Property
void maskedPanShowsAtMostLastFour(
    @ForAll @NumericChars @StringLength(min = 12, max = 19)
    String pan) {
  String masked = PanMasker.mask(pan);
  assertThat(masked).doesNotContainPattern("[0-9]{5,}");
  assertThat(masked).endsWith(pan.substring(pan.length() - 4));
}

🧪 Worked example: a statement path, found by Hypothesis

A service builds a file path from a statement name and "sanitizes" it by stripping ../:

BASE = "/srv/statements"

def statement_path(name: str) -> str:
    name = name.replace("../", "")        # VULNERABLE
    return os.path.join(BASE, name)

The property: whatever the name, the normalized path stays under BASE.

from hypothesis import given, strategies as st

@given(st.text(alphabet="./ab"))
def test_path_stays_inside_base(name):
    try:
        path = statement_path(name)
    except ValueError:
        return                      # rejecting input is fine
    real = os.path.normpath(path)
    assert os.path.commonpath([BASE, real]) == BASE

Real output (Hypothesis 6.170, pytest 9.2):

E       AssertionError: assert '/' == '/srv/statements'
E       Failing test case: test_path_stays_inside_base(
E           name='/',
E       )

os.path.join throws away BASE when the second part is absolute. "Fix" that with lstrip("/") and the next run fails with name='..'. Blocklists lose this game. The fix that passes is an allowlist:

NAME = re.compile(r"STM-[0-9]{6}\.pdf")

def statement_path(name: str) -> str:
    if not NAME.fullmatch(name):
        raise ValueError("bad statement name")
    return os.path.join(BASE, name)

One surprise: with plain st.text() (any Unicode), the vulnerable version passed 100 examples. Random Unicode almost never produces a lone /. A generator weighted toward characters that matter (., /) found it at once. Choosing the generator well matters as much as stating the property.

🛠 Try it (optional, ~5 min)

Everything runs locally, in a throwaway virtual environment:

mkdir -p ~/tmp/l59 && cd ~/tmp/l59
python3 -m venv .venv && . .venv/bin/activate
pip install -q hypothesis pytest

Save the vulnerable statement_path (with import os) as statements.py and the test (with import os and from statements import statement_path, BASE) as test_statements.py. Run pytest -q and look for the Failing test case line. Then try the lstrip fix, then the allowlist. Finally, change the strategy to st.text() and notice how it passes against the vulnerable code.

🧠 Key takeaways

• Every security fix gets a test that runs the attack and asserts it fails, so the bug can't come back silently.
• Make the attack possible in the test data, and pair each deny test with an allow test.
• Fuzzers find crashes and unhandled errors in parsers; keep their crash files as regression tests.
• Property tests check security rules ("never leaves the folder") across many inputs; the generator decides what they find.
• Allowlists survive generated input; blocklists usually don't.

📖 Words to know

⏭ Next time

L60 starts Module 9 (detection and incident response): what your services should log so you can spot an attack, and what they must never log.

Jazzer and Atheris fuzz one function. Schemathesis fuzzes a whole running API. It reads your OpenAPI spec, generates requests that match (and deliberately break) it, sends them, and checks every response. Under the hood it's built on Hypothesis, so failures are shrunk to a small request you can replay with curl.

How it works

 openapi.json ──▶ Schemathesis ──1. generate──▶ requests per operation
 (the contract)        │          (valid, invalid, boundary values)
                       │
                       │ 2. send            ┌─────────────────────┐
                       └──────────────────▶ │ your API, localhost │
                       ◀────────────────────│ (test data only)    │
                       3. run checks on     └─────────────────────┘
                          each response
                       4. failure? shrink it, print a curl command

Current versions run four phases: examples (from your spec), coverage (boundary and invalid values for each parameter), fuzzing (random generated data), and stateful (chains of calls that follow links in the spec, such as create a payee, then fetch and delete it).

The checks that matter for security

All checks run by default. The security-relevant ones:

Check names change between major versions, and newer releases list more (including an object_level_authorization check), so run st run --help for your version.

Worked example: a transactions API

A FastAPI service on your laptop serves /openapi.json. One operation:

@app.get("/v1/transactions")
def list_txns(limit: int = 20):
    rows = repo.for_customer(current_customer())
    pages = math.ceil(len(rows) / limit)   # limit=0 → crash
    return {"items": rows[:limit], "pages": pages}

Install it as a tool and point it at the local schema, with a token for a local test user:

pipx install schemathesis        # or: uv tool install schemathesis
st run http://localhost:8000/openapi.json \
   -H "Authorization: Bearer $LOCAL_TEST_TOKEN" \
   --max-examples 50

The coverage phase tries boundary and invalid values, such as limit=0 or a negative limit. The first gives ZeroDivisionError, a 500. A negative limit returns a strange slice (rows[:-5]), which shows up only if your spec says what a valid response looks like. The report gives the failing check and a reproduction (exact formatting varies by version):

 GET /v1/transactions
   Server error (not_a_server_error)
   [500] Internal Server Error
   Reproduce with:
     curl -X GET -H 'Authorization: Bearer [Filtered]' \
       'http://localhost:8000/v1/transactions?limit=0'

The fix lives in the contract and the code. limit: int = Query(20, ge=1, le=100) makes FastAPI reject bad values with a 422 and puts minimum: 1, maximum: 100 in the spec, so the next run checks that out-of-range values are rejected. In Spring, the same comes from @Min/@Max with @Validated, which springdoc copies into /v3/api-docs.

Running it safely

Schemathesis sends POST, PUT and DELETE calls with junk data. Run it against localhost or a throwaway environment you own, with test users and test data, never shared QA or production. If a gateway such as Kong validates the same spec at the edge (L43's deep dive), also run it straight at the service, so you find what the service does when the gateway is bypassed.

🧠 Extra takeaways

• Schemathesis turns your OpenAPI spec into thousands of requests and checks every response against it.
• 500s, accepted invalid input and undocumented response fields are all security signals.
• A tighter spec (minimum, maximum, additionalProperties: false) gives the fuzzer more to check.
• It writes and deletes data: localhost or an environment you own only.

A security test that can never fail is worse than none, because it looks like protection. Mutation testing checks your tests: a tool makes small changes to your code (mutants), such as <= to < or == to !=, and runs your tests against each one. If a test fails, the mutant is killed. If every test still passes, it survived, and some behavior has no test watching it.

Why security checks are the best place to start

Authorization and validation code is mostly conditions, and conditions are exactly what mutation tools change. A check like can_approve() should sometimes say no. If flipping or loosening one of its conditions doesn't fail a test, you've proved there's no test covering that case.

 code under test            mutant                   tests
 amount <= limit    ──▶     amount <  limit    ──▶   all pass?
                                                      │
                     yes ◀────────────────────────────┤
          SURVIVED: nobody tests amount == limit      │ no
                                          KILLED ◀────┘

Worked example: L27's approval rule

L27's rule, in Python, with two tests: an approver approves a $50,000 payment, and a non-approver can't.

def can_approve(user: User, p: Payment) -> bool:
    return (
        "payment-approver" in user.roles
        and p.status == "PENDING"
        and p.amount <= user.approval_limit
        and p.created_by != user.id
    )

Running mutmut (configured with source_paths=src/ in setup.cfg) gave this real output (mutmut 3.8):

$ mutmut run
11/11  🎉 10 🫥 0  ⏰ 0  🤔 0  🙁 1  🔇 0  🧙 0
$ mutmut results
    payments.authz.x_can_approve__mutmut_10: survived
$ mutmut show payments.authz.x_can_approve__mutmut_10
-        and p.amount <= user.approval_limit
+        and p.amount < user.approval_limit

Ten mutants died; one lived. Nothing tests a payment of exactly the limit, so nobody would notice if the boundary moved. Adding test_exactly_at_limit_allowed and test_one_over_limit_denied killed it: 11/11 🎉 11.

Notice what mutmut didn't flag. There's still no test that Alice can't approve her own payment. Flipping != to == was killed by the happy-path test, and deleting that whole line isn't one of mutmut's mutations. A clean mutation score means your tests notice the changes the tool knows how to make, not that every rule has a deny test. Add test_own_payment_denied from your threat model anyway.

Making it practical

• Java: PIT (pitest) with the pitest-junit5-plugin. Run mvn test-compile org.pitest:pitest-maven:mutationCoverage and read the HTML report in target/pit-reports. Its defaults include negating conditionals, changing boundaries and replacing boolean returns with true or false. A survived "replaced boolean return with true" on an authorization method means no test ever expects a "no".
• Scope it. Every mutant reruns tests, so full runs are slow. Point it at security packages only (PIT's targetClasses, mutmut's source_paths) and run it nightly or on PRs that touch them.
• Don't chase 100%. Some mutants are equivalent (they change nothing observable). Read each survivor in security code and decide.

🧠 Extra takeaways

• Mutation testing measures whether your tests can fail, which line coverage can't.
• A surviving mutant in an authorization check is a missing deny or boundary test.
• Tools only try their own list of changes; a clean score doesn't replace threat-model-driven abuse cases.
• Scope it to security-critical code and run it where the slowdown is acceptable.