Property-based & Load Testing

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

The reference material consulted while preparing these pages is listed in the section bibliography.

This page is written against FastAPI 0.142.2 (released 2026-09-30), Pydantic 2.13 and Starlette 1.7, together with Schemathesis 4.29, Hypothesis 6.168, Locust 2.46 and Faker 40.40, checked on 2026-10-02. Every example on the page was executed on Python 3.13: the Hypothesis, Schemathesis and security examples with pytest and TestClient, the Schemathesis CLI and Locust against a real uvicorn process, and the outputs shown are copied from those runs. The example-based tests of Testing check the cases you thought of. This page is about the cases you did not think of: a fuzzer that reads the OpenAPI document of the Bookshelf API and attacks every operation, random inputs that must keep an invariant true, a swarm of simulated users that shows how the service behaves under load, fake but realistic data to feed all of it, and a checklist of security behaviours worth pinning down in tests.

The Tools and Where They Fit

Four tools, one application. Each answers a different question and none replaces the example-based tests.

flowchart LR spec["/openapi.json"] --> st["Schemathesis
generates requests from the document"] hyp["Hypothesis
generates values from strategies"] --> st hyp --> prop["your property tests"] fake["Faker
realistic fake data"] --> seed["seed script and test fixtures"] st --> app(("Bookshelf API")) prop --> app seed --> app loc["Locust
simulated users"] --> app sec["security tests, bandit, pip-audit"] --> app

Table 1. The tools compared
Tool	Question it answers	Input it needs
Hypothesis	Does this property hold for every input of this kind, and what is the smallest input that breaks it?	A strategy for each argument, written by you.
Schemathesis	Does every operation behave as its own OpenAPI document promises, and does any input crash it?	The document (`/openapi.json`); it builds the strategies for you and uses Hypothesis underneath.
Locust	How many requests per second, and how slow at the 95th percentile, with this many concurrent users?	A `locustfile.py` that describes a user.
Faker	What does believable data look like?	A locale and, for reproducibility, a seed.
Security tests, bandit, pip-audit	Does the API refuse what it must refuse, and do the code and dependencies contain known problems?	Explicit assertions written from a checklist.

Because Schemathesis generates its requests from /openapi.json, the quality of that document decides the quality of the test. Testing the Schema, and Testing Against It shows the shortest possible run; this page builds the whole workflow around it.

The Application Under Test

One small application serves every example on the page. It protects all routes except /health with an API key (APIKeyHeader), validates input with bounded Pydantic fields, keeps authors, books and reviews in memory, and is built by a create_app(keys) factory so each test can create its own instance with its own keys. The keys of the server process come from an environment variable, never from the source.

bookshelf.py

import os
import secrets
from itertools import count
from typing import Annotated

from fastapi import Depends, FastAPI, HTTPException, Path, Query, Security
from fastapi.security import APIKeyHeader
from pydantic import BaseModel, ConfigDict, Field

STAR_LABELS = ["awful", "poor", "fine", "good", "great"]

api_key_header = APIKeyHeader(name="X-API-Key")


class AuthorIn(BaseModel):
    model_config = ConfigDict(extra="forbid")
    name: Annotated[str, Field(min_length=1, max_length=100)]


class BookIn(BaseModel):
    model_config = ConfigDict(extra="forbid")
    title: Annotated[str, Field(min_length=1, max_length=200)]
    author_id: Annotated[int, Field(ge=1)]
    pages: Annotated[int, Field(ge=1, le=10_000)]


class ReviewIn(BaseModel):
    model_config = ConfigDict(extra="forbid")
    rating: Annotated[int, Field(ge=1, le=5)]
    text: Annotated[str, Field(max_length=500)] = ""


def keys_from_environment() -> dict[str, str]:
    """BOOKSHELF_KEYS="alice:key-for-alice,bob:key-for-bob" -> {key: user}."""
    pairs = os.environ.get("BOOKSHELF_KEYS", "").split(",")
    return {key: user for user, _, key in (p.partition(":") for p in pairs if p)}


def create_app(keys: dict[str, str]) -> FastAPI:
    app = FastAPI(title="Bookshelf API")
    authors: dict[int, dict] = {}
    books: dict[int, dict] = {}
    reviews: dict[int, dict] = {}
    ids = count(1)

    def current_user(key: Annotated[str, Security(api_key_header)]) -> str:
        for known, user in keys.items():
            if secrets.compare_digest(key.encode(), known.encode()):
                return user
        raise HTTPException(status_code=401, detail="Invalid API key")

    User = Annotated[str, Depends(current_user)]
    unauthorized = {401: {"description": "Missing or invalid API key"}}
    not_found = {404: {"description": "Not found"}, **unauthorized}
    bad_body = {400: {"description": "Body is not valid JSON"}}

    @app.get("/health")
    def health() -> dict[str, str]:
        return {"status": "ok"}

    @app.post("/authors", status_code=201, responses=unauthorized | bad_body)
    def create_author(author: AuthorIn, user: User) -> dict:
        new_id = next(ids)
        authors[new_id] = {"id": new_id, **author.model_dump()}
        return authors[new_id]

    @app.post("/books", status_code=201, responses=not_found | bad_body)
    def create_book(book: BookIn, user: User) -> dict:
        if book.author_id not in authors:
            raise HTTPException(status_code=404, detail="Author not found")
        new_id = next(ids)
        books[new_id] = {"id": new_id, **book.model_dump()}
        return books[new_id]

    @app.get("/books", responses=unauthorized)
    def list_books(
        user: User,
        q: Annotated[str | None, Query(max_length=50)] = None,
        limit: Annotated[int, Query(ge=1, le=100)] = 20,
        offset: Annotated[int, Query(ge=0)] = 0,
    ) -> list[dict]:
        found = [b for b in books.values() if q is None or q.lower() in b["title"].lower()]
        return found[offset : offset + limit]

    @app.get("/books/{book_id}", responses=not_found)
    def read_book(book_id: int, user: User) -> dict:
        if book_id not in books:
            raise HTTPException(status_code=404, detail="Book not found")
        return books[book_id]

    @app.post("/books/{book_id}/reviews", status_code=201, responses=not_found | bad_body)
    def create_review(
        book_id: Annotated[int, Path(examples=[2])],
        review: ReviewIn,
        user: User,
    ) -> dict:
        if book_id not in books:
            raise HTTPException(status_code=404, detail="Book not found")
        new_id = next(ids)
        reviews[new_id] = {
            "id": new_id,
            "book_id": book_id,
            "owner": user,
            "label": STAR_LABELS[review.rating - 1],
            **review.model_dump(),
        }
        return reviews[new_id]

    @app.delete("/reviews/{review_id}", status_code=204, responses=not_found)
    def delete_review(review_id: int, user: User) -> None:
        review = reviews.get(review_id)
        if review is None or review["owner"] != user:
            raise HTTPException(status_code=404, detail="Review not found")
        del reviews[review_id]

    return app


app = create_app(keys_from_environment())

Source: FastAPI: Security - First Steps

A few decisions in this file matter for the tests that follow:

Every non-trivial route lists the extra status codes it can return in responses=: 401, 404 and 400 (the answer to a body that is not JSON at all). FastAPI adds 422 on its own. A fuzzer compares the statuses it sees with the documented ones, so an undocumented 404 is reported as a failure of the document, not the server. See Status Codes & Error Handling for the mechanics of responses.
extra="forbid" on the input models turns an unknown field into a 422 instead of silently ignoring it.
Path(examples=[2]) on book_id of the review route gives Schemathesis a value that exists in a seeded application; the section on In process, with pytest explains why that is needed.
The key is compared with secrets.compare_digest, which takes the same time wherever two strings differ.

Start it for the command-line examples with the keys in the environment (the values are placeholders):

export BOOKSHELF_KEYS="alice:change-me-alice,bob:change-me-bob"
uvicorn bookshelf:app --port 8000

Source: FastAPI: First Steps

Property-Based Testing with Hypothesis

An example-based test says "for this input, expect that output". A property-based test says "for every input of this shape, this statement is true", and a library generates the inputs. Hypothesis does the generating, hunts for an input that falsifies the statement, and then shrinks it to the simplest failing case, so a failure reads like a hand-written test. A property is something like "whatever the client sends, the server never answers with a 5xx`", "what I create I can read back unchanged" or "a rating is valid exactly when it is between 1 and 5". Tests are ordinary `pytest functions decorated with @given, and the generators are called strategies:

st.integers(), st.text(), st.just(value): basic strategies, usually constrained (st.integers(1, 10_000)).
st.builds(Model, field=strategy): builds instances of a class, such as a Pydantic model, from strategies for its arguments.
@settings(max_examples=50, deadline=None): how many cases to try, and no per-case time limit (an HTTP round trip, even in process, can exceed the default 200 ms deadline now and then and produce a flaky failure).

The next file states three properties of the Bookshelf API.

test_hypothesis.py

from fastapi.testclient import TestClient
from hypothesis import given, settings
from hypothesis import strategies as st
from pydantic import ValidationError

from bookshelf import BookIn, ReviewIn, create_app

KEY = {"X-API-Key": "change-me-alice"}
client = TestClient(create_app({"change-me-alice": "alice"}))
author_id = client.post("/authors", json={"name": "Octavia E. Butler"}, headers=KEY).json()["id"]

text = st.text(alphabet=st.characters(blacklist_categories=("Cs",)), min_size=1, max_size=200)
books = st.builds(BookIn, title=text, author_id=st.just(author_id), pages=st.integers(1, 10_000))


@settings(max_examples=50, deadline=None)
@given(book=books)
def test_created_book_can_be_read_back(book: BookIn) -> None:
    created = client.post("/books", json=book.model_dump(), headers=KEY)
    assert created.status_code == 201
    fetched = client.get(f"/books/{created.json()['id']}", headers=KEY)
    assert fetched.status_code == 200
    assert fetched.json() == created.json()


@settings(max_examples=50, deadline=None)
@given(q=st.text(alphabet=st.characters(blacklist_categories=("Cs",)), max_size=50),
       limit=st.integers(1, 100), offset=st.integers(0, 1_000))
def test_search_never_fails_and_respects_limit(q: str, limit: int, offset: int) -> None:
    response = client.get("/books", params={"q": q, "limit": limit, "offset": offset}, headers=KEY)
    assert response.status_code == 200
    assert len(response.json()) <= limit


@given(rating=st.integers())
def test_rating_is_valid_exactly_when_between_one_and_five(rating: int) -> None:
    try:
        ReviewIn(rating=rating)
    except ValidationError:
        assert not 1 <= rating <= 5
    else:
        assert 1 <= rating <= 5

Source: Hypothesis: Strategies reference

All three tests passed (3 passed). The first runs 50 times with different valid books and proves the create-then-read round trip; the second sends arbitrary search text with arbitrary limits and offsets and requires 200 and no more than limit results; the third needs no HTTP at all and checks the model’s validation rule against all integers, which is how a boundary (1 and 5) gets tested without writing the boundary cases by hand.

Rules that keep these tests honest:

The characters category Cs (surrogates) is excluded from the text strategy: surrogates cannot be encoded as JSON or in a URL, so a failure caused by them would be a bug in the test, not in the API.
TestClient is created once at module level. Hypothesis refuses a test that uses a function-scoped pytest fixture, with a FailedHealthCheck ("uses a function-scoped fixture"), because the fixture is not reset between generated inputs. Use a module-level object, a fixture with a wider scope, or a context manager inside the test.
State is shared between the generated cases (the books accumulate), so properties must hold regardless of what earlier cases created.
st.builds with a Pydantic model works because the model’s init validates its arguments: a strategy that produces an invalid value fails fast in the test instead of reaching the API.

Hypothesis is the engine behind Schemathesis; the next section lets it write the strategies for you.

Schemathesis: Tests Generated from `/openapi.json`

Schemathesis reads an OpenAPI (or GraphQL) document and, for every operation, generates requests that are valid (positive testing) and requests that deliberately break the schema (negative testing). It sends them, and checks every response against a set of built-in checks. All checks are enabled by default; the ones that matter first are:

Table 2. Built-in checks, summarised
Check	Fails when
`not_a_server_error`	The response is a `5xx`.
`status_code_conformance`	The status code is not documented for the operation.
`content_type_conformance`	The `Content-Type` is missing or not one of the documented media types.
`response_schema_conformance`	The body does not validate against the documented response schema.
`response_headers_conformance`	A documented response header is missing or does not match its schema.
`negative_data_rejection`	A request that violates the schema was accepted (answered with a success status).
`positive_data_acceptance`	A request that satisfies the schema was rejected.
`allow_header_conformance`	A `405` answer carries an `Allow` header that does not list the documented methods.
`ignored_auth`	An operation answered a request that had missing or invalid credentials as if it were authorised.

Source: Schemathesis: Checks reference

Two ways to run it follow: in process inside pytest, which is the better fit for CI of a FastAPI project, and the command line against a running server.

In process, with pytest

schemathesis.openapi.from_asgi("/openapi.json", app) loads the document straight from the ASGI application and sends requests to it without a network or a server. @schema.parametrize() turns one test function into one test per API operation, and case.call_and_validate() sends a generated request and runs the checks.

test_schemathesis.py

import schemathesis
from fastapi.testclient import TestClient

from bookshelf import create_app

KEY = {"X-API-Key": "change-me-alice"}
app = create_app({"change-me-alice": "alice"})

# Seed one author (id 1) and one book (id 2): the id 2 matches the
# path example declared on POST /books/{book_id}/reviews.
seed = TestClient(app)
seed.post("/authors", json={"name": "Ursula K. Le Guin"}, headers=KEY)
seed.post("/books", json={"title": "A Wizard of Earthsea", "author_id": 1, "pages": 183}, headers=KEY)

schema = schemathesis.openapi.from_asgi("/openapi.json", app)


@schema.parametrize()
def test_api(case):
    case.call_and_validate(headers=KEY)

Source: Schemathesis: Testing Python Apps

The application is seeded before the schema is loaded because generated ids are random: a fuzzer posting a review to /books/9183/reviews only ever sees the 404 branch, never the code behind it. The path example declared in bookshelf.py (Path(examples=[2])) is a value Schemathesis uses in its examples phase, and book 2 exists, so the review handler really runs.

A first run of an honest application normally fails, and the failures are informative. These were the first findings on the Bookshelf API, each one a fact about the API that the document did not state:

A 401 (a request without a key, which Schemathesis generates on purpose) and a 400 (an undecodable body) were undocumented. The fix was responses= on the routes, shown in the application above.
405 answers had an Allow header listing only the methods of the one route that matched, because GET /books and POST /books are separate FastAPI routes (allow_header_conformance).
A GET /books?limit=1&=null request, a query string with an unknown parameter, was accepted with 200 (negative_data_rejection). FastAPI ignores unknown query parameters by design, so this is a decision to record, not a bug to fix.

The last two are conscious decisions, so the checks are switched off in a schemathesis.toml next to the tests, which Schemathesis picks up from the working directory:

schemathesis.toml

[checks.allow_header_conformance]
enabled = false

[checks.negative_data_rejection]
enabled = false

Source: Schemathesis: Configuration reference

With those settings the run is green:

$ pytest test_schemathesis.py -q
.......                                                                  [100%]
7 passed

One test passed per operation (seven operations, /health included). Now the same test against a version of the application where the review rating is a plain int without the ge=1, le=5 bounds, and the label is looked up with STAR_LABELS[review.rating - 1]:

class ReviewIn(BaseModel):
    model_config = ConfigDict(extra="forbid")
    rating: int  # the bug: no bounds, so STAR_LABELS[rating - 1] can raise IndexError
    text: Annotated[str, Field(max_length=500)] = ""

Source: Pydantic: Fields

FAILED test_schemathesis_buggy.py::test_api[POST /books/{book_id}/reviews]
E   IndexError: list index out of range
E   Reproduce with:
E       curl -X POST -H 'X-API-Key: [Filtered]' -H 'Content-Type: application/json' -d '{"rating": -5}' http://testserver/books/2/reviews
1 failed, 6 passed

Six of seven operations passed; the review operation failed with a minimal, copy-and-paste reproduction. Putting the bounds back (Field(ge=1, le=5)) turns that input into a 422 and the test green again. Note that the key in the reproduction is shown as [Filtered]: Schemathesis sanitises sensitive header values in its output.

Command line against a running server

The command-line interface needs only the URL of the document and runs more phases than the pytest integration does: the examples from the schema, a coverage phase that systematically generates boundary values, a fuzzing phase of random cases, and a stateful phase that chains operations using links between them. Credentials go in with -H. Against the application above started with uvicorn (with the rating bug in place):

schemathesis run http://127.0.0.1:8000/openapi.json -H "X-API-Key: change-me-alice" --max-examples 50

Source: Schemathesis: CLI reference

The output of that run, trimmed to what matters:

     ✅ 79 passed  ❌ 1 failed

=================================== FAILURES ===================================
________________________ POST /books/{book_id}/reviews _________________________
1. Test Case ID: 3IG1JN

- Server error

- Undocumented HTTP status code

    Received: 500
    Documented: 201, 404, 401, 400, 422

[500] Internal Server Error:

    `Internal Server Error`

Reproduce with:

    curl -X POST -H 'X-API-Key: [Filtered]' -H 'Content-Type: application/json' -d '{"rating": -6945241615817928343552, "text": "..."}' http://127.0.0.1:8000/books/2/reviews

    st replay 3IG1JN

The same fuzzer found a different, equally valid input (a huge negative integer) than the pytest run did. The failure carries an id, and st replay 3IG1JN re-sends exactly that case after you fix the code. The st command is an alias of schemathesis; a run records every failing case as a crash file in a .schemathesis cache directory relative to the working directory, so replay from the folder where you ran the tests. Against the fixed application the whole run ended with:

Test Phases:
  ✅ Examples
  ✅ Coverage
  ✅ Fuzzing
  ✅ Stateful

Test cases:
  879 generated, 879 passed, 10 skipped

The run also printed a warning that deserves attention: Missing test data: 1 operation repeatedly returned 404 Not Found for DELETE /reviews/{review_id}. A fuzzer cannot invent the id of a review that exists. Either supply the identifiers (for example through example values, as above) or describe the relation between operations as links in the OpenAPI document so that the stateful phase can reuse the id returned by POST /books/{book_id}/reviews; the stateful-testing guide in the references explains how. Options that make the command useful in practice:

# pick phases, bound the work, fix the random seed so a failure repeats
schemathesis run http://127.0.0.1:8000/openapi.json -H "X-API-Key: change-me-alice" \
  --phases examples,fuzzing --max-examples 20 --seed 1

# only some checks, or a JUnit report for the CI dashboard
schemathesis run http://127.0.0.1:8000/openapi.json -H "X-API-Key: change-me-alice" \
  --checks not_a_server_error,status_code_conformance --report junit --report-dir reports

Source: Schemathesis: CLI reference

The first command completed with 128 generated, 128 passed and printed Seed: 1; the second form with --report junit wrote junit-<timestamp>.xml into the report directory. A baseline file (--baseline) records failures you have accepted so that only new ones fail the build. Run the CLI against a disposable instance with test data, never against production: it sends hostile input, and POST and DELETE operations really execute.

Realistic Data with Faker

Hypothesis explores the edges; Faker produces the middle: names, sentences, addresses, numbers that look like what a real client sends. Use it for seed data, demo environments, and load tests that should not hit one cached row forever. Faker() takes an optional locale, and Faker.seed(n) makes the sequence repeatable, which is the difference between a flaky test and a reproducible one.

seed_data.py

import sys

import httpx
from faker import Faker

fake = Faker("en_US")
Faker.seed(1234)  # same seed, same data: failures stay reproducible


def make_author() -> dict:
    return {"name": fake.name()}


def make_book(author_id: int) -> dict:
    return {
        "title": fake.sentence(nb_words=4).rstrip("."),
        "author_id": author_id,
        "pages": fake.random_int(min=40, max=1200),
    }


def seed(base_url: str, api_key: str, authors: int = 5, books_per_author: int = 4) -> None:
    headers = {"X-API-Key": api_key}
    with httpx.Client(base_url=base_url, headers=headers) as client:
        for _ in range(authors):
            author = client.post("/authors", json=make_author()).raise_for_status().json()
            for _ in range(books_per_author):
                client.post("/books", json=make_book(author["id"])).raise_for_status()


if __name__ == "__main__":
    seed(sys.argv[1], sys.argv[2])

Source: Faker: Seeding the Generator

Run against a fresh server it created five authors with four books each; the same seed produces the same catalogue every time:

python seed_data.py http://127.0.0.1:8000 change-me-alice

Source: Faker documentation

Faker also fills ordinary tests with values. fake.name(), fake.catch_phrase() and fake.random_int(min, max) are standard providers:

test_faker.py

from fastapi.testclient import TestClient
from faker import Faker

from bookshelf import BookIn, create_app

fake = Faker()
Faker.seed(7)
KEY = {"X-API-Key": "change-me-alice"}
client = TestClient(create_app({"change-me-alice": "alice"}))


def test_fake_catalogue_can_be_created_and_searched() -> None:
    for _ in range(10):
        author = client.post("/authors", json={"name": fake.name()}, headers=KEY).json()
        for _ in range(3):
            book = BookIn(title=fake.catch_phrase(), author_id=author["id"],
                          pages=fake.random_int(40, 1200))
            assert client.post("/books", json=book.model_dump(), headers=KEY).status_code == 201
    everything = client.get("/books", params={"limit": 100}, headers=KEY).json()
    assert len(everything) == 30
    word = everything[0]["title"].split()[0]
    hits = client.get("/books", params={"q": word}, headers=KEY).json()
    assert everything[0] in hits
    print(everything[0]["title"], "|", fake.name())

Source: Faker: Providers

Faker and Hypothesis are complementary, not interchangeable. Faker values look plausible but are not designed to find bugs (it will never produce an empty string or a '; DROP TABLE unless you ask for it), while Hypothesis values look odd but are chosen to corner your code and are shrunk when they fail. Use Faker to populate, Hypothesis to attack. The data model for a real database follows in SQL Databases with SQLModel, where the same factories can fill the tables of a test database.

Load Testing with Locust

A load test answers capacity questions: how many requests per second does one process sustain, what is the 95th percentile latency at 200 users, and where does it start to fail. Locust describes the traffic as Python code. A user class subclasses HttpUser; its @task methods are the actions a user repeats (the number in @task(10) is a relative weight); wait_time = between(a, b) is the pause between tasks; on_start runs once for every user when it is spawned; and self.client is a requests-style session that records the timing of every call.

locustfile.py

import os
import random

from faker import Faker
from locust import HttpUser, between, task

fake = Faker()


class BookshelfReader(HttpUser):
    wait_time = between(0.1, 0.5)  # think time between tasks, in seconds

    def on_start(self) -> None:
        self.client.headers["X-API-Key"] = os.environ["BOOKSHELF_API_KEY"]
        books = self.client.get("/books", params={"limit": 100}, name="/books").json()
        self.book_ids = [book["id"] for book in books]

    @task(10)
    def search_books(self) -> None:
        self.client.get("/books", params={"q": fake.word(), "limit": 20}, name="/books?q")

    @task(5)
    def read_book(self) -> None:
        book_id = random.choice(self.book_ids)
        self.client.get(f"/books/{book_id}", name="/books/{book_id}")

    @task(1)
    def write_review(self) -> None:
        book_id = random.choice(self.book_ids)
        with self.client.post(
            f"/books/{book_id}/reviews",
            json={"rating": random.randint(1, 5), "text": fake.sentence()},
            name="/books/{book_id}/reviews",
            catch_response=True,
        ) as response:
            if response.status_code != 201:
                response.failure(f"expected 201, got {response.status_code}")

Source: Locust: Writing a locustfile

Points worth noting in this file:

The key comes from BOOKSHELF_API_KEY, not from the source file.
name="/books/{book_id}" groups all requests to different ids under one row of the statistics. Without it Locust creates one row per URL and the report is unreadable.
catch_response=True lets the task decide what a failure is: here anything but 201. A plain 4xx/5xx already counts as a failure for the other tasks.
The mix, ten searches for five reads for one write, is an assumption about real traffic; take the real ratios from your access logs.

Start the application with seed data and run Locust without its web interface: twenty users, five new users per second, fifteen seconds.

export BOOKSHELF_KEYS="alice:change-me-alice,bob:change-me-bob"
export BOOKSHELF_API_KEY=change-me-alice
uvicorn bookshelf:app --port 8000 &
python seed_data.py http://127.0.0.1:8000 change-me-alice
locust -f locustfile.py --headless --host http://127.0.0.1:8000 -u 20 -r 5 -t 15s --only-summary --csv results

Source: Locust: Running without the web UI

The summary, as printed by Locust 2.46.6 on a laptop (response times in milliseconds):

Type     Name                        # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
--------|----------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
GET      /books                          20     0(0.00%) |      4       2      14      3 |    1.40        0.00
GET      /books/{book_id}               257     0(0.00%) |      1       0       8      1 |   18.01        0.00
POST     /books/{book_id}/reviews        67     0(0.00%) |      0       0       2      1 |    4.70        0.00
GET      /books?q                       522     0(0.00%) |      0       0       3      1 |   36.59        0.00
--------|----------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
         Aggregated                     866     0(0.00%) |      1       0      14      1 |   60.70        0.00

866 requests, no failures, 99th percentile of 4 ms, with the load generator and the server on one machine. Those numbers say little about production, and they are not a benchmark of FastAPI. What the run proves is that the script, the data and the authentication work; the numbers become meaningful only when the target is a production-like deployment (several worker processes, a real database) and the generator runs on a separate machine.

The options used above, and the ones that complete a CI job:

Table 3. Locust command-line options
Option	Meaning
`-f`, `--locustfile`	The file that defines the users.
`--headless`	Run without the web interface and start immediately. `-u` is the peak number of users, `-r` the spawn rate per second and `-t` the run time (`15s`, `5m`); without `-t` it runs until stopped.
`--host`	Base URL of the target; it replaces the `host` attribute of the user class.
`--csv`, `--html`	Write statistics as CSV files with the given prefix, or a self-contained HTML report.
`--only-summary`	Suppress the periodic statistics table during a headless run.
`--exit-code-on-error`	The process exit code to use when there were failed requests, so a pipeline can fail on them.

Source: Locust: Configuration

When one Locust process is the bottleneck (it warns about CPU use), switch the base class to FastHttpUser, which uses a lighter HTTP client, and run several processes in master and worker mode. Test against the server you will deploy, for example fastapi run with several workers; see Deployment.

from locust import FastHttpUser, task


class BookshelfFastReader(FastHttpUser):
    @task
    def health(self) -> None:
        self.client.get("/health")

Source: Locust: Increase performance with a more efficient HTTP client

Load-test discipline: never aim a load test at a service or third-party API you do not own, give it its own environment and database, warm the application up before you read numbers, and watch the server (CPU, memory, event-loop lag, database connections) while the test runs, since a clean response-time column can hide a saturated pool. Async & Concurrency explains the blocking-code mistakes that load tests usually reveal first.

A Basic Security-Testing Checklist

Security tests are example-based tests whose examples come from a list of known ways APIs fail. The categories are those of the OWASP API Security Top 10; the checklist below names what to assert for the Bookshelf API, and the tests after it run as part of the normal suite.

Table 4. Checklist, with the test or tool that covers each item
Concern	Assertion	Covered by
Authentication	Every route except health answers `401` without a key, with a wrong key and with an empty key.	`test_every_route_needs_a_valid_key`
Object-level authorization	A user cannot read or delete another user’s objects by guessing ids.	`test_users_cannot_touch_each_others_reviews`
Mass assignment	Fields the client must not set (here `owner`) are rejected, not ignored.	`extra="forbid"` and `test_clients_cannot_set_fields_they_do_not_own`
Input limits	Maximum lengths, ranges and page sizes exist and are enforced before the handler runs.	`test_oversized_and_out_of_range_input_is_rejected_early`
Injection	Hostile strings are stored and returned as data; database access uses bound parameters.	`test_hostile_strings_are_data_not_code`, and parameterised queries as shown in SQL Databases with SQLModel
Error leakage	A `500` returns a generic body, never a traceback, a path or a secret.	`test_unexpected_errors_do_not_leak_internals`
CORS	Only the listed origins receive CORS headers; no wildcard together with credentials. See Middleware, CORS & Proxies.	`test_cors_allows_only_listed_origins`
Exposed documentation	Interactive docs and the schema can be switched off where they should not be public.	`test_documentation_can_be_switched_off_in_production`
Fuzzing	No generated input produces a `5xx` and the API behaves as its document says.	Schemathesis, above
Static analysis	The code has no known dangerous patterns (hard-coded secrets, `eval`, weak hashes).	`bandit`
Dependencies	No installed or pinned package has a known vulnerability.	`pip-audit`
Rate limiting, TLS, headers	Enforced at the reverse proxy or gateway; verify with a smoke test against the deployed system.	Outside the application tests

Source: OWASP API Security Top 10 (2023)

test_security.py

import pytest
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from fastapi.testclient import TestClient

from bookshelf import create_app

ALICE = {"X-API-Key": "change-me-alice"}
BOB = {"X-API-Key": "change-me-bob"}
app = create_app({"change-me-alice": "alice", "change-me-bob": "bob"})
client = TestClient(app)
client.post("/authors", json={"name": "Ursula K. Le Guin"}, headers=ALICE)
client.post("/books", json={"title": "The Dispossessed", "author_id": 1, "pages": 387}, headers=ALICE)


@pytest.mark.parametrize("headers", [{}, {"X-API-Key": "wrong"}, {"X-API-Key": ""}])
def test_every_route_needs_a_valid_key(headers: dict[str, str]) -> None:
    for method, url in [("GET", "/books"), ("GET", "/books/2"), ("DELETE", "/reviews/3")]:
        response = client.request(method, url, headers=headers)
        assert response.status_code == 401


def test_users_cannot_touch_each_others_reviews() -> None:  # broken object-level authorization
    review = client.post("/books/2/reviews", json={"rating": 5}, headers=ALICE).json()
    assert client.delete(f"/reviews/{review['id']}", headers=BOB).status_code == 404
    assert client.delete(f"/reviews/{review['id']}", headers=ALICE).status_code == 204


def test_clients_cannot_set_fields_they_do_not_own() -> None:  # mass assignment
    response = client.post("/books/2/reviews", json={"rating": 5, "owner": "bob"}, headers=ALICE)
    assert response.status_code == 422


@pytest.mark.parametrize(
    "url", ["/books?limit=1000000", "/books?offset=-1", f"/books?q={'x' * 51}", "/books/99999999999999999999"]
)
def test_oversized_and_out_of_range_input_is_rejected_early(url: str) -> None:
    assert client.get(url, headers=ALICE).status_code in {404, 422}


def test_hostile_strings_are_data_not_code() -> None:
    payload = "'; DROP TABLE books; --"
    created = client.post("/authors", json={"name": payload}, headers=ALICE)
    assert created.status_code == 201
    assert created.json()["name"] == payload
    assert client.get("/books", params={"q": payload}, headers=ALICE).status_code == 200


def test_unexpected_errors_do_not_leak_internals() -> None:
    leaky = FastAPI()

    @leaky.get("/boom")
    def boom() -> None:
        raise RuntimeError("password=hunter2 at /srv/bookshelf/secret.py")

    response = TestClient(leaky, raise_server_exceptions=False).get("/boom")
    assert response.status_code == 500
    assert response.text == "Internal Server Error"


def test_cors_allows_only_listed_origins() -> None:
    cors = FastAPI()
    cors.add_middleware(CORSMiddleware, allow_origins=["https://app.example.com"], allow_methods=["GET"])
    preflight = {"Access-Control-Request-Method": "GET"}
    good = TestClient(cors).options("/", headers={"Origin": "https://app.example.com", **preflight})
    evil = TestClient(cors).options("/", headers={"Origin": "https://evil.example", **preflight})
    assert good.headers["access-control-allow-origin"] == "https://app.example.com"
    assert evil.status_code == 400
    assert "access-control-allow-origin" not in evil.headers


def test_documentation_can_be_switched_off_in_production() -> None:
    hidden = TestClient(FastAPI(docs_url=None, redoc_url=None, openapi_url=None))
    assert [hidden.get(u).status_code for u in ("/docs", "/redoc", "/openapi.json")] == [404, 404, 404]

Source: FastAPI: Testing

Thirteen test cases passed (the parametrised tests expand to several). Two of them show features worth knowing: raise_server_exceptions=False makes TestClient return the 500 response a real client would see instead of re-raising the exception inside the test, and FastAPI(docs_url=None, redoc_url=None, openapi_url=None) removes the interactive documentation and the schema entirely (a deployment that hides the schema also loses the Schemathesis workflow above, so run it before the switch is applied, in a test environment).

Two command-line tools cover what unit tests cannot: bandit scans the source for risky constructs, and pip-audit checks a requirements file (or the current environment) against a vulnerability database. Both ran clean here (No known vulnerabilities found), and both belong in CI so that a new advisory fails a build the day it is published.

bandit -r bookshelf.py
pip-audit -r requirements.txt

Source: Bandit documentation

Source: pip-audit documentation

The checklist is a floor. Authentication design, token handling and scopes are covered on Security Fundamentals and OAuth2 Password Flow & JWT; a penetration test and a threat model are not replaced by any of the above.

Putting It Together

A reasonable arrangement for a service like the Bookshelf API:

On every commit: the example tests of Testing, the Hypothesis properties, the in-process Schemathesis test (seven operations, a few seconds), the security tests, bandit and pip-audit.
Nightly or before a release: the Schemathesis CLI with more examples, against a deployed test environment with seeded data, and a headless Locust run with a failure threshold.
Before a launch or a capacity change: a longer Locust run on a production-like environment, read together with the server’s own metrics.

Keep the seed (Faker.seed, Schemathesis --seed) in the log of every run so that any failure can be repeated.

Where Older Guides Are Outdated, and What Changed Recently

Table 5. Older guides and today
Older guides and code	Today
`schemathesis.from_asgi(…)` and `schemathesis.from_uri(…)` from the 3.x series	Schemathesis 4 moves the loaders under `schemathesis.openapi` (`from_asgi`, `from_url`, `from_path`, `from_dict`), runs all data-generation modes and all checks by default, replays recorded crash files with `st replay`, and takes settings from a `schemathesis.toml`; its migration guide lists every rename
Only a hand-written list of positive and negative inputs per endpoint	Property-based generation from the OpenAPI document, which FastAPI emits as OpenAPI 3.1 since 0.99.0 (2023-06-30)
`locust --no-web`, `-c` / `--clients` and `--hatch-rate` in tutorials	`--headless` (renamed from `--no-web`), `-u` / `--users` and `-r` / `--spawn-rate` (the deprecated `--hatch-rate` was removed in Locust 2.32.6); Locust 2.46 needs Python 3.11 or newer
One load test number quoted as the speed of FastAPI	Results depend on workers, the database and the generator; run on production-like infrastructure
Test data typed by hand or copied from a book’s dataset	Faker with a seed, or Hypothesis strategies
`TestClient` examples without noise	Starlette 1.7 emits a deprecation warning when `starlette.testclient` runs on `httpx` 0.28 and suggests the `httpx2` package; the warning does not change behaviour, and the choice is discussed on Testing
401/403 assumptions for missing credentials	Security classes answer `401` when credentials are missing since 0.122.0 (2025-11-24)
Pydantic v1 `class Config` / `.dict()` in property-test helpers	`ConfigDict` and `model_dump()`; FastAPI dropped Pydantic v1 in 0.126.0 (2025-12-20)

Release history is collected on What’s Changed & Migration.

Pitfalls

Fuzzing an application with side effects. Schemathesis really calls POST and DELETE. Point it at a disposable instance with its own database, never at production, and never with a key that can reach real data.
Random ids that never exist. Without examples, seeded data or links, every path-parameter operation answers 404 and its handler is never exercised (the Missing test data warning).
Reading negative_data_rejection as a defect list. FastAPI ignores unknown query parameters and extra headers; decide per check, document the decision, and switch it off in configuration rather than ignoring red builds.
Forgetting that Schemathesis tests the unauthenticated case too. Even with a valid key passed, it sends requests without credentials on purpose (coverage phase and the ignored_auth check), so every protected operation must document 401, as the first failures of this page showed.
Function-scoped fixtures with @given. They are not reset per generated input and Hypothesis raises a health check error; use a module-level client or a wider scope.
Using a per-case deadline for HTTP tests. Leave the default deadline and a slow case becomes a flaky failure; set deadline=None.
Surrogate characters in generated text. They cannot be encoded in JSON or URLs; exclude category Cs.
Trusting a green load test on a laptop. Generator and server compete for the same CPU, there is one worker, and the database is in memory. The result measures your laptop.
One URL per row in Locust. Group requests with name=; otherwise the statistics have a row per id.
Hard-coding keys in locustfiles, seed scripts and CI. Read them from environment variables or a secret store; the placeholders on this page are not credentials.
Treating the checklist as a certificate. Passing these tests shows the listed behaviours, not that the API is secure.