Performance Tuning

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

The reference material consulted while preparing these pages is listed in the section bibliography.

This page is written against axum 0.8.9 with tokio 1.53, hyper-util 0.1.21, mimalloc 0.1.52 and oha 1.16 on Rust 1.99 (edition 2024), checked on 2026-10-06. axum adds very little overhead on top of hyper, so the performance of a Bookshelf service is decided by what handlers do: blocking calls on async workers, database round trips, allocation and serialisation. General Rust optimisation (build settings, allocation avoidance, profilers, benchmarks) is in the Rust Reference’s Performance & Profiling; this page covers the server side.

Measure first

Benchmark a release build on hardware close to production; debug builds of Rust can be an order of magnitude slower or more and tell nothing about production.
Report latency percentiles (p50, p99, p99.9) at a given request rate, not only requests per second. A service that does 20 000 req/s with a p99 of 2 s is not fast.
oha (Rust, a terminal UI) and k6 (JavaScript scenarios, thresholds for CI) generate load; run them from another machine for real numbers so the generator does not compete for CPU.
Change one thing at a time and keep the before and after numbers.

Release profile

[profile.release]
lto = "thin"                 # cross-crate inlining; "fat" squeezes a little more for much longer links
codegen-units = 1            # better optimisation, slower compile
debug = "line-tables-only"   # symbols for flamegraphs and panic backtraces, small binary cost
# panic = "abort"            # do NOT use in servers: one panicking handler would kill the whole process

Source: Cargo: the release profile

Keep panic = "unwind" (the default) for a server. With unwinding, a panic in a handler aborts that one request’s task (and CatchPanicLayer can turn it into a 500); with abort every in-flight request dies with it.
opt-level = 3 is already the release default. target-cpu=native is only safe when the build machine and the production CPUs match, which is rarely true for container images.

Never block the async workers

Tokio runs every task on a few worker threads (one per core by default). A handler that does CPU-heavy work or blocking I/O without yielding holds its worker, and every other task scheduled there — including unrelated, cheap requests — waits. Password hashing, image resizing, large JSON serialisation, compression of big files, std::fs and synchronous database drivers all qualify:

use argon2::{Argon2, password_hash::PasswordHasher};
use axum::{
    Router,
    body::Bytes,
    extract::State,
    http::{StatusCode, header},
    response::IntoResponse,
    routing::{get, post},
};
use tracing::{Span, info_span, instrument};

// a faster general-purpose allocator for allocation-heavy services (measure before and after)
#[global_allocator]
static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;

#[derive(Clone)]
struct AppState {
    cover: Bytes, // reference-counted: cloning shares the buffer instead of copying it
}

/// Anti-pattern: CPU-bound work (tens of milliseconds of Argon2) blocks the worker thread it runs on.
async fn hash_inline(body: String) -> Result<String, StatusCode> {
    Argon2::default()
        .hash_password(body.as_bytes())
        .map(|h| h.to_string())
        .map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)
}

/// The same work on the blocking pool, keeping the request span as the parent.
#[instrument(skip_all)]
async fn hash_offloaded(body: String) -> Result<String, StatusCode> {
    let span = Span::current();
    tokio::task::spawn_blocking(move || {
        let _guard = info_span!(parent: &span, "argon2_hash").entered();
        Argon2::default().hash_password(body.as_bytes()).map(|h| h.to_string())
    })
    .await
    .map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?
    .map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)
}

async fn cover(State(state): State<AppState>) -> impl IntoResponse {
    ([(header::CONTENT_TYPE, "image/png")], state.cover.clone()) // no copy of the bytes
}

async fn ping() -> &'static str {
    "pong\n"
}

#[tokio::main]
async fn main() {
    tracing_subscriber::fmt().with_env_filter("warn").init();
    let state = AppState { cover: Bytes::from(vec![0u8; 64 * 1024]) };

    let app = Router::new()
        .route("/hash/inline", post(hash_inline))
        .route("/hash", post(hash_offloaded))
        .route("/cover", get(cover))
        .route("/ping", get(ping))
        .with_state(state);

    let listener = tokio::net::TcpListener::bind("127.0.0.1:3000").await.unwrap();
    axum::serve(listener, app).await.unwrap();
}

Source: tokio: spawn_blocking, Alice Ryhl: Async: What is blocking? and mimalloc

Measured on 4 cores: 8 concurrent clients hash passwords for 10 s while a single client sends /ping requests.

$ oha -z 10s -c 8 -m POST -d 'correct horse battery staple' http://127.0.0.1:3000/hash/inline &
$ oha -n 200 -c 1 http://127.0.0.1:3000/ping
  Average:      26.2290 ms           # /ping waits for a worker that is busy hashing
  50.00% in     26.2437 ms
  99.00% in     63.1279 ms

$ oha -z 10s -c 8 -m POST -d 'correct horse battery staple' http://127.0.0.1:3000/hash &
$ oha -n 200 -c 1 http://127.0.0.1:3000/ping
  Average:       0.0438 ms           # workers stay free; hashing runs on the blocking pool
  50.00% in      0.0124 ms
  99.00% in      1.7052 ms

Source: oha and tokio: spawn_blocking

The hashing throughput is about the same either way (the CPU is the limit); what changes is that spawn_blocking keeps the latency of everything else unaffected — a median of 26 ms against 0.01 ms for /ping here.
spawn_blocking runs the closure on a separate pool (up to 512 threads by default). For CPU-bound work, bound the concurrency with a Semaphore or use a dedicated Rayon pool, so a burst cannot start hundreds of threads competing for the same cores.
The closure runs outside the request’s span. Capture Span::current() before spawning and enter a child span inside, as above, so logs and traces from the blocking code stay attached to the request (Gangappa & Rangaraj, Mastering Distributed Observability in Rust, chapter 10).
Use tokio::fs, async database drivers (sqlx) and async HTTP clients; tokio::task::block_in_place is an escape hatch for code that cannot be moved, on the multi-threaded runtime only.
Allocator: the #[global_allocator] line swaps the system allocator for mimalloc (or tikv-jemallocator, which also offers heap profiling). Allocation-heavy services often gain throughput and suffer less fragmentation; on musl targets, whose allocator is slow, the gain is much larger. Measure it — the effect depends on the workload.

Finding blocked and starved tasks with tokio-console

tokio-console shows every task, how long it was busy versus idle, how often it was polled, and warns about tasks that hold a worker for too long:

$ cargo add console-subscriber            # then call console_subscriber::init() in main (development builds only)
$ RUSTFLAGS="--cfg tokio_unstable" cargo run
$ cargo install --locked tokio-console && tokio-console   # connects to 127.0.0.1:6669

Source: tokio-console: Instrumenting your program and console-subscriber 0.5.0

A task with a long busy time and few polls is blocking its worker. Many tasks with long scheduled times (waiting to run) mean the runtime is starved — usually by such a blocker.
tokio_unstable and the console layer add overhead; enable them in development or a staging build, not in the production binary.

Lock contention

Shared state behind a lock — an in-process cache, a rate-limit table, per-book counters — is correct but can become the bottleneck: every request that needs the lock waits for the one that holds it, so latency grows with concurrency while CPU usage stays low. The rules for axum handlers:

std::sync::Mutex is the right default for state that is locked briefly and never across an .await. Lock, update, drop the guard. Clippy’s await_holding_lock lint catches a guard held across an .await, which can block a worker thread and, with a !Send guard, fails to compile in a handler anyway.
Keep hold times short. Compute outside the lock (serialisation, hashing, formatting), and never do I/O or logging inside it. Contention is proportional to hold time times request rate.
tokio::sync::Mutex is only for the rare case where the lock really must be held across an .await (a single shared connection, for example); it is slower than the std mutex and lets tasks queue for a long time.
RwLock helps when reads vastly outnumber writes and the read section is not trivially short; for configuration that changes rarely, an arc-swap ArcSwap or a tokio::sync::watch channel gives readers no lock at all.
Shard a hot map into N independently locked parts chosen by key hash (or use a concurrent map such as dashmap, or moka for caches), so requests for different keys rarely meet.

The example measures the same workload — eight threads incrementing per-book view counters — with one mutex and with sixteen shards, and records how long the threads spent waiting to acquire a lock:

use std::collections::HashMap;
use std::hash::{BuildHasher, RandomState};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};

/// Per-book view counters behind ONE lock: every request serialises on it.
#[derive(Default)]
struct Global(Mutex<HashMap<u32, u64>>);

/// The same counters split over N independently locked shards.
struct Sharded {
    shards: Vec<Mutex<HashMap<u32, u64>>>,
    hasher: RandomState,
}

impl Sharded {
    fn new(n: usize) -> Self {
        Self { shards: (0..n).map(|_| Mutex::default()).collect(), hasher: RandomState::new() }
    }
    fn shard(&self, book: u32) -> &Mutex<HashMap<u32, u64>> {
        &self.shards[self.hasher.hash_one(book) as usize % self.shards.len()]
    }
}

trait Counter: Send + Sync + 'static {
    fn lock_for(&self, book: u32) -> &Mutex<HashMap<u32, u64>>;
}
impl Counter for Global {
    fn lock_for(&self, _book: u32) -> &Mutex<HashMap<u32, u64>> {
        &self.0
    }
}
impl Counter for Sharded {
    fn lock_for(&self, book: u32) -> &Mutex<HashMap<u32, u64>> {
        self.shard(book)
    }
}

/// Runs 8 threads x 200 000 increments; reports the wall time and the time spent waiting for locks.
fn run(name: &str, counter: Arc<dyn Counter>) {
    let started = Instant::now();
    let handles: Vec<_> = (0..8u32)
        .map(|t| {
            let counter = Arc::clone(&counter);
            std::thread::spawn(move || {
                let mut waited = Duration::ZERO;
                for i in 0..200_000u32 {
                    let book = (i * 7 + t) % 1_000;
                    let asked = Instant::now();
                    let mut map = counter.lock_for(book).lock().expect("poisoned");
                    waited += asked.elapsed(); // time spent acquiring = contention
                    *map.entry(book).or_default() += 1;
                } // guard dropped here: hold the lock for the update only
                waited
            })
        })
        .collect();
    let waited: Duration = handles.into_iter().map(|h| h.join().expect("thread")).sum();
    println!("{name:<10} {:>8.1?} wall, {waited:>8.1?} waiting for locks", started.elapsed());
}

fn main() {
    run("one mutex", Arc::new(Global::default()));
    run("16 shards", Arc::new(Sharded::new(16)));
}

Source: std::sync::Mutex and tokio: Which kind of mutex should you use?

$ cargo run --release      # 8 threads on a 4-core machine
one mutex   493.1ms wall,     2.5s waiting for locks
16 shards   186.7ms wall,     1.0s waiting for locks

Source: Clippy: await_holding_lock

To find contention in a running service:

Time the acquisition, as the example does, around the few hot locks, and export it as a histogram (see Metrics & SLOs); rising acquire times under load are the signature.
tokio-console lists tokio’s synchronisation primitives (tokio::sync::Mutex, RwLock, Semaphore) as resources, with the tasks waiting on each; a resource with a long queue is contended. std mutexes do not appear there.
In a CPU profile, contention shows up as time in futex system calls and parking functions rather than in your code — or, for an async lock, as low CPU usage with high latency.
Database row locks are the same problem one level down; see Databases with sqlx for FOR UPDATE SKIP LOCKED and lock_timeout.

CPU flamegraphs

$ cargo install flamegraph
$ cargo flamegraph --release --bin bookshelf-web    # run load with oha meanwhile, then Ctrl+C
$ samply record ./target/release/bookshelf-web      # alternative with an interactive Firefox Profiler view

Source: cargo-flamegraph and samply

Wide frames are where the CPU time goes. Typical findings in axum services: JSON serialisation of large responses, regex compilation inside handlers (compile once, store in a LazyLock), cloning large state instead of sharing it in an Arc, logging at debug level in production, and TLS handshakes when clients do not reuse connections. debug = "line-tables-only" in the release profile gives readable frames.

Continuous profiling

A flamegraph recorded on a laptop answers the question you thought to ask. Continuous profiling samples production all the time at a low rate (around 100 Hz, typically 1-2 % CPU), stores the profiles with labels such as service, version and region, and lets you compare "this morning" with "last week", or one release with the previous one. It is the fourth signal next to traces, metrics and logs. The options for a Rust service are third-party tools, not part of axum or tokio:

Tool How it collects Notes

Tool	How it collects	Notes
Parca (Polar Signals)	an eBPF agent on each node samples every process; no code changes	needs frame pointers or DWARF unwinding for good Rust stacks (`-C force-frame-pointers=yes` helps) and symbols kept in the binary
Grafana Pyroscope	the `pyroscope` crate (2.x, feature `backend-pprof-rs`) samples in-process and pushes profiles every 10 seconds; Grafana Alloy can also collect with eBPF	profiles can be tagged per request type and linked from traces in Grafana
pprof-rs (`pprof` crate)	an in-process sampling profiler (`SIGPROF`) that produces pprof protobufs or SVG flamegraphs	the building block of the Pyroscope Rust agent; also usable for on-demand profiles (below)

Parca (Polar Signals)

an eBPF agent on each node samples every process; no code changes

needs frame pointers or DWARF unwinding for good Rust stacks (-C force-frame-pointers=yes helps) and symbols kept in the binary

Grafana Pyroscope

the pyroscope crate (2.x, feature backend-pprof-rs) samples in-process and pushes profiles every 10 seconds; Grafana Alloy can also collect with eBPF

profiles can be tagged per request type and linked from traces in Grafana

pprof-rs (pprof crate)

an in-process sampling profiler (SIGPROF) that produces pprof protobufs or SVG flamegraphs

the building block of the Pyroscope Rust agent; also usable for on-demand profiles (below)

The pyroscope 2.1.1 agent is started before the server and stopped after it:

use pyroscope::{
    backend::{BackendConfig, PprofConfig, pprof_backend},
    pyroscope::PyroscopeAgentBuilder,
};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let url = std::env::var("PYROSCOPE_URL").unwrap_or_else(|_| "http://localhost:4040".into());
    let agent = PyroscopeAgentBuilder::new(
        url,
        "bookshelf-web",
        100, // samples per second
        "pyroscope-rs",
        env!("CARGO_PKG_VERSION"),
        pprof_backend(PprofConfig::default(), BackendConfig::default()),
    )
    .tags(vec![("env", "staging"), ("region", "eu-west-1")])
    .build()?
    .start()?;

    // ... build the runtime and run axum::serve(...) here; profiles are pushed every 10 s ...

    agent.stop()?.shutdown(); // flush the last profile on shutdown
    Ok(())
}

Source: pyroscope 2.1.1 and Grafana Pyroscope: Rust SDK

When continuous profiling is not available, an on-demand endpoint on the internal port gives the same insight for one incident: it profiles the live process for a few seconds and returns a flamegraph. pprof-rs 0.15 with the flamegraph feature:

use std::time::Duration;

use axum::{
    Router,
    extract::Query,
    http::{StatusCode, header},
    response::IntoResponse,
    routing::get,
};
use serde::Deserialize;

#[derive(Deserialize)]
struct ProfileParams {
    seconds: Option<u64>,
}

/// GET /debug/pprof/flamegraph?seconds=10 -- samples every thread of the process at 99 Hz
/// for a few seconds and returns an SVG flamegraph. Serve it on the internal port only.
async fn flamegraph(Query(params): Query<ProfileParams>) -> Result<impl IntoResponse, (StatusCode, String)> {
    let seconds = params.seconds.unwrap_or(10).clamp(1, 60);
    let svg = tokio::task::spawn_blocking(move || -> Result<Vec<u8>, pprof::Error> {
        let guard = pprof::ProfilerGuardBuilder::default()
            .frequency(99)
            .blocklist(&["libc", "libgcc", "pthread", "vdso"])
            .build()?;
        std::thread::sleep(Duration::from_secs(seconds)); // the request keeps being served meanwhile
        let mut svg = Vec::new();
        guard.report().build()?.flamegraph(&mut svg)?;
        Ok(svg)
    })
    .await
    .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?
    .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?;
    Ok(([(header::CONTENT_TYPE, "image/svg+xml")], svg))
}

#[tokio::main]
async fn main() {
    let internal = Router::new().route("/debug/pprof/flamegraph", get(flamegraph));
    let listener = tokio::net::TcpListener::bind("127.0.0.1:9000").await.unwrap();
    axum::serve(listener, internal).await.unwrap();
}

Source: pprof 0.15.0

$ curl -s -o bookshelf.svg 'localhost:9000/debug/pprof/flamegraph?seconds=10'   # while the service is under load

Source: pprof-rs repository

The profiler samples every thread, so the profile shows what the service did during those seconds, not just the profiling request; an idle process yields an empty flamegraph.
Profiling uses SIGPROF and stack unwinding in a signal handler; only one profile can run at a time, and the blocklist skips frames inside libc and the dynamic loader that cannot be unwound safely.
Keep the endpoint (and the Pyroscope agent’s credentials) away from the public port, like /metrics.

Database and pools

Pool size is a throughput limit, not a knob to turn up: every connection costs memory and CPU on the database. Start from the database’s capacity (see Databases with sqlx) and divide by the number of replicas.
Watch the time spent waiting for a connection (see Metrics & SLOs). If it grows, look for slow queries, N+1 patterns and long transactions before adding connections.
Set acquire_timeout so requests fail fast instead of queuing behind an exhausted pool, and bound the whole request with TimeoutLayer.

Responses, compression and connections

Bytes and zero-copy: Bytes is reference counted, so serving a cached image or a pre-rendered page from state clones a pointer, not the data (/cover above). Stream large files with ServeFile or a body stream instead of reading them into memory (see Static Files & SPAs).
Compression costs CPU on every response. Compress text (JSON, HTML) above a size threshold, skip already compressed formats (images, video), prefer pre-compressed static files (ServeDir::precompressed_br), and use a lower level for dynamic content (see CORS & Compression). Behind a CDN or a proxy that compresses, do not compress twice.
Keep-alive: hyper keeps HTTP/1.1 connections open by default; make sure clients and proxies reuse them, since a new TCP plus TLS handshake per request dominates the cost of small requests.
HTTP/2: with the http2 feature, axum::serve also accepts cleartext HTTP/2 (prior knowledge); HTTP/2 over TLS is negotiated by the TLS acceptor (see TLS & HTTPS). Many requests share one connection, which helps browsers and gRPC; behind a load balancer, HTTP/2 between proxy and service is mostly about fewer connections.
Per-connection settings: axum::serve exposes few knobs. For a header-read timeout, HTTP/2 stream limits or keep-alive intervals, drive connections with hyper-util’s auto::Builder as shown in Request Limits & Timeouts (axum’s serve-with-hyper example). serve::ListenerExt::tap_io sets TCP_NODELAY on accepted sockets.

Load testing

// k6 run books.js -- fails the run (non-zero exit) when the thresholds are missed, so it can gate a release
import http from 'k6/http';

export const options = {
  scenarios: { steady: { executor: 'constant-arrival-rate', rate: 500, timeUnit: '1s', duration: '2m',
                         preAllocatedVUs: 50 } },
  thresholds: { http_req_failed: ['rate<0.001'], http_req_duration: ['p(99)<300'] },
};

export default function () {
  http.get('http://staging.bookshelf.internal/books/9781718503106');
}

Source: k6 documentation and oha

A constant arrival rate keeps sending requests when the service slows down, which reveals queueing; a fixed number of looping virtual users backs off and hides it (coordinated omission).
The thresholds mirror the SLOs from Metrics & SLOs.

Where the books are outdated

Gangappa & Rangaraj (chapters 9-10) profile with flamegraphs, jemalloc and tokio-console; the approach is current, and tokio-console still needs the tokio_unstable cfg. Material that recommends panic = "abort" for smaller, faster binaries applies to CLIs, not servers.