Media Optimization

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

See the section bibliography for the reference material consulted while preparing these pages.

Users upload photos and videos in whatever format their device produced, and a platform has to serve each client a file it can decode, at a size it needs, with as few bytes as possible. This sub-section describes how the backend does that: turn every image into a set of formats and sizes (with a BlurHash placeholder), turn every video into a ladder of renditions that stream adaptively, produce covers and thumbnails, and do all of it in background workers rather than in the HTTP request. It is current as of 2026-10-07. Each page states the versions it was checked against and flags unmaintained or restrictively licensed libraries.

flowchart LR U["Upload"] --> V["Validate, strip metadata"] V --> I["Image: variants in AVIF, WebP, JPEG"] V --> VD["Video: renditions, HLS and DASH"] I --> B["BlurHash and thumbnails"] VD --> CV["Cover frame, sprite sheet, BlurHash"] B --> S["Object storage"] CV --> S S --> CDN["CDN"] CDN --> CL["Web and app clients"]

Pages

Page What it covers

Image Formats and Encoding

JPEG, PNG, WebP, AVIF, JPEG XL and HEIC compared. Quality settings, EXIF orientation, colour profiles, and the libvips, cwebp and avifenc commands.

Responsive Images and Thumbnails

Width breakpoints, fit and crop modes, storage keys, eager versus on-the-fly generation, cache headers, and what the API must return for srcset and picture.

BlurHash: The Algorithm

The equations, the string layout, decoding and punch, a worked example, ThumbHash, and the implementations per language.

Video Encoding Fundamentals

Codecs, containers, rate control, GOP, audio, FFmpeg commands, and the bitrate ladder from the Apple HLS Authoring Specification.

Video Covers and Thumbnails

Seeking, the thumbnail filter, scene detection, sprite sheets, rotation, and BlurHash on the cover.

Video Streaming: HLS, DASH and CMAF

Progressive versus adaptive streaming, playlists, segment length, FFmpeg and packagers, CDN delivery, signed cookies and DRM.

Java Image and Video Libraries

Thumbnailator, imgscalr (legacy), TwelveMonkeys, Scrimage, libvips bindings, blurhash-java, Jaffree and JAVE2.

Spring Boot and Quarkus Pipeline

A media service with pre-signed uploads, a queue, workers, a status endpoint and Spring Batch backfills, in both frameworks.

Python Image and Video Libraries

Pillow, pillow-heif, pyvips, BlurHash, ffmpeg-python, MoviePy 2, subprocess and a FastAPI upload endpoint.

Node.js and Rust Libraries

sharp, Jimp, blurhash, fluent-ffmpeg (legacy), a NestJS and BullMQ example, and Rust with axum, the image crate and tokio.

.NET Image and Video Libraries

ImageSharp and its license, SkiaSharp, NetVips, BlurHash, FFMpegCore, Xabe.FFmpeg and an ASP.NET Core upload endpoint.

Upload Pipelines, Queues and Jobs

The reference architecture: pre-signed uploads, events, queues, idempotent workers, retries, dead-letter queues and status.

Security and Operations

Upload validation, decompression bombs, metadata stripping, sandboxing FFmpeg, patching, cost, lifecycle and monitoring.

Cheat Sheet (PDF)

A one-page summary of formats, BlurHash, FFmpeg commands, the ladder and a library picker.

Reading paths

Existing pages this section builds on

Managed alternatives in the cloud

Self-hosted FFmpeg and image workers are one choice. Each cloud offers a managed one, described in its own section:

  • AWS: Elemental MediaConvert, S3 and Lambda image processing, CloudFront delivery and a Video on Demand pipeline.

  • Google Cloud: the Transcoder API, Cloud Run jobs, Media CDN and storage-triggered resizing.

  • Azure: Media Services was retired, so self-hosted FFmpeg in Container Apps jobs, Event Grid with Functions, and Front Door.

The frontend counterparts are Media Optimization for the web and Media Optimization in apps.

Bibliography

Every source below is the official site, repository or documentation page of the library, service or standard it supports. Links were checked from an authoring environment with restricted outbound access. Entries marked (unverified) could not be opened from there (the site refused or blocked the request), so open them and confirm the content, version and status before you rely on them. Entries without the mark were opened successfully, or, for package pages, confirmed through the package registry.

BlurHash and ThumbHash

Image formats, encoders and libvips

Java

Python

Node.js

Rust

.NET

Video, FFmpeg and streaming

Security

Other sources