Transformers Pipelines

A pipeline chains three stages: pre-processing (tokenizer, feature extractor or image processor turns raw text, audio or pixels into tensors), the model forward pass (or generate() for generative models) and post-processing (logits become labels, token IDs become text). You choose the stage set with a task identifier; every task has a default model, but you should always pass model= explicitly so the behaviour is reproducible.

Inputs can be a file path, a URL or an in-memory object; a list of inputs returns a list of results. Task-specific keyword arguments (top_k, threshold, candidate_labels, return_timestamps) go on the call, and arguments for generate() go in generate_kwargs. Everything in this section is downloaded from the Hub on first use, so the repository must be one you trust: pipelines load safetensors when they exist, and only models that ship custom code need trust_remote_code=True, which runs the author’s Python on your machine (see Model Formats).

Reference: Transformers — Pipelines.

The automatic-speech-recognition task turns audio into text. Whisper-family models are the usual choice; the pipeline resamples the audio to the model’s rate (ffmpeg must be installed to decode compressed files) and can split long recordings into chunks.

For long files chunk_length_s=30 splits the audio, but the library documents chunking with Whisper as experimental and recommends the model’s own long-form generate() route for best accuracy (see the AutoModelForSpeechSeq2Seq example below); return_timestamps adds a chunks list. A smaller model such as openai/whisper-tiny.en (Apache-2.0) is enough for a first test. Other speech models and the faster runtimes are compared on Speech Models: STT & TTS.

Reference: Transformers — AutomaticSpeechRecognitionPipeline.

The text-to-audio task (alias text-to-speech) generates a waveform from text. The pipeline returns a dictionary with the samples (audio, a NumPy float array, squeezed to one dimension for mono output) and their sampling_rate; you write or play it yourself.

Not every TTS model is wired into the pipeline: check the model card for a transformers snippet. Kokoro-82M, for example, has its own kokoro package. Licences differ widely (facebook/mms-tts-eng is CC-BY-NC-4.0), so check the card before shipping. Generation options (do_sample, temperature, max_new_tokens) go in generate_kwargs.

Reference: Transformers — TextToAudioPipeline.

The image-classification task assigns one of the model’s fixed labels to an image and returns the best top_k as label and score pairs. Scores are softmax probabilities by default (function_to_apply changes that).

The label set is whatever the checkpoint was trained on; for arbitrary labels use the zero-shot variant.

Reference: Transformers — ImageClassificationPipeline.

The zero-shot-image-classification task compares an image with text labels you supply at call time, using a contrastive model such as CLIP or SigLIP 2. There is no retraining: changing the label list changes the classifier.

Phrase labels the way the model’s card suggests (for example "a photo of a …​" through hypothesis_template), because prompt wording changes scores.

Reference: Transformers — ZeroShotImageClassificationPipeline.

The object-detection task returns a list of score, label and box (xmin, ymin, xmax, ymax in pixels) entries for each object above threshold. RT-DETR v2 is an Apache-2.0 real-time detector.

The classic facebook/detr-resnet-50 (Apache-2.0) works the same way. The pipeline also offers zero-shot-object-detection for detection with free-text labels.

Reference: Transformers — ObjectDetectionPipeline.

The image-segmentation task returns a list of segments, each with a label, an optional score and a mask (a PIL image the size of the input). Depending on the model the segments are semantic, instance or panoptic; models that support several modes accept subtask=.

Check the licence of each checkpoint: several segmentation models on the Hub (for example some SegFormer and Mask2Former checkpoints) carry a custom licence tag rather than Apache-2.0.

Reference: Transformers — ImageSegmentationPipeline.

The mask-generation task runs a promptless "segment everything" pass with a SAM-style model: it proposes masks for all objects it finds, without class labels. It is the pipeline behind automatic masks in SAM demos.

points_per_batch controls how many prompt points are processed at once and therefore peak memory. Prompted segmentation (points or boxes) uses the model class and processor directly; the catalogue of SAM generations is in Vision Models.

Reference: Transformers — MaskGenerationPipeline.

The image-text-to-text task feeds an image and a prompt to a vision-language model and returns generated text. It takes chat-style messages whose content is a list of typed parts; in v5 the image must be embedded in the message content, not passed as a separate argument next to a chat. In v5 this task replaces the removed image-to-text and visual-question-answering pipelines (see the migration section).

Keyword names for the prompt have changed between minor releases, so confirm the call signature on the pipeline’s API page for your installed version.

Reference: Transformers — ImageTextToTextPipeline.

The text-generation task continues a prompt with a causal language model. Given chat messages it applies the model’s chat template automatically, and the result contains the whole conversation, with the assistant’s reply last.

With a plain string prompt, return_full_text=False returns only the new text, and num_return_sequences with do_sample=True produces several samples. Some models, such as Qwen3, emit a reasoning block first; see the model card for how to switch it off. Gemma checkpoints (for example google/gemma-3-1b-it) are gated: accept the licence on the Hub and authenticate with $HF_TOKEN or hf auth login first, as described in Hub Essentials. For serving a model to many users, pipelines are the wrong tool; see vLLM and SGLang.

Reference: Transformers — TextGenerationPipeline.

The v5 pipeline reference lists, among others, audio-classification, depth-estimation, feature-extraction, fill-mask, text-classification, token-classification, zero-shot-classification and video-classification. The v5 migration guide states that the question-answering, summarization, translation, image-to-text, visual-question-answering and image-to-image pipelines were removed. For those tasks load the model with the matching Auto* class (or use a chat model through text-generation or image-text-to-text). The pipeline documentation still mentions a few of these names in places, so when a task identifier raises an error, the list on the API page for your installed version is authoritative.

Reference: Transformers — v5 migration guide.

A pipeline hides three objects that you can also create yourself. The Auto* classes read config.json in the repository and instantiate the right architecture, so your code does not name it. AutoModel returns the bare network (hidden states); AutoModelFor…​ classes add a task head (AutoModelForCausalLM, AutoModelForSequenceClassification, AutoModelForSpeechSeq2Seq, AutoModelForImageClassification, AutoModelForImageTextToText). Pre-processing objects come from AutoTokenizer for text, AutoImageProcessor and AutoFeatureExtractor for pixels and audio, and AutoProcessor, which bundles whichever of them a multimodal model needs behind one object.

For vision-language and speech models the same pattern uses AutoProcessor, which turns images, audio and text into the model’s inputs and decodes its outputs.

Use this route when you need custom decoding, your own batching or to hand the model to another library. Save what you load with save_pretrained() (safetensors) to keep an offline copy.

Reference: Transformers — Loading models and Transformers — Auto Classes.

Device. When device is unset, a pipeline places the model on the first available accelerator (CUDA, Apple silicon MPS, XPU) and falls back to the CPU; pass device="cpu" to force the CPU, device=0 for the first GPU or device="mps" for Apple silicon. device_map="auto" hands placement to Accelerate, which fills the fastest device first and spills the rest to CPU RAM and disk; it needs accelerate installed and is the option for models larger than one GPU. Do not combine device and device_map.

dtype. The dtype argument sets the PyTorch data type of the weights. By default from_pretrained() uses the value stored in the model’s config.json and otherwise the dtype of the first floating-point weight in the checkpoint, so a checkpoint saved in bfloat16 is no longer silently upcast to float32. Pass dtype=torch.bfloat16 where the hardware supports it (more range) or torch.float16 to halve memory against float32, and dtype="auto" to ask for the stored dtype explicitly. The former name torch_dtype is still accepted as a legacy alias; write dtype in new code.

Batching is off by default; batch_size=N may help on a GPU with regular input lengths and rarely helps on a CPU or when latency matters, so measure. Quantization options are compared in Quantization for Inference, and memory estimates in Hardware Sizing.

Reference: Transformers — Pipeline device and Transformers — Model data type.

transformers serve starts a local HTTP server that speaks the OpenAI API, so existing OpenAI SDK code, curl and tools that accept a custom base URL can talk to a Hub model loaded through transformers. It is meant for evaluation, experiments and moderate load, not for large-scale production; the documentation points to vLLM or SGLang with transformers as the backend for that. Install the extra, then start it; the default address is http://localhost:8000.

The server exposes /v1/chat/completions (text, image, audio and video inputs), /v1/completions, /v1/responses, /v1/audio/transcriptions (multipart upload with a speech model such as Whisper), /v1/models and /load_model, which streams loading progress. Without a model argument, models load on demand and are unloaded after 300 seconds of inactivity (--model-timeout changes it, -1 disables unloading). The local server has no authentication and binds to localhost by default; if you expose it, put it behind a reverse proxy with authentication or an SSH tunnel. Compare it with the other local servers in Ollama and vLLM and SGLang; for Spring applications the endpoint plugs into Spring AI through its OpenAI integration.

Reference: Transformers — Serve CLI.

Code written for the books and tutorials of the v4 era often needs small edits. The changes that matter most for the code on this page are below; the official guide lists the rest.

Pin transformers in requirements.txt (for example transformers>=5,<6) and run your tests after each minor upgrade: pipeline signatures and task lists still change between v5 minors.

Reference: Transformers — v5 migration guide.