Multimodal Models

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Examples on this page target Python 3.12, the anthropic Python SDK 1.8, transformers 5.17 and the openai/clip-vit-base-patch32 CLIP checkpoint, verified as current as of 2026-09-28. They reuse the DocsAssistant scenario introduced on Vector Databases & RAG: an assistant that answers questions about this site’s own AsciiDoc pages.

From pixels to tokens: ViT patches

Everything on the previous pages of this sub-section — tokens, attention, sampling — assumes the model’s input is already a sequence of discrete tokens. Text gets there through the tokenizers Tokens & Tokenization covers. An image has no natural "words" to split on, so the Vision Transformer (ViT) manufactures a sequence out of one instead:

  • the image is cut into a grid of fixed-size, non-overlapping patches (16×16 or 14×14 pixels are common choices);

  • each patch is flattened into a vector of raw pixel values and passed through a single learned linear projection, turning it into an embedding of the same width the transformer expects;

  • a learnable position embedding is added per patch (the grid position replaces the word order a text tokenizer would encode), plus, in the original ViT, a special classification token prepended to the sequence;

  • the resulting sequence of patch embeddings is fed to an ordinary transformer encoder — the same self-attention and feed-forward blocks Transformer Architecture describes, with no convolution anywhere.

In other words, ViT’s whole contribution is the recipe for turning an image into "tokens"; once that is done, an image is just another sequence a transformer already knows how to process. This is what makes ViT the common front end for both CLIP’s image encoder (below) and the image-input path of native multimodal LLMs.

CLIP: aligning images and text by contrastive training

CLIP (Contrastive Language-Image Pre-training) pairs a ViT-style image encoder with a text encoder and trains the two together so that an image and a caption describing it land close together in one shared vector space — the same contrastive-training idea Embeddings' CLIP section introduces for text-to-image retrieval. Concretely:

  • the image encoder (a ViT, or historically a ResNet) reduces an image to a single embedding;

  • the text encoder reduces a caption to an embedding of the same dimensionality;

  • both are trained on hundreds of millions of (image, caption) pairs scraped from the web, with a contrastive objective that pulls a genuine image-caption pair’s embeddings together and pushes every other pairing in the training batch apart;

  • because the two encoders are trained jointly into the same space, cosine similarity between any image embedding and any text embedding is meaningful, even for images and captions the model never saw together during training.

That last property is what makes CLIP a zero-shot classifier: turn each candidate class name into a short caption ("a photo of a cat"), embed every caption once, embed the image, and pick the caption with the highest similarity — no classification head and no labelled training data for the specific classes involved. The snippet below classifies a DocsAssistant page screenshot by which part of the site it belongs to, reusing the generic AutoModel / AutoProcessor classes that current transformers documentation recommends as the entry point for CLIP (the model-specific CLIPModel / CLIPProcessor classes still exist and work the same way):

import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

model = AutoModel.from_pretrained("openai/clip-vit-base-patch32")
processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32")

image = Image.open(DOCS_PAGE_SCREENSHOT_PATH)  # a screenshot of one of this site's own docs pages
labels = ["a screenshot of a vector database and RAG documentation page",
          "a screenshot of a Spring AI documentation page",
          "a screenshot of a cloud infrastructure documentation page"]

inputs = processor(text=labels, images=image, return_tensors="pt", padding=True)

with torch.inference_mode():
    outputs = model(**inputs)

probs = outputs.logits_per_image.softmax(dim=1)
best = probs.argmax(dim=1).item()
print(labels[best], float(probs[0][best]))

See the HF transformers CLIP model documentation.

Native multimodal LLMs: image and PDF input

CLIP produces embeddings for retrieval and classification; it does not converse. Current-generation LLMs such as Claude take images (and, built on the same vision pipeline, PDF pages) directly as request content and answer questions, describe, extract or reason about them in the same turn as ordinary text — no separate embedding step, no separate model to call. Internally, the vision-capable models still turn each image into a patch-based token sequence conceptually close to ViT’s, but that is an implementation detail the API hides: callers just add an image (or document) content block next to a text block.

The following request sends a base64-encoded PNG screenshot of one of this site’s own docs pages (part of the DocsAssistant corpus Document Ingestion & Chunking introduces) and asks Claude to describe it:

import anthropic

client = anthropic.Anthropic()

# base64-encoded PNG bytes of a docs page screenshot, produced by your own capture step
docs_screenshot_base64 = DOCS_PAGE_SCREENSHOT_BASE64

message = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": "image/png",
                        "data": docs_screenshot_base64,
                    },
                },
                {"type": "text", "text": "Describe this page."},
            ],
        }
    ],
)
print(message.content)

See the Anthropic vision documentation.

PDF input reuses the same request shape with a document content block instead of an image one, so a whole DocsAssistant source page can be handed over as a PDF export rather than pre-chunked text:

import base64
import anthropic

client = anthropic.Anthropic()

with open("hnsw-indexes.pdf", "rb") as f:
    pdf_data = base64.standard_b64encode(f.read()).decode("utf-8")

message = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "document",
                    "source": {
                        "type": "base64",
                        "media_type": "application/pdf",
                        "data": pdf_data,
                    },
                },
                {"type": "text", "text": "Summarize the key points of this document."},
            ],
        }
    ],
)
print(message.content)

See the Anthropic PDF support documentation. Both request shapes work the same way through the OpenAI SDK, using its own image_url / input_file content-part names; consult that SDK’s current reference when porting an example, since the two providers' field names differ even though the concept (an extra content block alongside text) is identical.

Audio in and out

The same "add a content block, get a richer answer" pattern extends to audio: current hosted APIs accept spoken audio as input (for transcription or direct question-answering over what was said) and can return synthesized speech as output, instead of requiring a separate speech-to-text step in front of the LLM and a separate text-to-speech step behind it. Building a full voice interaction loop — turn-taking, latency budgets, streaming audio in both directions — is enough of its own topic that it is out of scope here; the Voice Agents sub-section covers it end to end.

Image generation vs. image understanding

Every example on this page so far — ViT, CLIP, Claude’s vision input — understands an image that already exists: it classifies, embeds or describes pixels someone else produced. That is a different capability from generating an image from a text description, which uses an entirely different model family (diffusion models, or autoregressive image generators) trained to produce pixels, not to reason over them. A vision-input LLM such as Claude cannot generate or edit images itself; the Anthropic vision documentation states this limitation explicitly. Keep the two capabilities distinct: a DocsAssistant-style application that needs to read a screenshot or a diagram uses image understanding (this page); one that needs to produce a diagram or illustration needs a dedicated image-generation model instead, called as a separate step in the pipeline.

Related pages

References