Vision Models

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

This page was written against the model cards on the Hugging Face Hub and the transformers 5.17.0 documentation as checked on 2026-09-30. Licence and gating were read from the Hub API for each repository named here; they are properties of the repository, so a sibling size of the same family can differ (the Qwen2.5-VL family is the example below). Benchmark figures are deliberately not quoted: compare candidates on your own images, as recommended in Choosing Models & Licences.

In the running scenario of this sub-section, the DocsAssistant app must classify or describe a screenshot. That is one of four vision jobs, and each has a different family of models:

classify or match an image to labels (SigLIP 2, DINOv2 and DINOv3 features);
find and box objects (RT-DETRv2, D-FINE, YOLO);
cut out masks (SAM 2.1, SAM 3);
describe, read or answer questions about an image (Florence-2, SmolVLM, Qwen-VL, FastVLM).

The shortcut for choosing between them is the diagram below. The per-task pipeline() signatures are introduced in Transformers Pipelines; this page adds the model catalogue and the Auto*-class code where a pipeline does not exist.

flowchart TD Q{"What must the app do with the image?"} Q -->|"Assign a label or find similar images"| C["SigLIP 2 (labels at call time)
DINOv2 / DINOv3 (embeddings)"] Q -->|"Draw boxes around objects"| D{"Can the app ship AGPL-3.0?"} D -->|"No, or unsure"| D1["RT-DETRv2 / D-FINE (Apache-2.0)"] D -->|"Yes, source is released"| D2["Ultralytics YOLO (AGPL-3.0)"] Q -->|"Cut out object masks"| S["SAM 2.1 (Apache-2.0)
SAM 3 (gated, Meta licence)"] Q -->|"Describe, read text, answer questions"| V["Florence-2 (MIT)
SmolVLM (Apache-2.0)
Qwen3-VL (Apache-2.0)
FastVLM (research only)"]

Figure 1. Which family for which vision job

The catalogue

Licence values are the license field of each repository on 2026-09-30. "Gated" is the Hub’s gated flag: manual means an access request that the authors approve, False means anonymous download (see Hub Essentials for tokens and gating).

Table 1. Vision models checked for this page
Family	Example repository	Licence / gated	Task	Runtime and format notes
SigLIP 2	`google/siglip2-base-patch16-224`	Apache-2.0 / not gated	Zero-shot image classification, image-text retrieval, vision encoder for VLMs	`transformers` (safetensors); a `so400m` size also exists in the same collection
DINOv2	`facebook/dinov2-base`, `facebook/dinov2-small-imagenet1k-1-layer`	Apache-2.0 / not gated	Image features (embeddings); ImageNet classification head on some checkpoints	`transformers`; used for similarity search and as a backbone
DINOv3	`facebook/dinov3-vits16-pretrain-lvd1689m`	`other` (the DINOv3 licence) / gated, manual	Image features (embeddings)	`transformers`; access request and the Meta licence terms must be accepted before download
YOLO (Ultralytics)	`Ultralytics/YOLO11`	AGPL-3.0 / not gated	Detection, instance segmentation, classification, pose, oriented boxes, tracking	`ultralytics` package with `.pt` weights; the card points to an Enterprise licence for commercial use
RT-DETRv2	`PekingU/rtdetr_v2_r50vd`, `PekingU/rtdetr_v2_r18vd`	Apache-2.0 / not gated	Real-time object detection	`transformers` (`RTDetrV2ForObjectDetection`)
D-FINE	`ustc-community/dfine-small-coco`	Apache-2.0 / not gated	Real-time object detection	`transformers` (`DFineForObjectDetection`)
SAM 2.1	`facebook/sam2.1-hiera-small`	Apache-2.0 / not gated	Promptable image and video segmentation, automatic masks	`transformers` (`Sam2Model`, `mask-generation` pipeline) or the official `sam2` package
SAM 3	`facebook/sam3`	`other` (a custom licence file in the repository) / gated, manual	Segmentation prompted by text, boxes or exemplars	`transformers` (`Sam3Model`); the repository ships both safetensors and a `.pt` file
Florence-2	`florence-community/Florence-2-base`, `microsoft/Florence-2-base`	MIT / not gated	Captioning, detection, grounding, OCR through task prompts	`transformers` (`Florence2ForConditionalGeneration`); card lists 0.23B and 0.77B sizes
SmolVLM / SmolVLM2	`HuggingFaceTB/SmolVLM-256M-Instruct`, `HuggingFaceTB/SmolVLM2-2.2B-Instruct`	Apache-2.0 / not gated	Image (and, for SmolVLM2, video) question answering and captioning	`transformers` (`AutoModelForImageTextToText`); the SmolVLM2 card lists 256M, 500M and 2.2B sizes
Qwen-VL	`Qwen/Qwen3-VL-2B-Instruct`, `Qwen/Qwen3-VL-8B-Instruct`	Apache-2.0 / not gated	Image and video chat, document and screenshot reading	`transformers` (`Qwen3VLForConditionalGeneration`)
Qwen2.5-VL (older generation)	`Qwen/Qwen2.5-VL-3B-Instruct` and `Qwen/Qwen2.5-VL-7B-Instruct`	3B: `qwen-research`; 7B: Apache-2.0 / not gated	Image and video chat	Same family, different licence per size: read each card
FastVLM	`apple/FastVLM-0.5B`, `apple/FastVLM-1.5B`	`apple-amlr` (Apple Machine Learning Research Model licence) / not gated	Efficient high-resolution image understanding	Card shows `trust_remote_code` loading or Apple’s own code; see the licence paragraph below

Reference: Hugging Face — Gated models and Hugging Face — Licenses.

Classification and embeddings: SigLIP 2, DINOv2 and DINOv3

SigLIP 2 is a contrastive image-text model from Google: the text encoder turns your label strings into vectors at call time, so the classifier changes when the label list changes and needs no retraining. The card lists zero-shot classification and image-text retrieval as intended uses, and recommends the vision tower as an encoder for other models. DINOv2 and DINOv3 are self-supervised vision backbones without a text side: you use the image embedding for nearest-neighbour search, clustering or a small classifier trained on top, rather than asking for labels. DINOv2 checkpoints are Apache-2.0 and open; the DINOv3 repository checked here is gated with manual approval and points at a separate DINOv3 licence, so a build pipeline or a CI job must authenticate with $HF_TOKEN and the account must have been approved first.

Zero-shot classification of a screenshot with SigLIP 2 (Apache-2.0)

from transformers import pipeline

clf = pipeline(
    task="zero-shot-image-classification",
    model="google/siglip2-base-patch16-224",
)
result = clf(
    "screenshot.png",
    candidate_labels=["error dialog", "code editor", "settings page", "chart"],
)
print(result[0])   # highest score first

Reference: Transformers — SigLIP2.

Image embeddings for similarity search with DINOv2 (Apache-2.0)

import torch
from PIL import Image
from transformers import AutoImageProcessor, AutoModel

processor = AutoImageProcessor.from_pretrained("facebook/dinov2-small")
model = AutoModel.from_pretrained("facebook/dinov2-small", device_map="auto")

def embed(path: str) -> torch.Tensor:
    inputs = processor(images=Image.open(path).convert("RGB"), return_tensors="pt").to(model.device)
    with torch.inference_mode():
        out = model(**inputs)
    vec = out.pooler_output[0]                    # one vector per image
    return torch.nn.functional.normalize(vec, dim=0)

similarity = float(embed("a.png") @ embed("b.png"))   # cosine similarity of unit vectors
print(similarity)

The same two classes load DINOv3 after the access request is approved. Store the vectors in a vector index; see Generating Embeddings in Practice for the text counterpart.

Reference: Transformers — DINOv2 and Transformers — DINOv3.

Detection: YOLO (AGPL-3.0) versus RT-DETRv2 and D-FINE

Ultralytics YOLO is the best-known real-time detector and its tooling covers detection, segmentation, pose and tracking in one package. The model repository on the Hub is licensed AGPL-3.0, and the card directs commercial users to an Enterprise licence. AGPL-3.0 is a network copyleft licence: offering a modified or combined work to users over a network triggers the obligation to offer the corresponding source, so an app or backend that embeds the weights and code must either be released under compatible terms or be covered by a commercial licence. That is a legal question to settle with your counsel before shipping, not a property you can configure away; the licence classes are summarised in Choosing Models & Licences.

RT-DETRv2 and D-FINE are real-time DETR-style detectors with Apache-2.0 weights and native transformers classes, so a closed-source app can ship them after the usual licence check of the exact checkpoint (the COCO-trained checkpoints inherit the COCO label set). They need no extra package, export to ONNX through the usual route (see Conversion and Quantization) and are the default suggestion when AGPL is a problem. The pipeline form is shown in Transformers Pipelines; the explicit form below shows how boxes are rescaled to the original image.

Detecting objects with RT-DETRv2 and the Auto classes (Apache-2.0)

import torch
from PIL import Image
from transformers import AutoImageProcessor, AutoModelForObjectDetection

model_id = "PekingU/rtdetr_v2_r50vd"
processor = AutoImageProcessor.from_pretrained(model_id)
model = AutoModelForObjectDetection.from_pretrained(model_id)

image = Image.open("street.jpg").convert("RGB")
inputs = processor(images=image, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

results = processor.post_process_object_detection(
    outputs,
    target_sizes=torch.tensor([(image.height, image.width)]),
    threshold=0.5,
)[0]
for score, label_id, box in zip(results["scores"], results["labels"], results["boxes"]):
    print(model.config.id2label[label_id.item()], round(score.item(), 2), [round(v, 1) for v in box.tolist()])

Reference: Transformers — RT-DETRv2 and Transformers — D-FINE. The D-FINE card loads the same way with DFineForObjectDetection and ustc-community/dfine-small-coco.

Running YOLO with the Ultralytics package (AGPL-3.0: check your obligations first)

from ultralytics import YOLO

model = YOLO("yolo11n.pt")          # downloads official weights on first use
results = model("street.jpg")       # list of Results objects
for r in results:
    print(r.boxes.xyxy, r.boxes.conf, r.boxes.cls)

Weights are .pt files, which are pickle-based: download them only from the official Ultralytics sources, as explained in Model Formats.

Reference: Ultralytics — Documentation and Ultralytics — Licensing.

Segmentation: SAM 2.1 and SAM 3

The Segment Anything models produce masks. SAM 2.1 (Apache-2.0, open) is promptable: you give a point or a box and it returns candidate masks, it can also generate masks for everything in the image, and it tracks objects through video. SAM 3 adds prompting by concept: a short text phrase such as "ear" or an exemplar box makes it return every matching instance with masks, boxes and scores. The SAM 3 repository is gated with manual approval and carries a custom licence file; the access form asks for personal details that Meta processes under its privacy policy, so treat it like any other click-through agreement and have it accepted by the person or organisation that owns the product before you build on it.

Prompting SAM 2.1 with a point click (Apache-2.0)

import torch
from PIL import Image
from transformers import Sam2Model, Sam2Processor

device = "cuda" if torch.cuda.is_available() else "cpu"
model = Sam2Model.from_pretrained("facebook/sam2.1-hiera-small").to(device)
processor = Sam2Processor.from_pretrained("facebook/sam2.1-hiera-small")

image = Image.open("truck.jpg").convert("RGB")
input_points = [[[[500, 375]]]]   # image, object, point, (x, y)
input_labels = [[[1]]]            # 1 = positive click, 0 = negative click

inputs = processor(
    images=image, input_points=input_points, input_labels=input_labels, return_tensors="pt"
).to(device)
with torch.no_grad():
    outputs = model(**inputs)

masks = processor.post_process_masks(outputs.pred_masks.cpu(), inputs["original_sizes"])[0]
print(masks.shape)   # several candidate masks for the point, ranked by quality score

The mask-generation pipeline, shown in Transformers Pipelines, produces masks for the whole image without prompts.

Reference: Transformers — SAM2 and SAM 2.1 model card.

Text-prompted segmentation with SAM 3 (gated, run hf auth login or export $HF_TOKEN first)

import torch
from PIL import Image
from transformers import Sam3Model, Sam3Processor

model = Sam3Model.from_pretrained("facebook/sam3", device_map="auto")
processor = Sam3Processor.from_pretrained("facebook/sam3")

image = Image.open("photo.jpg").convert("RGB")
inputs = processor(images=image, text="ear", return_tensors="pt").to(model.device)
with torch.no_grad():
    outputs = model(**inputs)

results = processor.post_process_instance_segmentation(
    outputs,
    threshold=0.5,
    mask_threshold=0.5,
    target_sizes=inputs.get("original_sizes").tolist(),
)[0]
print(len(results["masks"]), "instances")   # also results["boxes"] and results["scores"]

Reference: Transformers — SAM3. The model and processor class names come from that page; check it against your installed transformers version.

Captioning and visual question answering: VLMs

A vision-language model (VLM) takes an image and a prompt and returns text, so one model can caption, read text, answer questions or extract fields. Four families matter for on-device use, and they differ mainly in size and licence.

Florence-2 (MIT) is not a chat model: a short task prompt such as <CAPTION>, <OD> or <OCR> selects the job and post_process_generation parses the answer into labels and boxes. Its card lists 0.23B and 0.77B variants, which makes it a good fit for small devices and for structured output such as boxes from text. SmolVLM and SmolVLM2 (Apache-2.0) are Hugging Face’s small chat VLMs; the SmolVLM2 card lists 256M, 500M and 2.2B sizes and adds video input, and the card positions the smaller models for on-device use. Qwen-VL (the Qwen3-VL checkpoints checked here are Apache-2.0) is the larger, more capable family, strong at reading screenshots and documents; the older Qwen2.5-VL family mixes licences by size, so check each card.

Captioning a screenshot with SmolVLM2 through the chat template (Apache-2.0)

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "HuggingFaceTB/SmolVLM2-2.2B-Instruct"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://example.com/screenshot.png"},
            {"type": "text", "text": "Describe this screenshot in one sentence."},
        ],
    }
]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt"
).to(model.device, dtype=torch.bfloat16)

generated = model.generate(**inputs, do_sample=False, max_new_tokens=64)
print(processor.batch_decode(generated, skip_special_tokens=True)[0])

The decoded text contains the whole conversation (prompt plus answer); slice the generated ids after the input length, as the Qwen example below does, to keep only the answer. The card recommends flash_attention_2 on supported GPUs; on other hardware leave the default attention implementation. The SmolVLM-256M form is also shown as a one-call pipeline in Transformers Pipelines.

Reference: Transformers — SmolVLM and SmolVLM2 model card.

Structured tasks with Florence-2: detection through a task prompt (MIT)

import torch
from PIL import Image
from transformers import AutoProcessor, Florence2ForConditionalGeneration

model_id = "florence-community/Florence-2-base"
model = Florence2ForConditionalGeneration.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

image = Image.open("screenshot.png").convert("RGB")
task = "<OD>"                          # also <CAPTION>, <DETAILED_CAPTION>, <OCR>
inputs = processor(text=task, images=image, return_tensors="pt").to(model.device, torch.bfloat16)

ids = model.generate(**inputs, max_new_tokens=1024, num_beams=3)
text = processor.batch_decode(ids, skip_special_tokens=False)[0]
print(processor.post_process_generation(text, task=task, image_size=image.size))
# {'<OD>': {'bboxes': [[x1, y1, x2, y2], ...], 'labels': [...]}}

Reference: Transformers — Florence-2 and Florence-2 model card. The microsoft/ repositories hold the original checkpoints, which older instructions load with trust_remote_code=True; the florence-community/ copy is the transformers-native conversion and needs no remote code.

Reading a screenshot with Qwen3-VL-2B (Apache-2.0)

from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

model_id = "Qwen/Qwen3-VL-2B-Instruct"
model = Qwen3VLForConditionalGeneration.from_pretrained(model_id, dtype="auto", device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://example.com/screenshot.png"},
            {"type": "text", "text": "What error does this dialog show?"},
        ],
    }
]
inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt"
).to(model.device)

generated = model.generate(**inputs, max_new_tokens=128)
trimmed = [out[len(inp):] for inp, out in zip(inputs.input_ids, generated)]
print(processor.batch_decode(trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0])

Reference: Transformers — Qwen3-VL and Qwen3-VL-2B-Instruct model card.

The same models run through the local servers described in the Running LLMs Locally sub-section once a runtime supports their architecture, and transformers serve accepts image inputs on /v1/chat/completions (see Transformers Pipelines). Whether a given VLM has a GGUF, ONNX or Core ML export depends on the architecture; see Model Formats. To adapt a vision model to your own images, see Fine-Tuning with PEFT and TRL.

FastVLM and Apple’s licence terms

FastVLM is Apple’s CVPR 2025 VLM whose FastViTHD encoder aims at fewer visual tokens and a short time to first token on high-resolution images, which is why it is interesting for phones. The Hub repositories checked here (apple/FastVLM-0.5B, apple/FastVLM-1.5B) are not gated, but their license field is apple-amlr, the Apple Machine Learning Research Model licence, and the repository’s LICENSE file states its scope in plain terms:

the model is released solely for scientific research;
the grant is limited to Research Purposes, defined as non-commercial scientific research and academic development, and explicitly excludes commercial exploitation, product development and use in any commercial product or service;
models derived by fine-tuning or modification are limited to the same research purposes;
redistribution requires a copy of the agreement, an attribution notice and clear labelling of derivatives;
the licence is revocable and is provided without warranty.

Read in that way, FastVLM is for evaluation and research, and a shipping app cannot rely on it without a separate agreement from Apple. Being free to download is not the same as being free to ship; the same reading applies to any model whose licence is other or a named custom licence (DINOv3 and SAM 3 above). The model card’s own loading example calls trust_remote_code=True, which executes Python from the repository on your machine: pin the commit and read the code first, or prefer Apple’s official code path described on the card.

Reference: FastVLM-0.5B model card and LICENSE file and FastVLM paper.

Choosing for the DocsAssistant screenshot task

For the running scenario, start with SigLIP 2 for "which kind of screen is this?" because the label list is yours and the model is Apache-2.0; use SmolVLM (or Qwen3-VL-2B when the device has memory to spare) when the app needs a sentence or the text in the image. Both are open, need no access request and run through transformers on a backend; the platform pages (Apple Platforms, Android, Windows & .NET and Web & Cross-Platform) show how each runtime reaches them. Add RT-DETRv2 only if you need boxes, and keep YOLO out of a closed-source build unless you hold a commercial licence. Before shipping any model, run the licence checklist from Choosing Models & Licences and record the repository, the commit hash and the licence text you read.