Small LLMs & Embeddings

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

This page was written on 2026-09-30 against the Hugging Face Hub model cards and repository metadata as checked that day (huggingface_hub 2.0.0, transformers 5.17.0, sentence-transformers 6.1.0, llama-cpp-python 0.3.35, mlx-lm 0.31.3, LiteRT-LM 0.17.1). The Hub changes daily: new quantized repositories appear and licences change between releases, so treat every table row as a snapshot and re-read the model card of the exact repository you ship. The DocsAssistant goes offline scenario of this sub-section needs two kinds of small model: a chat model that answers the transcribed question on the device, and an embedding model that finds the relevant documentation passages first. Both are covered here. No benchmark numbers are quoted; evaluate the two or three candidates on your own prompts.

The larger open-weight families (DeepSeek, Mistral, gpt-oss, Llama 4 and the bigger Qwen and Gemma sizes) are in Open-Weight Models; this page is the on-device end of that table. For the embedding side of a retrieval pipeline — chunking, similarity metrics and vector stores — see Generating Embeddings in Practice; here only the models and their on-device formats are added.

What "small" means on a phone or in an app

A phone or an app shares memory with the operating system and everything else the user has open, so the practical ceiling is a few gigabytes of weights. The memory needed just to hold the weights of a model with parameters stored at bits per weight is

For a 4-billion-parameter model that is about 8 GB at 16 bits, 4 GB at 8 bits and 2 GB at 4 bits, before the KV-cache, activations and runtime overhead are added. That is why the models below are almost always run quantized, and why the formats on the Hub matter: see Quantization for Inference for the quantization types and Hardware Sizing for the full memory budget.

Two catalogue terms need explaining first:

"E2B" / "E4B" (effective parameters). Gemma 3n and Gemma 4 use Per-Layer Embeddings: each decoder layer has its own small embedding table per token. The tables are large but are only looked up, so the compute and working memory correspond to the smaller "effective" count. The repository still stores the larger total, which is what you download and what must fit on disk. The gemma-4-E2B-it card states about 2.3 billion effective parameters out of about 5.1 billion including embeddings; the Hub API reports the same order of magnitude for the safetensors total.
Instruction-tuned vs. base. Repositories ending in -it, -Instruct or without a suffix (Qwen3.5) are chat-tuned; -Base or no suffix on Gemma 4 (google/gemma-4-E2B) are pre-trained checkpoints meant for fine-tuning.

The small LLMs

All rows were read from the Hub API (license, gated, safetensors parameter count) on 2026-09-30. "Manual" gating means you must accept the licence on the model page and a maintainer approves the request; "auto" approves immediately after you accept; see Hub Essentials for tokens and gated downloads.

Table 1. The small chat models, with licence and gating
Model (repository)	Safetensors size	Licence, gating	Notes from the model card
Gemma 4 E2B (`google/gemma-4-E2B-it`)	about 5.1 B total, 2.3 B effective	Apache-2.0, not gated	Text, image and audio input, text output; 128K context; configurable thinking mode; native system role and function calling
Gemma 4 E4B (`google/gemma-4-E4B-it`)	about 8 B total, 4.5 B effective	Apache-2.0, not gated	Same design as E2B, larger; the step up when E2B answers are not good enough
Gemma 3n E2B / E4B (`google/gemma-3n-E2B-it`, `google/gemma-3n-E4B-it`)	about 5.4 B / 7.8 B total	Gemma terms, manual	The previous on-device generation; superseded by Gemma 4 but still the base of many mobile demos
Gemma 3 1B / 4B (`google/gemma-3-1b-it`, `google/gemma-3-4b-it`)	1.0 B / 4.3 B	Gemma terms, manual	1B is text only and the smallest of the family; 4B adds image input
Phi-4-mini (`microsoft/Phi-4-mini-instruct`)	about 3.8 B	MIT, not gated	Text only, 128K context (see the card for the languages it supports)
Qwen3 0.6B / 1.7B / 4B (`Qwen/Qwen3-0.6B`, `Qwen/Qwen3-1.7B`, `Qwen/Qwen3-4B`)	0.75 B / 2.0 B / 4.0 B	Apache-2.0, not gated	Hybrid thinking and non-thinking modes; tool calling; the 0.6B is the smallest practical chat model here
Qwen3.5 0.8B / 2B / 4B (`Qwen/Qwen3.5-0.8B`, `Qwen/Qwen3.5-2B`, `Qwen/Qwen3.5-4B`)	0.87 B / 2.3 B / 4.7 B	Apache-2.0, not gated	Vision-language models with a vision encoder; the card states 201 languages and dialects
SmolLM3 (`HuggingFaceTB/SmolLM3-3B`)	about 3.1 B	Apache-2.0, not gated	Fully open (weights and training recipe); dual reasoning mode via `/think` and `/no_think` or `enable_thinking`; six languages; 64k trained context extensible to 128k with YaRN
Llama 3.2 1B / 3B Instruct (`meta-llama/Llama-3.2-1B-Instruct`, `meta-llama/Llama-3.2-3B-Instruct`)	1.2 B / 3.2 B	Llama 3.2 community licence, manual	Text only; the only family here with Meta-published quantized variants (next section)

Notes on choosing between them:

Licence first. Gemma 4, Phi-4-mini, Qwen3, Qwen3.5 and SmolLM3 are permissive (Apache-2.0 or MIT) and ungated, so an app can download them at first launch without asking the user to log in. Gemma 3n, Gemma 3 and Llama 3.2 carry custom terms and are gated: your app cannot fetch them anonymously, and the terms (for example acceptable-use policies) travel with redistribution. The Gemma 4 instruct repository checked here is Apache-2.0 while the Gemma 3 and 3n repositories are not, so "Gemma" does not tell you the licence; the repository does. The general licence classes are explained on Choosing Models & Licences.
Multimodal or text only. Gemma 4 E2B / E4B and Qwen3.5 accept images (Gemma 4 E2B / E4B also audio), which can replace a separate vision model for the describe a screenshot step of the running scenario; Phi-4-mini, SmolLM3, Qwen3 and Llama 3.2 are text only. Multimodal inputs cost extra memory and not every runtime implements every modality.
Hosted vs. open names. These are the open counterparts of the hosted models discussed in The Model Landscape; an on-device model is far smaller than a hosted frontier model, which is the subject of the quality section below.

Running a small model directly with transformers is the quickest way to try one before choosing a runtime. A text-only model loads with the Auto classes; device_map="auto" places it on the GPU (or MPS) when one is available, and only checkpoints whose cards you trust should be loaded, never with trust_remote_code=True unless you have read the repository’s code.

Chatting with a small model through transformers (Qwen3-4B, non-thinking mode)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Qwen/Qwen3-4B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="auto"
)

messages = [{"role": "user", "content": "In one sentence, what is a Hub model card?"}]
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(output[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Reference: Qwen3-4B model card and Transformers documentation. The pipeline-based equivalent is on Transformers Pipelines.

Official quantized formats per model

A safetensors repository in 16-bit precision is rarely what ships in an app. The rows below list the quantized repositories that exist on the Hub as of 2026-09-30, and who publishes them. "Official" here means published by the model’s own vendor (Google, Microsoft, Qwen, Hugging Face, Meta) or by the format’s maintainer organisation (ggml-org for GGUF, litert-community for LiteRT, onnx-community for ONNX, mlx-community for MLX); each of these repositories was confirmed to exist on the Hub API. Community organisations are curated but are not the model vendor, so read the licence and the card of every one; the format background is on Model Formats.

Table 2. Quantized repositories that exist for each small LLM
Model	Repositories confirmed on the Hub
Gemma 4 E2B / E4B	GGUF: `ggml-org/gemma-4-E2B-it-GGUF`, `ggml-org/gemma-4-E4B-it-GGUF` (BF16, Q8_0, Q4_0 and a separate `mmproj` file for the vision and audio encoders). LiteRT-LM: `litert-community/gemma-4-E2B-it-litert-lm`, `litert-community/gemma-4-E4B-it-litert-lm` (`.litertlm` bundles, including a `-gpu` and a `-web` variant). ONNX: `onnx-community/gemma-4-E2B-it-ONNX`, `onnx-community/gemma-4-E4B-it-ONNX` (separate audio, vision and decoder graphs in several precisions, laid out for Transformers.js). MLX: `mlx-community/gemma-4-e2b-it-4bit` and `mlx-community/gemma-4-E2B-it-qat-4bit`
Gemma 3n E2B / E4B	LiteRT-LM: `google/gemma-3n-E2B-it-litert-lm`, `google/gemma-3n-E4B-it-litert-lm` (published by Google, gated with the Gemma terms). ONNX: `onnx-community/gemma-3n-E2B-it-ONNX`. MLX: `mlx-community/gemma-3n-E2B-it-4bit`
Gemma 3 1B / 4B	Quantization-aware-trained GGUF from Google: `google/gemma-3-1b-it-qat-q4_0-gguf`, `google/gemma-3-4b-it-qat-q4_0-gguf` (gated). GGUF without gating: `ggml-org/gemma-3-1b-it-GGUF`, `ggml-org/gemma-3-4b-it-GGUF` (Q4_K_M, Q8_0, f16). LiteRT: `litert-community/Gemma3-1B-IT`. ONNX: `onnx-community/gemma-3-1b-it-ONNX`
Phi-4-mini	ONNX from Microsoft for ONNX Runtime GenAI, with `cpu_and_mobile` and `gpu` int4 folders: `microsoft/Phi-4-mini-instruct-onnx`. Transformers.js-style ONNX: `onnx-community/Phi-4-mini-instruct-ONNX`. LiteRT: `litert-community/Phi-4-mini-instruct`. MLX: `mlx-community/Phi-4-mini-instruct-4bit`. Microsoft does not publish a GGUF on the Hub that was found here; community GGUF repositories exist (for example `unsloth/Phi-4-mini-instruct-GGUF`)
Qwen3 0.6B / 4B	GGUF from the Qwen organisation: `Qwen/Qwen3-0.6B-GGUF`, `Qwen/Qwen3-4B-GGUF`. MLX from the Qwen organisation: `Qwen/Qwen3-4B-MLX-4bit`. AWQ for GPU servers: `Qwen/Qwen3-4B-AWQ`. ONNX: `onnx-community/Qwen3-0.6B-ONNX`. LiteRT-LM: `litert-community/Qwen3-0.6B`
Qwen3.5 0.8B / 2B / 4B	GGUF: `ggml-org/Qwen3.5-0.8B-GGUF` (found for 0.8B only among the three). ONNX: `onnx-community/Qwen3.5-0.8B-ONNX`, `onnx-community/Qwen3.5-4B-ONNX`. LiteRT-LM: `litert-community/Qwen3.5-0.8B`, `litert-community/Qwen3.5-2B`, `litert-community/Qwen3.5-4B`. No `Qwen/Qwen3.5-*-GGUF` repositories were found; community GGUF repositories exist
SmolLM3	GGUF: `ggml-org/SmolLM3-3B-GGUF`. ONNX from Hugging Face: `HuggingFaceTB/SmolLM3-3B-ONNX`. MLX: `mlx-community/SmolLM3-3B-4bit`. LiteRT-LM: `litert-community/SmolLM3-3B`
Llama 3.2 1B / 3B	Quantized by Meta: `meta-llama/Llama-3.2-1B-Instruct-SpinQuant_INT4_EO8`, `meta-llama/Llama-3.2-3B-Instruct-SpinQuant_INT4_EO8`, `meta-llama/Llama-3.2-1B-Instruct-QLORA_INT4_EO8` and `meta-llama/Llama-3.2-3B-Instruct-QLORA_INT4_EO8` (gated; two quantization schemes aimed at on-device runtimes such as ExecuTorch, see the cards for the exact file format and required runtime, and Meta’s Llama quantization guide). Community-organisation formats: `onnx-community/Llama-3.2-1B-Instruct-ONNX`, `mlx-community/Llama-3.2-1B-Instruct-4bit`, `mlx-community/Llama-3.2-3B-Instruct-4bit`, `litert-community/Llama-3.2-1B` and `litert-community/Llama-3.2-3B` (the LiteRT pair is gated)

A pattern to note: GGUF is the widest net (llama.cpp, Ollama, LM Studio and Jan all read it); LiteRT-LM bundles target Android and the web; MLX repositories target Apple silicon; ONNX targets ONNX Runtime and Transformers.js. The Gemma 4 and Gemma 3 cards describe Google’s own quantization-aware-trained (QAT) checkpoints: the model is trained to tolerate 4-bit weights, which preserves more quality than quantizing a finished model afterwards. When you need a precision nobody published, convert the model yourself as shown on Conversion & Quantization.

Loading one official GGUF from the Hub and chatting with it through llama-cpp-python:

Running the official Gemma 4 E2B GGUF (Q4_0) with llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="ggml-org/gemma-4-E2B-it-GGUF",
    filename="gemma-4-E2B-it-Q4_0.gguf",   # the exact file; the repo also holds BF16 and Q8_0
    n_ctx=4096,
    verbose=False,
)
reply = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Summarise what DocsAssistant does in one sentence."}],
    max_tokens=128,
)
print(reply["choices"][0]["message"]["content"])

Reference: llama-cpp-python documentation and ggml-org/gemma-4-E2B-it-GGUF. The vision and audio inputs of a multimodal GGUF need the separate mmproj file and a runtime that supports it; check the llama.cpp documentation for the current multimodal tooling before relying on it.

The Apple-silicon equivalent uses an MLX repository with mlx-lm:

Running a 4-bit MLX model on Apple silicon (Phi-4-mini)

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Phi-4-mini-instruct-4bit")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "List two benefits of on-device inference."}],
    add_generation_prompt=True,
    tokenize=False,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=128))

Reference: mlx-lm and Hugging Face — MLX. For Android the .litertlm bundle is fetched with hf download and given to the LiteRT-LM runtime; that path is covered on the Android page.

Downloading only the LiteRT-LM bundle of Gemma 4 E2B

hf download litert-community/gemma-4-E2B-it-litert-lm gemma-4-E2B-it.litertlm --local-dir ./models

Reference: Hugging Face Hub — CLI guide and LiteRT-LM overview.

Picking a model

The diagram condenses the choices above into the questions you actually answer: what the licence lets you ship, what the device can hold, and which inputs are needed.

flowchart TD A["Need an on-device chat model"] --> B{"Can the app download it
without a login?"} B -- "yes: permissive licence" --> C{"Images or audio needed?"} B -- "no: gated terms acceptable" --> G["Gemma 3n / 3 (incl. Gemma 3 1B),
Llama 3.2 (incl. 1B)"] C -- "yes" --> D["Gemma 4 E2B / E4B
or Qwen3.5"] C -- "no" --> E{"Memory budget"} E -- "under about 1 GB" --> F["Qwen3 0.6B or Qwen3.5 0.8B"] E -- "2 to 4 GB" --> H["Phi-4-mini, SmolLM3, Qwen3 4B,
Gemma 4 E2B"] D --> I["Quantize: GGUF, LiteRT-LM,
MLX or ONNX"] F --> I H --> I G --> I

Figure 1. Choosing a small LLM for an app

Reference: Gemma documentation and the model cards linked in the tables.

What quality to expect on device

Quality is relative to a hosted frontier model, and the gap is real. Hugging Face, Google, Microsoft and Qwen publish benchmark tables on each card, which are useful to compare siblings but are measured in their own harnesses on full-precision weights; this site quotes none of them.

Table 3. Qualitative expectations for models in the 0.5B to 5B range
Task	What to expect
Short factual or conversational replies	Fluent and usually sensible; factual recall is shallower than a hosted model, so ground answers in retrieved text (the embeddings below) rather than in the model’s memory
Summarising and rewriting supplied text	Good for a paragraph or a page; quality drops as the input approaches the context limit, and the KV-cache grows with it
Extraction and classification into a fixed schema	Works well with a constrained output format or JSON schema and a few examples; validate the output anyway
Reasoning, maths and multi-step planning	Thinking modes (Gemma 4, Qwen3 and Qwen3.5, SmolLM3) help but cost many tokens and therefore latency and battery; keep thinking off for interactive replies
Tool calling	Supported by the Gemma 4, Qwen3 and SmolLM3 cards, but reliability on small models is lower than on hosted ones; give few tools with simple schemas
Non-English text	Varies by model (see the language claims in the table); 4-bit quantization hurts lower-resource languages first
Long documents	Advertised contexts are large, but on a phone the usable length is set by memory for the KV-cache, not by the model card

Practical consequences for DocsAssistant: keep the retrieved context short, prefer a 4-bit model that fits comfortably over a larger one that makes the OS kill the app, stream tokens so perceived latency is low, and keep a cloud fallback for questions the local model cannot handle (the hosted counterpart is covered in Inference Providers & Spaces).

Reference: Gemma 4 E2B model card (intended use, limitations) and Hugging Face — Model cards.

Embedding models

A retrieval pipeline needs a second, much smaller model that maps text to a vector so similar texts land near each other. Embedding models are encoder-style: no text generation, one forward pass per input, and they run comfortably on a CPU, a phone or in a browser. All rows were read from the Hub API on 2026-09-30.

Table 4. The embedding models
Model (repository)	Size	Licence, gating	Notes from the model card
MiniLM (`sentence-transformers/all-MiniLM-L6-v2`; `all-MiniLM-L12-v2`)	22.7 M / 33.4 M parameters	Apache-2.0, not gated	The classic baseline used by most tutorials; trained for English and short inputs (see the card); the repository already contains `onnx/` exports
Qwen3-Embedding (`Qwen/Qwen3-Embedding-0.6B`, `Qwen/Qwen3-Embedding-4B`)	0.6 B / 4 B parameters	Apache-2.0, not gated	100+ languages including programming languages; 32k context; instruction-aware (queries carry a one-sentence task instruction, documents do not); output dimension up to 1024 for the 0.6B with user-defined sizes from 32 to 1024 (Matryoshka-style)
EmbeddingGemma (`google/embeddinggemma-300m`)	about 303 M parameters	Gemma terms, manual	Google’s small embedding model (the card is gated; read it after accepting the terms for languages, dimensions and prompts); also published as a quantization-aware variant `google/embeddinggemma-300m-qat-q4_0-unquantized`

Official quantized repositories, confirmed on the Hub API:

MiniLM: ONNX inside the original repository, plus Transformers.js-style repositories Xenova/all-MiniLM-L6-v2 and onnx-community/all-MiniLM-L6-v2-ONNX.
Qwen3-Embedding: GGUF Qwen/Qwen3-Embedding-0.6B-GGUF (Q8_0 and f16) from the Qwen organisation, and ONNX onnx-community/Qwen3-Embedding-0.6B-ONNX.
EmbeddingGemma: ONNX onnx-community/embeddinggemma-300m-ONNX, GGUF ggml-org/embeddinggemma-300M-qat-q4_0-GGUF (not gated), and LiteRT litert-community/embeddinggemma-300m (gated, includes .tflite variants for specific phone NPUs).

Both the query and the document side of a retriever must use the same model, and the vectors are only comparable within one model (and one output dimension): switching models means re-embedding the whole corpus, as discussed on Generating Embeddings in Practice.

Embedding and ranking documents with sentence-transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B")

queries = ["How do I download a gated model?"]
documents = [
    "Accept the licence on the model page, then authenticate with a token.",
    "Use the cache directory to free disk space.",
]
query_embeddings = model.encode(queries, prompt_name="query")   # instruction-aware side
document_embeddings = model.encode(documents)                    # no instruction
print(model.similarity(query_embeddings, document_embeddings))

Reference: Qwen3-Embedding-0.6B model card and Sentence Transformers documentation. SentenceTransformer("google/embeddinggemma-300m") works the same way after you have accepted the Gemma terms and authenticated; read the card for its recommended query and document prompts.

For the browser and for Spring Boot the same models run as ONNX. In Transformers.js:

Embedding text in the browser with Transformers.js (MiniLM ONNX)

import { pipeline } from "@huggingface/transformers";

const extractor = await pipeline("feature-extraction", "Xenova/all-MiniLM-L6-v2");
const output = await extractor(["On-device retrieval"], { pooling: "mean", normalize: true });
console.log(output.dims);

Reference: Transformers.js documentation. In-process ONNX embeddings on the JVM are covered in Java & Spring Boot, and the browser runtime in Web & Cross-Platform.

Choosing among the three: MiniLM when English, short passages and a tiny footprint are enough; EmbeddingGemma for multilingual on-device retrieval when the Gemma terms and gating are acceptable; Qwen3-Embedding for the widest language and code coverage and longer inputs, with the 0.6B as the on-device choice and the 4B for servers. On quality, public leaderboards such as MTEB compare them, but always evaluate on a sample of your queries and documents (see Choosing Models & Licences for how to read a leaderboard). Smaller output dimensions and quantized weights reduce storage and search cost at some quality loss.