Hugging Face & On-Device Models

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Examples across this sub-section were written against the following versions, all read from PyPI, npm, Maven Central, NuGet, the projects' release pages and their official documentation on 2026-09-30. Hugging Face libraries: huggingface_hub 2.0.0 (released 2026-09-24; it ships the hf command-line client), transformers 5.17.0 (PyTorch only), safetensors 0.8.0, datasets 5.0.1, accelerate 1.15.0, peft 0.21.1, trl 1.14.1, bitsandbytes 0.50.2, sentence-transformers 6.1.0, optimum 2.3.0 (and, in a separate environment, optimum-onnx 0.1.0, which pins optimum 2.1.x), smolagents 1.26.0 and gradio 6.29.0. Runtimes: ONNX Runtime 1.30.0, llama.cpp v0.5.0, llama-cpp-python 0.3.35, faster-whisper 1.2.1 (CTranslate2 4.8.2), kokoro 0.9.4, Piper 1.8.0, mlx-lm 0.31.3, coremltools 9.0, LiteRT 2.2.0, LiteRT-LM 0.17.1, litert-torch 0.9.4, ExecuTorch 1.4.0 (the Maven Central executorch-android artifact; the PyPI and GitHub releases have moved on to 1.5.1), Transformers.js (@huggingface/transformers) 4.3.0, react-native-executorch 0.10.4, DJL 0.38.0, swift-transformers 1.3.4, mlx-swift-lm 3.31.4 and argmax-oss-swift v1.1.0. Platforms: iOS, iPadOS and macOS 27 with Xcode 27 (for Core AI), Android with Kotlin, Windows with .NET 8 or later, and browsers with WebGPU. The Python, Java and Spring baseline is shared with the rest of the AI section: Python 3.12 with PyTorch 2.14.0 and FastAPI 0.142.1, Java 21, Spring Boot 4.1, Spring AI 2.0.1 and LangChain4j 1.20.2, and Node.js 24 — the same as Running LLMs Locally and LLM Foundations & Prompting. Facts that could not be confirmed on a page are hedged there (for example, the Apple page lists which of its Swift snippets were not built or run, although every Apple API name it uses was read from Apple’s documentation), and model licences and gating are always properties of one repository: read the model card of the exact model you ship. If a page and this baseline disagree after a later update, trust this baseline.

The obtain, convert, integrate workflow

Every page of this sub-section is one step of a three-stage loop. Obtain a model: find it on the Hub, read its card, check its licence and size, and download it. Convert it if the runtime you need does not already have a published copy in its own format: GGUF, ONNX, MLX, Core ML or Core AI, LiteRT, ExecuTorch. Integrate it: load it from a Python or Spring Boot service, from a native app, or in a browser, and decide how the weights reach the device. Adapting the model with PEFT or TRL is an optional side trip between the first two stages.

flowchart LR NEED(["Need: task, device, licence"]) --> FIND["Find on the Hub
search, filters, model card"] FIND --> CHECK{"Licence, gating
and size acceptable?"} CHECK -->|"no"| FIND CHECK -->|"yes"| GET["Download
hf download / hf_hub_download"] GET --> FMT{"Does a published copy
exist in the runtime's format?"} FMT -->|"yes: GGUF, ONNX, MLX, LiteRT"| INT FMT -->|"no"| CONV["Convert and quantize
GGUF, ONNX, Core ML, Core AI,
LiteRT, MLX, ExecuTorch"] GET -.->|"optional"| TUNE["Fine-tune
PEFT / TRL"] TUNE -.-> CONV CONV --> INT["Integrate
server, native app or browser"] INT --> SHIP["Ship
model delivery, licence notices,
offline mode"]

The first stage is two commands, shown below with the current hf client and the huggingface_hub Python API. The token is read from the HF_TOKEN environment variable and is only needed for gated or private repositories; never paste it into a file or a command line.

Obtain: download a whole repository filtered by pattern, or one file, then work offline
export HF_TOKEN="<your access token>"   # only for gated or private repositories

# The safetensors weights of a model into the shared cache (add patterns for config files)
hf download openai/whisper-small.en --include "*.safetensors"

# One GGUF file into a folder you can ship with an application
hf download ggml-org/gemma-3-1b-it-GGUF gemma-3-1b-it-Q4_K_M.gguf --local-dir ./models

# From now on, forbid every network call to the Hub
export HF_HUB_OFFLINE=1

Official reference: Hugging Face — hf CLI guide. The details (authentication, gated models, cache management, HF_HOME, Xet storage) are on Hub Essentials.

Obtain from Python: a single file and a filtered snapshot
from huggingface_hub import hf_hub_download, snapshot_download

gguf_path = hf_hub_download(
    repo_id="ggml-org/gemma-3-1b-it-GGUF",
    filename="gemma-3-1b-it-Q4_K_M.gguf",
)
model_dir = snapshot_download(
    repo_id="openai/whisper-small.en",
    allow_patterns=["*.safetensors", "*.json", "*.txt"],
)
print(gguf_path, model_dir)

Official reference: Hugging Face — Download files from the Hub.

The convert stage is one command per target format. The exact commands, their flags and the quality trade-offs are on Conversion & Quantization; the list of formats and which runtime reads which is on Model Formats.

Convert: two examples, a GGUF file for llama.cpp and an ONNX export for ONNX Runtime
# GGUF for llama.cpp, Ollama and llama-cpp-python (run from a llama.cpp checkout)
python convert_hf_to_gguf.py ./Qwen3-0.6B --outfile qwen3-0.6b-bf16.gguf --outtype bf16
./llama-quantize qwen3-0.6b-bf16.gguf qwen3-0.6b-Q4_K_M.gguf Q4_K_M

# ONNX for ONNX Runtime, Transformers.js and the JVM and .NET runtimes
pip install "optimum-onnx[onnxruntime]"
optimum-cli export onnx --model distilbert/distilbert-base-uncased-finetuned-sst-2-english ./onnx-out

Official reference: llama.cpp — quantize README and Optimum ONNX — Export a model. optimum-onnx 0.1.0, the latest release on PyPI on 2026-09-30, pins optimum~=2.1.0 and transformers>=4.36,<4.58, so installing it downgrades a transformers 5 environment; run the export in a separate virtual environment (for example python -m venv .venv-onnx) and keep the transformers 5 baseline in your main one.

The integrate stage is the rest of the sub-section, and the running scenario below shows what it looks like on each platform.

DocsAssistant goes offline

DocsAssistant, the running example of this site’s AI and vector-database sections, normally calls hosted APIs. In this sub-section it goes offline: one example app per platform performs the same four tasks with models from the Hub and no network call at run time.

Table 1. The four tasks and the Hub models the pages use for them
Task Models Where they are catalogued

Transcribe a spoken question

Whisper (openai/whisper-large-v3-turbo, openai/whisper-small.en), Moonshine (moonshine-ai/moonshine-base), Parakeet (nvidia/parakeet-tdt-0.6b-v3)

Speech Models: STT & TTS

Answer it with a small local LLM

Gemma 4 E2B (google/gemma-4-E2B-it), Qwen3.5 (Qwen/Qwen3.5-4B), Phi-4-mini (microsoft/Phi-4-mini-instruct), plus an embedding model to retrieve the passages (sentence-transformers/all-MiniLM-L6-v2)

Small LLMs & Embeddings

Speak the answer

Kokoro (hexgrad/Kokoro-82M), Piper voices (rhasspy/piper-voices)

Speech Models: STT & TTS

Classify or describe a screenshot

SigLIP 2 (google/siglip2-base-patch16-224) to classify against labels, SmolVLM2 (HuggingFaceTB/SmolVLM2-2.2B-Instruct) to describe

Vision Models

Each platform page implements the four tasks in its own language, and each snippet is followed by its official documentation link. Model IDs on the pages were read from the Hub when the pages were written; confirm that an ID still exists and re-read its licence before you ship it.

Table 2. Where each platform implements the scenario
Platform and language Page What the page wires together

Python backend (FastAPI)

Python Backends

faster-whisper for transcription, llama-cpp-python or transformers serve for the answer, Kokoro for speech and a transformers pipeline for the screenshot, each model loaded once in lifespan.

Spring Boot (Java)

Java & Spring Boot

Spring AI in front of a local server for chat, in-process ONNX embeddings, and a sidecar or ONNX for audio.

iOS, iPadOS and macOS (Swift)

Apple Platforms

WhisperKit and TTSKit from argmax-oss-swift, mlx-swift-lm for the LLM, and Core ML or Core AI for the image model.

Android (Kotlin)

Android

whisper.cpp through JNI, LiteRT-LM with a .litertlm bundle, Android TextToSpeech and LiteRT for vision, with Gemini Nano as a system-provided alternative.

Windows (C#)

Windows & .NET

ONNX Runtime GenAI, Foundry Local behind IChatClient, the system speech synthesizer and Windows ML.

Browser (JavaScript)

Web & Cross-Platform

Transformers.js pipelines in a Web Worker on WebGPU or WASM; React Native and Flutter variants on the same page.

Hosted fallback (Python and JavaScript)

Inference Providers & Spaces

The cloud counterpart through the Inference Providers router, for prototyping and for devices that cannot run a model.

The decision matrix

The matrix answers the question every integration starts with: on this platform, for this task, which runtime should I try first, and which Hub format does it read? Each cell names the preferred runtime and the format of the weights it loads. It is a starting point, not a guarantee: a cell holds only while a package maintainer keeps exporting and testing the model, so check the platform page and the model card before you commit.

Decision matrix with platforms as rows (Python server

The same matrix as a table, with the page that explains each row:

Platform STT LLM TTS Vision

Python server (page)

faster-whisper, CTranslate2 format

llama-cpp-python, GGUF (or transformers serve, safetensors)

Kokoro through the kokoro package, safetensors

transformers pipeline, safetensors (SigLIP 2, SmolVLM)

JVM and Spring Boot (page)

A sidecar running faster-whisper, or an ONNX Whisper

A local server (Ollama, llama.cpp) behind Spring AI, GGUF

A Kokoro sidecar, or an ONNX export

ONNX Runtime Java with an ONNX SigLIP 2, or a multimodal local server

Apple (page)

WhisperKit, Core ML

mlx-swift-lm, MLX 4-bit (or Core AI, or the llama.cpp XCFramework with GGUF)

TTSKit, Core ML

Core ML or Core AI image encoder (.mlpackage, .aimodel)

Android (page)

whisper.cpp through JNI, ggml model file (or Moonshine on LiteRT)

LiteRT-LM, .litertlm (or Gemini Nano through ML Kit GenAI)

Android TextToSpeech, system voices (or Kokoro and Piper through ONNX Runtime)

LiteRT CompiledModel, .tflite (or SigLIP 2 through ONNX Runtime)

Windows and .NET (page)

ONNX Runtime GenAI or Foundry Local, ONNX Whisper (or Whisper.net)

Foundry Local behind IChatClient, ONNX

SpeechSynthesizer, system voices (or Kokoro through ONNX)

Windows ML, ONNX

Web (page)

Transformers.js, ONNX on WebGPU or WASM

Transformers.js text-generation, ONNX at q4

kokoro-js, ONNX

Transformers.js zero-shot-image-classification, ONNX

React Native and Flutter (page)

react-native-executorch useSpeechToText, .pte

llama.rn (GGUF), react-native-executorch (.pte), flutter_gemma (.litertlm) or llama_cpp_dart (GGUF)

A native or platform TTS, or ONNX through onnxruntime

ExecuTorch .pte, a multimodal GGUF with llama.rn, or ONNX through onnxruntime

Three rules explain most of the matrix. The format follows the runtime: GGUF for llama.cpp and the tools built on it, ONNX for ONNX Runtime and everything that embeds it (Java, .NET, browsers, Windows ML), MLX and Core ML or Core AI for Apple, .tflite and .litertlm for Google’s stack, .pte for ExecuTorch. Chat LLMs on a server are best served, not embedded: a JVM or .NET service points at a local server rather than loading the model itself. And the operating system sometimes already ships the model: Android’s Gemini Nano, Windows' built-in language and text-recognition models and Apple’s system model need no Hub download at all, at the cost of a fixed, smaller feature set and a limited device list (see Android, Windows & .NET and Apple Intelligence and Machine Learning).

Recent renames and deprecations

Older tutorials and the books in the bibliography pre-date a wave of renames in 2025 and 2026. Code that follows them usually still tells you what it does, but the name, the repository or the documentation host has moved. Every row was re-checked against the official sources on 2026-09-30 except where the last column says otherwise.

Old Now Status on 2026-09-30

huggingface-cli, the original command-line client (removed in huggingface_hub 1.0; use hf)

The hf CLI: hf auth login, hf download, hf cache ls

Confirmed. huggingface_hub 2.0.0 (2026-09-24) also removes the APIs that were deprecated during the 1.x line; see the migration guide in the bibliography.

The free, serverless Inference API (api-inference.huggingface.co) and the hosted-inference widget

Inference Providers: the OpenAI-compatible router and InferenceClient(provider=…​)

Confirmed on the pages; the router base URL is on Inference Providers & Spaces.

smolagents HfApiModel

InferenceClientModel

Confirmed in the smolagents 1.26.0 documentation read for Inference Providers & Spaces.

Optimum’s ONNX export (optimum[exporters])

The separate optimum-onnx package (0.1.0) with the same optimum-cli export onnx command

Confirmed; see Conversion & Quantization. The 0.1.0 release (still the latest on PyPI on 2026-09-30) pins optimum~=2.1.0 and transformers below 4.58, so install it in a separate virtual environment from the optimum 2.3.0 and transformers 5 baseline.

transformers v4 (TensorFlow and Flax, slow and fast tokenizers, torch_dtype, load_in_8bit=True, pipeline("summarization"))

transformers v5: PyTorch only, one tokenizer backend, dtype, quantization_config, transformers serve

Confirmed against the v5 migration guide; the table of edits is on Transformers Pipelines.

Gradio 5 and earlier tutorials

Gradio 6 (6.29.0 on the check date)

Confirmed.

ai-edge-torch

litert-torch

Confirmed.

TensorFlow Lite (TFLite)

LiteRT (repository google-ai-edge/LiteRT)

Confirmed.

MediaPipe LLM Inference

Maintenance only; migrate to LiteRT-LM

Confirmed on Google’s MediaPipe page; migration notes are on Android.

Documentation at ai.google.dev/edge

developers.google.com/edge

Confirmed; the bibliography uses the new host.

The WhisperKit repository

argmax-oss-swift (WhisperKit, TTSKit and SpeakerKit as products of one package)

Confirmed.

mlx-swift-examples LLM libraries

mlx-swift-lm

Repository confirmed; see Apple Platforms.

rhasspy/piper

OHF-Voice/piper1-gpl (GPL-3.0; the PyPI package is piper-tts)

Confirmed, and the GPL licence matters for shipping in an app (see Choosing Models & Licences). The voices are still listed under rhasspy/piper-voices on the Hub.

Moonshine under UsefulSensors

moonshine-ai

Confirmed.

DirectML

In "sustained engineering"; use Windows ML for Windows-based ONNX Runtime deployments

Confirmed on Microsoft Learn; see Windows & .NET.

Phi Silica

Microsoft’s Phi Silica documentation page announces that a new on-device model, Aion Instruct, is replacing it in late 2026

A vendor announcement read on the official page; it was not tested and the schedule may move, so isolate Phi Silica calls behind your own interface (details on Windows & .NET).

AutoAWQ

Deprecated; its functionality moved into llm-compressor (vLLM project)

llm-compressor 0.14.0 confirmed; the deprecation of AutoAWQ is stated by the projects, but its repository archive status was not re-checked.

Core ML as the only Apple route

Core ML plus Core AI (.aimodel, iOS and macOS 27 and later); Foundation Models gains a LanguageModelExecutor protocol for running your own model in a session

Core AI and the LanguageModelExecutor protocol confirmed in Apple’s documentation for the 27 SDK; neither could be tested here; see Apple Platforms.

What’s covered

The Hub

  • Hub Essentials — model, dataset and Space repositories, searching and filtering, model cards, tokens and gated models, the hf CLI, the huggingface_hub 2.0 API, the cache and HF_HOME, offline mode and Xet storage.

  • Choosing Models & Licences — reading a model card, size versus hardware, licence classes from Apache-2.0 to AGPL-3.0 and non-commercial, gated access, leaderboards and a licence checklist for shipping in an app.

  • Model Formats — safetensors versus pickle, GGUF, ONNX, MLX, Core ML and Core AI, LiteRT, ExecuTorch and CTranslate2, and which runtime reads which format.

Python and models

  • Transformers Pipelines — the v5 pipeline() per task, the Auto* classes and processors, device and dtype, transformers serve and the v4 to v5 migration.

  • Speech Models: STT & TTS — a curated catalogue of Whisper, Parakeet, Moonshine, Kokoro, Piper and the other speech models, with licence, size, languages and streaming support.

  • Vision Models — SigLIP 2, DINOv2 and DINOv3, YOLO under AGPL-3.0 versus RT-DETRv2 and D-FINE, SAM 2.1 and SAM 3, Florence-2, SmolVLM, Qwen-VL and FastVLM, with code for classification, detection, segmentation and captioning.

  • Small LLMs & Embeddings — Gemma 4, Phi-4-mini, Qwen3 and Qwen3.5, SmolLM3 and Llama 3.2, the MiniLM, Qwen3-Embedding and EmbeddingGemma embedding models, their official quantized repositories and memory sizing.

  • Conversion & Quantization — GGUF, ONNX, Core ML, Core AI, LiteRT, MLX and ExecuTorch conversion, and GPU-server quantization with bitsandbytes, GPTQModel and llm-compressor.

  • Fine-Tuning with PEFT & TRL — when to fine-tune, the LoRA update, QLoRA, TRL SFTTrainer and DPO, dataset preparation, merging adapters and exporting them for on-device use.

Integrating into apps

  • Python Backends — FastAPI with models loaded once in lifespan, keeping the event loop free, batching, streaming, the DocsAssistant endpoints, ONNX Runtime sessions and text-embeddings-inference.

  • Java & Spring Boot — DJL, ONNX Runtime Java, Spring AI TransformersEmbeddingModel, LangChain4j in-process embeddings, the caveats of Jlama and java-llama.cpp, and the recommended local-server path.

  • Apple Platforms — Core ML versus Core AI, swift-transformers, MLX, WhisperKit and TTSKit, LanguageModelExecutor, the llama.cpp XCFramework and on-demand model download.

  • Android — LiteRT and the CompiledModel API, LiteRT-LM, Gemini Nano through ML Kit GenAI, ONNX Runtime Mobile, ExecuTorch, llama.cpp and whisper.cpp, and model delivery.

  • Windows & .NET — Windows ML, the Windows AI APIs, Foundry Local, ONNX Runtime GenAI, the DirectML status and Microsoft.Extensions.AI.

  • Web & Cross-Platform — Transformers.js on WebGPU or WASM, ONNX Runtime Web, ExecuTorch, React Native and Flutter.

  • Inference Providers & Spaces — the hosted counterparts: the Inference Providers router, Inference Endpoints, Gradio 6, Spaces and ZeroGPU, smolagents, and when to prefer local.

Cheat sheet

Related sub-sections: LLM Foundations & Prompting explains the models themselves and how to prompt them; Running LLMs Locally covers the engines (Ollama, llama.cpp, vLLM) that serve the GGUF and safetensors models found here; LangChain and Spring AI wrap local and hosted models in application code; Model Context Protocol (MCP) and AI Agents build on them; and Vector Databases & RAG uses the embedding models. Platform basics live in Apple Intelligence and Machine Learning, Android, React Native, Spring Boot and Python. Real-time voice pipelines belong to the Voice Agents sub-section, serving at scale to LLMOps: Serving at Scale, the security of the model supply chain to AI Supply Chain Security, FastAPI itself to the planned Python FastAPI section, and Windows packaging to the planned Windows Apps section.

Bibliography

The following sources were consulted while preparing this sub-section’s pages. Every command, flag, environment variable, package name, API name, model ID, licence and version was checked against the projects' official documentation, model cards, release pages and package registries on 2026-09-30 — never copied from a book. Where something could not be confirmed or tested, the page that cites it says so.

Requester-provided books, consulted for concepts only and never the source of an example:

  • Lee, Wei-Meng. Hugging Face in Action. Manning Publications, 2026. ISBN 978-1-63343-671-8. Publisher page: https://www.manning.com/books/hugging-face-in-action . Official code repository: https://github.com/weimenglee/HuggingFaceInAction . Consulted for the Hub and model-card workflow, the pipeline() versus Auto* classes distinction, vision models (detection, classification, segmentation, CLIP), datasets, fine-tuning with Trainer, smolagents, Gradio and Spaces, and local GGUF models. It is outdated on the huggingface-cli command (now hf), the serverless Inference API (now Inference Providers), HfApiModel (now InferenceClientModel), transformers v4 idioms and GPT4All; those pages show the current API instead.

  • Alammar, Jay, and Maarten Grootendorst. Hands-On Large Language Models: Language Understanding and Generation. O’Reilly Media, Sep 2024. ISBN 978-1-098-15096-9. Publisher page: https://www.oreilly.com/library/view/hands-on-large-language/9781098150952/ . Official code repository: https://github.com/HandsOnLLM/Hands-On-Large-Language-Models . Consulted (Chapters 2, 4, 5 and 9 to 12) for tokenizers, classification with task models versus embeddings, CLIP and BLIP-2, training embedding models, fine-tuning BERT-style models and SetFit, and QLoRA (NF4 through bitsandbytes, LoRA rank and alpha, TRL SFTTrainer, merging) and DPO. Its library calls predate transformers v5 and TRL 1.x.

  • Huyen, Chip. AI Engineering: Building Applications with Foundation Models. O’Reilly Media, Dec 2024. ISBN 978-1-098-16630-4. Publisher page: https://www.oreilly.com/library/view/ai-engineering/9781098166298/ . Official code repository: https://github.com/chiphuyen/aie-book . Consulted (Chapter 7, fine-tuning, and Chapter 8, dataset engineering) for when to fine-tune, the memory arithmetic of fine-tuning, PEFT, LoRA and QLoRA, merging adapters, and dataset curation, synthesis and deduplication.

  • Wilson, Steve. The Developer’s Playbook for Large Language Model Security. O’Reilly Media, Sep 2024. ISBN 978-1-098-16220-7. Publisher page: https://www.oreilly.com/library/view/the-developers-playbook/9781098162191/ . No official code repository. Consulted (Chapter 9) for the model supply chain: pickle malware, model cards and an ML bill of materials.

oreilly.com answers automated fetches with an HTTP 403 response, but the publisher pages above are live pages when opened in a browser.

Official documentation

Hugging Face Hub:

huggingface_hub:

Hugging Face libraries:

Model cards (linked for the licence, gating and usage of each model named on the catalogue pages):

Inference Providers and Spaces:

Runtimes and engines:

Python backends:

Java and Spring:

Apple:

Android and Google:

Windows and .NET:

Web and cross-platform:

Specifications and standards

Papers and engineering articles

These books are consulted references, not the primary source of any example on this sub-section’s pages. On any discrepancy between a book and a tool’s own official documentation, that documentation is authoritative.