Conversion & Quantization

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

This page was written against transformers 5.17.0, optimum 2.3.0 (and, in a separate environment, optimum-onnx 0.1.0 with optimum 2.1.x), ONNX Runtime 1.30.0, coremltools 9.0, litert-torch 0.9.4, LiteRT-LM 0.17.1, mlx-lm 0.31.3, bitsandbytes 0.50.2, llm-compressor 0.14.0 and llama.cpp v0.5.0, together with Apple’s Core AI documentation for iOS and macOS 27, as checked on 2026-09-30. Most models you need are already published in the format you want (see Model Formats); you convert yourself when the model is new, when it is your own fine-tune (see Fine-tuning with PEFT and TRL), or when you need a quantization level nobody published. The theory of what quantization does to the weights, the scale and zero-point formula, and the GGUF quant types is on Quantization for Inference and is not repeated here; this page is the tool-by-tool how-to.

A converted model is a derivative work: it inherits the licence and any gating terms of the original checkpoint (see Choosing Models and Licences), so check the model card before you publish a conversion. Converters also execute code from the checkpoint repository when you pass trust_remote_code: leave it off unless you have read the code and pinned a revision, and prefer safetensors checkpoints over pickled pytorch_model.bin files.

The conversion pipelines at a glance

Every path starts from the same place, a Hub checkpoint in safetensors, and ends in a runtime-specific artefact. The diagram shows the tool that performs each step; the sections below give one command per path.

flowchart LR hub["Hub checkpoint (safetensors + config)"] hub --> gguf_conv["convert_hf_to_gguf.py"] gguf_conv --> gguf16["GGUF (BF16 / F16)"] gguf16 --> imat["llama-imatrix (optional)"] imat --> gguf_q["llama-quantize"] gguf16 --> gguf_q gguf_q --> gguf_out["GGUF Q4_K_M etc."] gguf_out --> llamacpp["llama.cpp, Ollama, LM Studio"] hub --> onnx_exp["optimum-cli export onnx / ORT GenAI builder / Olive"] onnx_exp --> onnx_out["ONNX (fp16 / int4)"] onnx_out --> ort["ONNX Runtime, Transformers.js"] hub --> ct["torch.export or trace, then coremltools.convert"] ct --> ctq["coremltools.optimize"] ctq --> mlpkg["Core ML .mlpackage"] hub --> coreai["torch.export, then coreai-torch"] coreai --> aimodel["Core AI .aimodel"] hub --> litert["litert-torch"] litert --> tflite[".tflite / .litertlm"] tflite --> litertrt["LiteRT, LiteRT-LM"] hub --> mlxc["mlx_lm.convert -q"] mlxc --> mlxout["MLX safetensors (4-bit)"] hub --> et["torch.export, then ExecuTorch lowering or optimum-executorch"] et --> pte["ExecuTorch .pte"] hub --> gpuq["bitsandbytes / GPTQModel / llm-compressor"] gpuq --> gpuout["NF4, GPTQ, FP8 checkpoints"] gpuout --> vllm["transformers, vLLM, SGLang"]
Figure 1. Conversion and quantization pipelines from a Hub checkpoint

The same choice as a table, from the runtime back to the tool:

Target runtime Artefact Tool that produces it

llama.cpp, Ollama, LM Studio

.gguf

convert_hf_to_gguf.py plus llama-quantize

ONNX Runtime, Transformers.js

.onnx

optimum-cli export onnx, the ORT GenAI model builder, Olive

Core ML (Apple, all chips)

.mlpackage

coremltools.convert, then coremltools.optimize

Core AI (iOS and macOS 27 and later)

.aimodel

coreai-torch on a torch.export program

LiteRT, LiteRT-LM (Android, iOS, web)

.tflite, .litertlm

litert-torch

MLX (Apple silicon)

MLX safetensors

mlx_lm.convert

ExecuTorch (Android, iOS, microcontrollers)

.pte

torch.export with the ExecuTorch lowering API, or optimum-executorch

vLLM, SGLang, transformers on a GPU

NF4, GPTQ, FP8 safetensors

bitsandbytes, GPTQModel, llm-compressor

GGUF

llama.cpp ships a Python converter and a C++ quantizer. The flow is two steps, with an optional calibration step in between: convert the checkpoint to a high-precision GGUF, optionally measure an importance matrix, then quantize.

From a Hub checkpoint to a Q4_K_M GGUF
hf download Qwen/Qwen3-0.6B --local-dir ./Qwen3-0.6B

# 1. Convert: safetensors -> 16-bit GGUF (run from a llama.cpp checkout)
python convert_hf_to_gguf.py ./Qwen3-0.6B --outfile qwen3-0.6b-bf16.gguf --outtype bf16

# 2. Quantize: 16-bit GGUF -> Q4_K_M
./llama-quantize qwen3-0.6b-bf16.gguf qwen3-0.6b-q4_k_m.gguf Q4_K_M

--outtype auto keeps the precision the checkpoint was stored in, which is what you want when the model is already 16-bit. The converter needs a model architecture that llama.cpp implements; an unsupported architecture fails at this step, and the fix is a newer llama.cpp, not a flag.

An importance matrix (imatrix) records which weights matter most on a calibration text; the quantizer then spends its bits where they hurt least, which helps most at 4 bits and below. You measure it on the 16-bit GGUF and pass it to the quantizer:

Measuring and using an importance matrix
./llama-imatrix -m qwen3-0.6b-bf16.gguf -f calibration.txt -o imatrix.gguf -ngl 99
./llama-quantize --imatrix imatrix.gguf qwen3-0.6b-bf16.gguf qwen3-0.6b-q4_k_m.gguf Q4_K_M

Use calibration text that resembles what the model will see (chat, code, or your domain language); a mismatched calibration set can make a quant worse for your use case than the plain one. Quant type names and their size and quality trade-offs are on Quantization for Inference, and the runtime on llama.cpp.

If you would rather not run anything locally, the GGUF-my-repo Space (ggml-org/gguf-my-repo) runs the same conversion and quantization in the browser and pushes the result to your Hub account. It is convenient for one-off conversions of models you are allowed to redistribute, and it needs a Hub token with write access, so sign in only to the Space hosted under the ggml-org organisation.

References: llama.cpp — quantize, llama.cpp — imatrix and GGUF-my-repo.

ONNX

Three tools produce ONNX, and they differ in how much they do for you.

Optimum exports any supported transformers model to ONNX Runtime graphs. The exporter lives in the separate optimum-onnx package; it validates the exported graph against the PyTorch model and, for decoder models, includes the past-key-value cache by default.

Exporting a model with optimum-cli
pip install "optimum-onnx[onnxruntime]"
optimum-cli export onnx --model HuggingFaceTB/SmolLM2-360M-Instruct --dtype fp16 smollm2_onnx/

optimum-onnx 0.1.0, the latest release on PyPI on 2026-09-30, pins optimum~=2.1.0 and transformers>=4.36,<4.58, so installing it downgrades a transformers 5 environment; run the export in a separate virtual environment (for example python -m venv .venv-onnx) and keep the transformers 5 baseline in your main one.

Loading the export with ONNX Runtime through Optimum
from optimum.onnxruntime import ORTModelForCausalLM
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("smollm2_onnx")
model = ORTModelForCausalLM.from_pretrained("smollm2_onnx")
inputs = tokenizer("ONNX is", return_tensors="pt")
print(tokenizer.batch_decode(model.generate(**inputs, max_new_tokens=20)))

The ONNX Runtime GenAI model builder is the route for LLMs you will run with the onnxruntime-genai API (the generate loop, KV cache and tokenizer handling included). It exports and quantizes in one command and selects kernels for the execution provider you name:

Building an int4 model for ONNX Runtime GenAI
pip install onnxruntime-genai
python -m onnxruntime_genai.models.builder \
  -m HuggingFaceTB/SmolLM2-360M-Instruct -o ./smollm2-int4 -p int4 -e cpu -c ./cache

-p is the precision (fp32, fp16, bf16, int4, int8) and -e the execution provider (for example cpu, cuda, dml or webgpu).

Olive is Microsoft’s optimisation toolkit for ONNX Runtime: it chains conversion, graph optimisation and quantization and targets a device and execution provider, which makes it the tool for NPUs and mixed hardware.

Optimising to int4 with Olive
pip install olive-ai
olive optimize --model_name_or_path HuggingFaceTB/SmolLM2-360M-Instruct \
  --device cpu --provider CPUExecutionProvider --precision int4

Olive’s command set has been renamed before (auto-opt earlier, optimize in the documentation checked on the page date), so run olive --help against your installed version. For Transformers.js (see Model Formats) the onnx-community repositories already contain the multi-precision layout the library expects; mirror that layout if you publish your own.

References: Optimum ONNX — Export a model, ONNX Runtime GenAI model builder and Olive.

Core ML

coremltools converts a traced or exported PyTorch model to an ML Program (.mlpackage). Conversion is done at half precision by default for ML Programs; compression is a separate, post-training step in coremltools.optimize (linear 8-bit or 4-bit quantization, palettization, pruning).

Converting a PyTorch module and compressing its weights
import torch
import coremltools as ct
from coremltools.optimize.coreml import (
    OpLinearQuantizerConfig, OptimizationConfig, linear_quantize_weights,
)

model = MyModel().eval()                      # any torch.nn.Module
example = torch.rand(1, 3, 224, 224)
traced = torch.jit.trace(model, example)

mlmodel = ct.convert(
    traced,
    inputs=[ct.TensorType(name="image", shape=example.shape)],
    convert_to="mlprogram",
    minimum_deployment_target=ct.target.iOS17,
    compute_precision=ct.precision.FLOAT16,
)

config = OptimizationConfig(global_config=OpLinearQuantizerConfig(mode="linear_symmetric"))
compressed = linear_quantize_weights(mlmodel, config=config)
compressed.save("Model.mlpackage")

Large language models need extra care (stateful KV cache, flexible shapes) that the generic converter does not give you; for those, start from an existing Apple or community conversion or from the conversion recipes in Apple’s Core ML documentation, and compare the output with the PyTorch model before shipping. On the device, Core ML chooses CPU, GPU or Neural Engine per layer. The Apple-side integration is on Apple Platforms; Apple’s own frameworks are also covered in Apple Intelligence and Machine Learning.

References: coremltools — Converting from PyTorch and coremltools — Quantization.

Core AI

Core AI is Apple’s newer on-device stack for iOS and macOS 27 and later; its model asset is the .aimodel. According to Apple’s documentation and the apple/coreai-torch repository, the authoring path is: export the PyTorch model with torch.export, hand the resulting ExportedProgram to the coreai-torch package, which lowers ATen operations into Core AI operations, optimise and compress the program (Apple publishes coreai-optimization for that), and save the .aimodel. Apple also publishes coreai-models (apple/coreai-models on GitHub), a gallery of export recipes and Swift runtime helpers for popular open models, where each model’s README gives the export command.

Installing the Core AI Python packages and starting from the export step
pip install coreai-torch coreai-core
The export step that every Core AI conversion starts from
import torch

model = MyModel().eval()
example = (torch.rand(1, 3, 224, 224),)
exported = torch.export.export(model, example)   # the ExportedProgram coreai-torch consumes

The converter classes, option names and compression API of coreai-torch are deliberately not reproduced here: the framework is new, the pages this sub-section could confirm on 2026-09-30 describe the workflow but not a stable signature, and the API may change between releases. Follow the coreai-torch documentation for the conversion call and the README of the model in coreai-models for a ready-made recipe. Because Core AI requires the newest operating systems, keep a Core ML .mlpackage (previous section) as the fallback for older devices.

References: Apple — Core AI, coreai-torch and coreai-models.

LiteRT and LiteRT-LM

LiteRT is the successor to TensorFlow Lite, and litert-torch is the successor to ai-edge-torch: it converts PyTorch models to .tflite for LiteRT on Android, iOS, embedded and web. For generative models it has a Hugging Face export path that writes a .litertlm bundle, a single file containing the quantized prefill and decode graphs, token embedder, tokenizer and LLM metadata, ready for the LiteRT-LM runtime on CPU, GPU or NPU.

Exporting a Hub model to a LiteRT-LM bundle
pip install litert-torch
litert-torch export_hf --model=google/gemma-3-270m-it --output_dir=./gemma3-270m-litertlm
The same export from Python, with a quantization recipe
from litert_torch.generative.export_hf import export

export.export(
    model="google/gemma-3-270m-it",          # Hub ID or local safetensors directory
    output_dir="./gemma3-270m-litertlm",
    quantization_recipe="dynamic_wi8_afp32",  # built-in recipe name or a JSON recipe path
)

Gated models such as Gemma need you to accept the licence on the Hub and authenticate (hf auth login) before the export can download them. MediaPipe’s LLM Inference API is in maintenance; new Android and cross-platform projects should target LiteRT-LM. Pre-converted bundles live under the litert-community organisation.

References: LiteRT — Convert PyTorch GenAI models and litert-torch.

MLX

mlx-lm converts and quantizes in one command and can push the result to the Hub, which is how the mlx-community repositories are made. The output is MLX-format safetensors for Apple silicon.

Converting to 4-bit MLX
pip install mlx-lm
mlx_lm.convert --hf-path Qwen/Qwen3-0.6B -q --q-bits 4 --q-group-size 64 --mlx-path ./qwen3-0.6b-4bit
Loading the result
from mlx_lm import load, generate

model, tokenizer = load("./qwen3-0.6b-4bit")
print(generate(model, tokenizer, prompt="MLX is", max_tokens=30))

-q turns quantization on; mlx_lm.convert --help lists the bit-width, group-size and upload options of your version. MLX quantization is weight-only with a group scale, the same idea as the GPTQ and AWQ group sizes on Quantization for Inference.

References: mlx-lm and mlx-community.

ExecuTorch

ExecuTorch is PyTorch’s on-device runtime. The pipeline exports the model with torch.export, lowers it to a backend through a partitioner (XNNPACK for CPUs, Core ML for iOS, Vulkan and Qualcomm for Android GPUs and NPUs), and serialises a .pte file.

Exporting a module to .pte with the XNNPACK backend
import torch
from executorch.backends.xnnpack.partition.xnnpack_partitioner import XnnpackPartitioner
from executorch.exir import to_edge_transform_and_lower

model = MyModel().eval()
example = (torch.rand(1, 3, 224, 224),)
exported = torch.export.export(model, example)
program = to_edge_transform_and_lower(exported, partitioner=[XnnpackPartitioner()]).to_executorch()

with open("model.pte", "wb") as f:
    f.write(program.buffer)

For Hub models, optimum-executorch wraps the export, including LLM-specific optimisations and quantization:

Exporting a Hub LLM with optimum-executorch
pip install optimum-executorch
optimum-cli export executorch --model HuggingFaceTB/SmolLM2-135M-Instruct --task text-generation \
  --recipe xnnpack --use_custom_sdpa --use_custom_kv_cache --qlinear 8da4w --qembedding 8w \
  --output_dir ./smollm2-executorch

The optimum-executorch README documents installing from source; check it for the current install route. --qlinear 8da4w means 8-bit dynamic activations with 4-bit weights for linear layers, and --qembedding 8w quantizes the embeddings to 8-bit weights. Pre-exported .pte files are published by the pytorch and software-mansion organisations; react-native-executorch (see React Native) consumes them.

References: ExecuTorch — Getting started and optimum-executorch.

GPU-server quantization

The tools above target devices. To serve a model from a GPU with transformers, vLLM or SGLang (see vLLM and SGLang) you use one of three libraries.

bitsandbytes quantizes at load time, with no calibration data and no separate step, through transformers. It is the basis of QLoRA (see Fine-tuning with PEFT and TRL).

Loading a model in 4-bit NF4
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B", quantization_config=bnb, device_map="auto")

GPTQModel runs calibrated GPTQ quantization (and other methods) and saves a checkpoint that transformers, vLLM and SGLang load directly:

Quantizing to 4-bit GPTQ
from gptqmodel import GPTQModel, GPTQConfig

calibration = [...]   # a few hundred representative text samples
model = GPTQModel.load("Qwen/Qwen3-0.6B", GPTQConfig(bits=4, group_size=128))
model.quantize(calibration, batch_size=1)
model.save("Qwen3-0.6B-gptq-4bit")

Source: GPTQModel README, "Quantization" (GPTQModel 7.5.0 on PyPI on 2026-09-30). GPTQConfig here is GPTQModel’s own class, imported from gptqmodel, not the transformers class of the same name; the README keeps QuantizeConfig as the broad factory and uses the concrete GPTQConfig, AWQConfig, FP8Config and similar classes for each method. Older GPTQModel releases passed a QuantizeConfig(bits=4, group_size=128) instead, so match the example to the version you install.

llm-compressor (the vLLM project’s library, which absorbed the deprecated AutoAWQ) applies weight, activation and KV-cache quantization and saves in the compressed-tensors format that vLLM reads. FP8 dynamic quantization needs no calibration data (the PyPI package is named llmcompressor, installed with pip install llmcompressor):

FP8 dynamic quantization with llm-compressor
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Qwen/Qwen3-0.6B"
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

recipe = QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"])
oneshot(model=model, recipe=recipe)
model.save_pretrained("Qwen3-0.6B-FP8-Dynamic")
tokenizer.save_pretrained("Qwen3-0.6B-FP8-Dynamic")

Pick bitsandbytes when you fine-tune or prototype, and GPTQModel or llm-compressor when you publish a checkpoint for a server. The formats and their quality trade-offs are on Quantization for Inference.

References: Transformers — bitsandbytes, GPTQModel and llm-compressor.

After converting

  • Compare the converted model with the original on a handful of prompts (or a small evaluation set) before you ship; conversions can silently change tokenizer, chat template or numerics.

  • Keep the tokenizer and chat template alongside the weights: GGUF embeds them, the other formats ship them as files.

  • Record the source revision, the tool version and the quantization settings in the model card if you publish it.

  • Publish with hf upload only if the source licence allows redistribution, and keep the original licence file.

References