Apple Platforms

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

This page was written on 2026-09-30 against iOS, iPadOS and macOS 27 with Xcode 27 (Core AI and the Foundation Models executor API need the 27 SDK), swift-transformers 1.3.4, mlx-swift-lm 3.31.4, argmax-oss-swift v1.1.0 (the project that was called WhisperKit; it now ships WhisperKit, TTSKit and SpeakerKit as products of one package), coremltools 9.0 and the llama.cpp release v0.5.0, as checked that day. It is the Apple chapter of the DocsAssistant goes offline scenario of this sub-section: transcribe a spoken question, answer it with a small local LLM, speak the answer, and classify or describe a screenshot, all on the device. It covers only how to bring Hub models to Apple platforms. Foundation Models basics (@Generable, tool calling, the system model, Core ML versus Foundation Models as a decision) are in Apple Intelligence and Machine Learning and are not repeated. Model formats are in Model Formats, producing Core ML or MLX weights is in Conversion & Quantization, and the models themselves are in Speech Models: STT & TTS, Vision Models and Small LLMs & Embeddings. No benchmark numbers or prices are quoted.

Nothing on this page was compiled or run on a device or in the simulator. The Swift snippets are adapted from the official documentation and READMEs linked after each of them. Treat them as illustrations to check against your SDK version, not as tested code. The Hub repositories named here were confirmed to exist on 2026-09-30, but licences can change, so check each one on its model card before shipping (see Choosing Models & Licences).

The shape of the integration

Apple gives you several runtimes, and each Hub model format lands on one of them. The Hub has Apple-ready repositories too (mlx-community for MLX, argmaxinc for Core ML builds of Whisper and Qwen3-TTS), so often no conversion is needed.

Figure 1. DocsAssistant on Apple platforms with no network

Table 1. Which Apple option fits which DocsAssistant task
Task	Preferred option	Alternative	Main cost
Transcribe a spoken question	WhisperKit (Core ML Whisper from `argmaxinc/whisperkit-coreml`)	Your own Core ML or Core AI conversion of a Hub ASR model	Model download on first use; choose the variant for the device class
Answer it	mlx-swift-lm `ChatSession` with a 4-bit model from `mlx-community`	Core AI language model in a Foundation Models session, or the llama.cpp XCFramework with a GGUF file	Weights must fit the memory budget of the oldest supported device
Speak the answer	TTSKit (Qwen3-TTS on Core ML)	A Kokoro port through Core ML or MLX (community work, hedged below)	Larger download than a small voice model; licence of the voice model
Classify or describe a screenshot	A converted image encoder (SigLIP 2) with Core ML and the Vision framework	A small vision-language model such as FastVLM through MLX (check its licence)	Preprocessing must match the model card exactly

Core ML versus Core AI

Core ML is the established runtime. It loads .mlmodel and .mlpackage files produced by coremltools, runs on the CPU, GPU and Neural Engine, works on every Apple platform and is the right choice for the existing catalogue of Core ML models, including classical (tree or tabular) models. Conversion from PyTorch with coremltools is covered in Conversion & Quantization.

Core AI is new in the 27 OS releases. Apple documents it as the framework to "build, run, and deploy AI models" on Apple silicon, with the Swift API giving you control over specialization, caching and inference performance. You start from an .aimodel file, which you get by converting a PyTorch model with the Core AI PyTorch Extensions Python package (after preparing it with Core AI Optimization), or by exporting a model from Apple’s open-source coreai-models registry. The three Swift types to know, as named in Apple’s documentation:

AIModelAsset is the unspecialized source model asset.
AIModel is a model specialized for the device; creating one is asynchronous because specialization (choosing among CPU, GPU and Neural Engine) must finish first, and it can take a while for large models.
InferenceFunction is one callable function of the model. AIModel.loadFunction(named:) returns it, and run(inputs:) feeds it NDArray tensors or pixel buffers.

Loading a Core AI model and inspecting its function

import CoreAI

// Specialize the model for this device and load it (asynchronous, can be slow).
let model = try await AIModel(contentsOf: urlOfModel)

// Load a function from the model.
guard let function = try model.loadFunction(named: "main") else {
    // Handle the case where the expected function is not found.
    return
}

// Check the input signature before feeding data.
let descriptor = function.descriptor
guard let input = descriptor.inputDescriptor(of: "input"),
      case .ndArray(let array) = input, array.scalarType == .float32 else {
    return
}

Source: Apple: Integrating on-device AI models in your app with Core AI. The function name "main" and the input name are placeholders; read the real signature in Xcode’s model viewer.

Two practical points from the same documentation. Builds that include an .aimodel file need the Metal Toolchain component installed in Xcode, otherwise they fail with a missing Metal compiler error. And Apple lets you compile models ahead of time with the coreai-build tool to cut on-device specialization time (see Compiling Core AI models ahead of time). The conversion tooling is Apple’s, not the Hub’s: Core AI PyTorch Extensions produces the .aimodel, while for Core ML you use coremltools.

Use Core ML when you must support OS versions before 27 or watchOS, or already have a .mlpackage; choose Core AI when the minimum deployment target is 27 and you want modern architectures and Apple’s tooling (debugger, instrument, registry of exportable models). The broader comparison lives in Apple Intelligence and Machine Learning.

Loading a Core ML image encoder for screenshots (classification task)

import CoreML
import Vision

func makeScreenshotRequest() throws -> VNCoreMLRequest {
    let config = MLModelConfiguration()
    config.computeUnits = .all
    // "ScreenshotEncoder" is a Core ML model you converted from a Hub vision model.
    let url = Bundle.main.url(forResource: "ScreenshotEncoder", withExtension: "mlmodelc")!
    let model = try VNCoreMLModel(for: MLModel(contentsOf: url, configuration: config))
    let request = VNCoreMLRequest(model: model)
    request.imageCropAndScaleOption = .scaleFill
    return request
}

func embedding(of screenshot: CGImage) throws -> [Float]? {
    let request = try makeScreenshotRequest()
    try VNImageRequestHandler(cgImage: screenshot).perform([request])
    guard let observation = request.results?.first as? VNCoreMLFeatureValueObservation,
          let array = observation.featureValue.multiArrayValue else { return nil }
    return (0..<array.count).map { array[$0].floatValue }
}

Source: Apple: VNCoreMLRequest. With a SigLIP 2 style model you convert the image encoder to Core ML, precompute the text embeddings of your candidate labels once, and pick the label with the highest cosine similarity; the preprocessing (size, mean, standard deviation) must match the model card, see Vision Models. Capturing images is covered in Camera, Media and Photos.

swift-transformers: tokenizers and the Hub

swift-transformers (Apache 2.0, 1.3.4 as of this page) is Hugging Face’s Swift package. Its modules are Tokenizers (tokenization with chat-template support), Hub (model downloads with progress), Generation (language-model utilities) and Models (Core ML integration tools). It does not run a model by itself; it supplies the parts around a runtime. Core ML language-model pipelines, MLX and your own code use it so that a Swift app tokenizes text exactly as the Python transformers library does.

Tokenizing with a chat template and downloading a snapshot with swift-transformers

import Hub
import Tokenizers

let tokenizer = try await AutoTokenizer.from(pretrained: "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B")
let messages = [["role": "user", "content": "Describe the Swift programming language."]]
let encoded = try tokenizer.applyChatTemplate(messages: messages)

let repo = Hub.Repo(id: "mlx-community/Qwen2.5-0.5B-Instruct-2bit-mlx")
let modelDirectory = try await Hub.snapshot(from: repo, matching: ["config.json", "*.safetensors"])

Source: swift-transformers README. The repository identifiers are the README’s own examples. For private or gated repositories you supply a token from the environment or the keychain ($HF_TOKEN on a build machine); never embed one in the app binary. The same package hosts the download layer that mlx-swift-lm uses, see the next section.

MLX and mlx-swift-lm

mlx-swift-lm is the Swift package for running large language models and vision language models with MLX on Apple silicon (3.31.4 as of this page). It replaces the LLM libraries that used to live in mlx-swift-examples; that repository is now a set of example applications and tools that depend on it. Version 3 is a major release that decouples the package from any particular tokenizer or downloader, so you add an integration. The convenient one is the MLXHuggingFace product, which uses swift-huggingface for downloads and swift-transformers' Tokenizers for tokenization. Other products of the same package cover embeddings (MLXEmbedders), vision language models (MLXVLM) and grammar-constrained generation (MLXGuidedGeneration). MLX weights come from the Hub, mostly under the mlx-community organisation, see Model Formats.

Package.swift: mlx-swift-lm with the Hugging Face integration

dependencies: [
    .package(url: "https://github.com/ml-explore/mlx-swift-lm", .upToNextMajor(from: "3.31.3")),
    .package(url: "https://github.com/huggingface/swift-huggingface", from: "0.9.0"),
    .package(url: "https://github.com/huggingface/swift-transformers", from: "1.3.0"),
],
targets: [
    .target(
        name: "DocsAssistant",
        dependencies: [
            .product(name: "MLXLLM", package: "mlx-swift-lm"),
            .product(name: "MLXLMCommon", package: "mlx-swift-lm"),
            .product(name: "MLXHuggingFace", package: "mlx-swift-lm"),
            .product(name: "HuggingFace", package: "swift-huggingface"),
            .product(name: "Tokenizers", package: "swift-transformers"),
        ]),
]

Source: mlx-swift-lm README. The versions in the snippet are the README’s; use the current ones when you add the packages.

Answering a transcribed question with ChatSession

import HuggingFace
import MLXHuggingFace
import MLXLLM
import MLXLMCommon
import Tokenizers

// Downloads the weights on first use and keeps them in the local cache.
let model = try await #huggingFaceLoadModelContainer(
    configuration: LLMRegistry.gemma3_1B_qat_4bit
)

let session = ChatSession(model)
let answer = try await session.respond(to: "How do I rotate an API key in DocsAssistant?")
print(answer)
// A second call on the same session keeps the conversation context.
print(try await session.respond(to: "And for the staging environment?"))

Source: MLXLMCommon documentation and the README above. LLMRegistry.gemma3_1B_qat_4bit is the README’s example, a 4-bit quantization-aware-trained Gemma 3 1B build. For the DocsAssistant, pick from the small instruction-tuned models listed in Small LLMs & Embeddings; an mlx-community 4-bit build of a small model is the usual choice, but check the exact repository name and licence on the Hub first and look up how to register a model that is not in LLMRegistry in the package documentation. Models run under unified memory, so a chat model that fits the oldest device you support is a hard requirement (see the sizing section below). The Python side of MLX is in Desktop Runners and Quantization for Inference.

WhisperKit and TTSKit (argmax-oss-swift)

The WhisperKit repository was renamed argmax-oss-swift (MIT, v1.1.0 as of this page) and now holds three libraries as products of one Swift package: WhisperKit for speech-to-text, TTSKit for text-to-speech with Qwen3-TTS models, and SpeakerKit for speaker diarization. There is also an ArgmaxOSS umbrella product. The README lists macOS 14 and Xcode 16 as the prerequisites of the package, and macOS 15 or iOS 18 for TTSKit. Argmax also sells a separate Pro SDK with more models and features; this page covers only the open-source package. Whisper model variants are Core ML builds hosted on the Hub (argmaxinc/whisperkit-coreml); the compressed large-v3-turbo build is the one the README recommends for iOS and macOS, and tiny is for debugging. See Speech Models: STT & TTS for the model families.

Transcribing a spoken question with WhisperKit

import WhisperKit

func transcribe(_ audioURL: URL) async throws -> String {
    // Downloads this Core ML Whisper build on first use (pass no model to get the device's recommended default).
    let pipe = try await WhisperKit(WhisperKitConfig(model: "large-v3-v20240930_626MB"))
    let results = try await pipe.transcribe(audioPath: audioURL.path)
    return results.map(\.text).joined(separator: " ")
}

Source: argmax-oss-swift README. The model name is the README’s recommended variant; pass a different name, or a modelRepo of your own, to use a fine-tuned build. Recording the audio is covered in Camera, Media and Photos.

Speaking the answer with TTSKit

import TTSKit

func speak(_ text: String) async throws {
    // Downloads the default 0.6B model on first use; models load lazily.
    let tts = try await TTSKit()
    try await tts.play(text: text)   // streams audio while it is still being generated
}

Source: the TTSKit section of the argmax-oss-swift README. The 0.6B model runs on every platform TTSKit supports; the larger 1.7B model is macOS-only and adds style instructions. Both are Qwen3-TTS weights, so check their licence on the argmaxinc/ttskit-coreml model card. Kokoro-82M, the small permissive voice model discussed in Speech Models: STT & TTS, has community Core ML and MLX ports, but none was validated for this page; if you prefer Kokoro, plan to convert it yourself with the workflow in Conversion & Quantization and handle phonemization in Swift, which a Core ML graph does not include. The system speech synthesizer (AVSpeechSynthesizer) remains the zero-download fallback.

Running your own model inside Foundation Models

Foundation Models lets you use any language model through the same LanguageModelSession API you use for Apple’s on-device model, so prompting, streaming, tool calling and @Generable output carry over unchanged (see Apple Intelligence and Machine Learning for those basics). Apple’s documentation for the 27 SDK describes a protocol, LanguageModelExecutor, as "the bridge between the framework types and the system that actually generates the tokens, like a server API or a local inference engine". A LanguageModel pairs with exactly one executor type, and the framework creates the executor from the configuration the model provides. The executor’s main method receives the request (transcript plus GenerationOptions and ContextOptions) and streams deltas into a channel. Apple’s WWDC26 session 339, "Bring an LLM provider to the Foundation Models framework", covers writing one.

The documented shape of an executor (method body elided)

func respond(
    to request: LanguageModelExecutorGenerationRequest,
    model: MyLanguageModel,
    streamingInto channel: LanguageModelExecutorGenerationChannel
) async throws {
    // Convert the transcript to your model's prompt format, apply the
    // generation options, and stream incremental events into `channel`.
}

Source: Apple: LanguageModelExecutor. This is the documented signature only; a complete conforming type also needs a Configuration, an initializer and the LanguageModel that names it, which this page does not reproduce. Most apps never write one, because two ready-made LanguageModel implementations already exist and have the same effect:

CoreAILanguageModel, from Apple’s open-source coreai-models Swift package, loads a Core AI export of an open model (the registry lists the exportable models and their Hub identifiers) and is passed to LanguageModelSession(model:).
MLXLanguageModel, from the MLXFoundationModels product of mlx-swift-lm, bridges an MLX model into the same API and can combine with grammar-constrained generation for @Generable output. It requires the 27 SDK.

An MLX model behind a Foundation Models session

import FoundationModels
import HuggingFace
import MLXFoundationModels
import MLXHuggingFace
import MLXLLM
import MLXLMCommon
import Tokenizers

if #available(iOS 27.0, macOS 27.0, visionOS 27.0, *) {
    let model = #huggingFaceLanguageModel(
        configuration: LLMRegistry.gemma3_1B_qat_4bit,
        capabilities: [.guidedGeneration])
    let session = LanguageModelSession(model: model)
    let reply = try await session.respond(to: "Summarize the DocsAssistant setup steps.")
    print(reply.content)
}

Source: mlx-swift-lm README (the FoundationModels integration section). The Core AI variant, including the export commands, is in Apple: Running a Core AI model in a Foundation Models session. Use this route when you want one session API across Apple’s model, a bundled open model and a fallback for devices that cannot run Apple Intelligence.

The llama.cpp XCFramework

If you already ship GGUF models (see Model Formats and llama.cpp), the llama.cpp project documents a precompiled XCFramework for iOS, visionOS, tvOS and macOS, consumable from Swift Package Manager as a binary target that points at a zip published with a llama.cpp release. The upstream example pins an old intermediate build; the release tag scheme has since changed (the latest release on 2026-09-30 was v0.5.0 and its asset list did not show an XCFramework zip), so before depending on it, open the releases page, find the release you want and confirm that it carries the XCFramework asset, then copy the asset’s URL and compute its checksum.

Package.swift: the llama.cpp XCFramework as a binary target

// swift-tools-version: 5.10
import PackageDescription

let package = Package(
    name: "MyLlamaPackage",
    targets: [
        .executableTarget(name: "MyLlamaPackage", dependencies: ["LlamaFramework"]),
        .binaryTarget(
            name: "LlamaFramework",
            url: "{url of the xcframework zip on the llama.cpp release you verified}",
            checksum: "{output of swift package compute-checksum on that zip}"
        ),
    ]
)

Source: llama.cpp: XCFramework. The URL and checksum are placeholders on purpose. The framework gives you the C API of llama.cpp, not a Swift chat abstraction, so you write or adopt a thin Swift wrapper; choose it when you need GGUF quantizations that MLX does not offer, or one code base across Apple and other platforms. It was not built or run for this page.

App size, memory and on-demand download

A model is the largest thing your app will ever load, so plan size and memory before choosing one. The weights of a model with parameters stored at bits per weight need about

before the key-value cache, activations and runtime overhead, which add to it. A 4-billion-parameter model at 4 bits per weight is therefore about 2 GB of weights, which is already a large share of the memory of a phone that also runs your UI and the rest of the system. The table in Hardware Sizing and the quantization trade-offs in Quantization for Inference apply unchanged. Measure on the oldest device you support with Instruments, see Performance and Instruments.

Bundling the weights makes the download of the app itself that much larger, so most apps ship a small binary and fetch models afterwards. There are three options, from simplest to most integrated:

Let the library download on first use, as mlx-swift-lm, WhisperKit and TTSKit do from the Hub. It needs no extra work, but the first launch needs the network and you depend on the Hub’s availability and on the model licence permitting redistribution.
Host the model yourself or use Apple-hosted Background Assets, where the system downloads asset packs with a policy you set in the pack manifest: essential (before launch), prefetch (without blocking launch) or on demand. Apple-hosted packs are uploaded to App Store Connect; self-hosted ones use the framework’s unmanaged APIs.
Bundle the weights when the feature must work from the very first launch offline, and accept the app size.

Downloading a model asset pack on demand with Background Assets

import BackgroundAssets

func modelFolderReady() async throws {
    let pack = try await AssetPackManager.shared.assetPack(withID: "DocsAssistantLLM")
    // Initiates the download for an on-demand pack and waits for it to finish.
    try await AssetPackManager.shared.ensureLocalAvailability(of: pack)
    // Files are addressed by their path relative to the folder you packaged.
    let weights = try AssetPackManager.shared.contents(at: "llm/model.safetensors")
    _ = weights
}

func removeModel() async throws {
    // The system never removes packs on its own while the app is installed.
    try await AssetPackManager.shared.remove(assetPackWithID: "DocsAssistantLLM")
}

Source: Apple: Downloading Apple-hosted asset packs. The pack identifier and file path are placeholders. Project setup needs a managed Background Download extension target, a shared app group and the BAHasManagedAssetPacks and BAUsesAppleHosting Info.plist keys, and you can test locally with the mock server that ships with Xcode, see Testing asset packs locally. Memory-mapped reads (the default of contents(at:)) suit large weight files. A library that loads weights by directory, such as mlx-swift-lm, needs a file URL: AssetPackManager.url(for:) returns a URL for a relative path inside the downloaded packs (do not persist it beyond the current process), and the mlx-swift-lm documentation explains how to point the loader at local weights (using mlx-swift-lm). The combination was not run for this page. Apple also restricts what Background Assets may be used for, namely only downloading additional assets for the app. Related topics are in Background Execution and App Store Connect, TestFlight and Review.

Describing a screenshot with a vision-language model

Classification with an image encoder answers "which of these labels", not "what is on the screen". For a free-text description you need a vision-language model. mlx-swift-lm provides MLXVLM for such models, and Apple’s apple/FastVLM-0.5B (paper: FastVLM, CVPR 2025) is a small candidate designed for fast encoding of high-resolution images. Its licence on the model card is apple-amlr, Apple’s own research licence, not an OSI-approved open-source licence, so read it before using the model in a product; see Choosing Models & Licences and Vision Models for alternatives such as SmolVLM2. No code is shown here because the image-input API of ChatSession and the registration of FastVLM were not confirmed against the documentation for this page; follow the MLXVLM documentation linked in the references.

What could not be validated

For the record, no code on this page was built, run on a device or run in the simulator. The Hub repositories named (argmaxinc/whisperkit-coreml, argmaxinc/ttskit-coreml, apple/FastVLM-0.5B) were confirmed to exist on 2026-09-30 and the WhisperKit variant name in the snippet exists in argmaxinc/whisperkit-coreml, but the files and licences must still be re-read on the model cards (the argmaxinc/ttskit-coreml card carries no licence tag in the Hub metadata). The latest llama.cpp release (v0.5.0) lists no XCFramework asset, which is why the snippet uses placeholders. Specifically unvalidated: the Swift snippets in every section (they are adapted from Apple documentation and project READMEs only), Kokoro on Core ML or MLX, the use of Background Assets content as the weight directory for mlx-swift-lm, and FastVLM on MLX. The names AIModelAsset, AIModel, InferenceFunction, LanguageModelExecutor, CoreAILanguageModel, AssetPackManager, ChatSession, WhisperKit and TTSKit were confirmed in the official pages linked below on 2026-09-30 (the Apple names, and the two Apple quotations on this page, in Apple’s developer.apple.com documentation), but the signatures may still change between SDK and package releases.