Android

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

This page was written on 2026-09-30 against LiteRT 2.2.0 (com.google.ai.edge.litert:litert), LiteRT-LM 0.17.1 (com.google.ai.edge.litertlm:litertlm-android), ML Kit GenAI Prompt API 1.0.0-beta4, ONNX Runtime 1.30.0 (onnxruntime-android on Maven Central), ExecuTorch 1.4.0 (org.pytorch:executorch-android) and the Play Asset Delivery library 2.3.0, as those artifacts were listed in their Maven repositories that day. Kotlin is the language of every snippet. An Android app cannot simply call the Python transformers library, so a Hub model reaches a phone in one of two ways: converted to a mobile runtime format and executed by a runtime that ships inside the APK (LiteRT, LiteRT-LM, ONNX Runtime, ExecuTorch, llama.cpp), or served by the operating system itself (Gemini Nano through AICore). This page walks through each option for the DocsAssistant goes offline scenario of this sub-section: transcribe a spoken question, answer it with a small local model, speak the answer, and classify or describe a screenshot. The formats are in Model Formats; producing .tflite, .litertlm, ONNX and .pte files is in Conversion & Quantization; the models are in Speech Models: STT & TTS, Vision Models and Small LLMs & Embeddings. The same scenario on a server is in Python Backends and Java & Spring Boot. The Apple, Windows, web and hosted counterparts are Apple Platforms, Windows & .NET, Web & Cross-Platform and Inference Providers & Spaces. No benchmark numbers or prices are quoted.

The shape of the integration

Android has no single machine-learning runtime that runs every Hub format. Google maintains LiteRT (and LiteRT-LM on top of it) for its own model families, Microsoft maintains ONNX Runtime, the PyTorch project maintains ExecuTorch, and the llama.cpp and whisper.cpp projects ship Android example apps that wrap their native libraries through JNI. The picture below shows where each one sits in the offline DocsAssistant.

flowchart LR M["Microphone
AudioRecord, 16 kHz PCM"] --> T["Transcribe
whisper.cpp or Moonshine on LiteRT"] T --> A["Answer
LiteRT-LM with Gemma 4 E2B
or Gemini Nano via AICore"] A --> S["Speak
Android TextToSpeech
or Kokoro and Piper via ONNX Runtime"] C["Screenshot bitmap"] --> V["Classify or describe
LiteRT image classifier
or SigLIP 2 via ONNX Runtime"] V --> A A --> U["Compose UI"] S --> U D[("Model files
AI pack or on-demand download")] --> T D --> A D --> V
Figure 1. DocsAssistant goes offline on Android
Table 1. Which Android option fits which DocsAssistant task
Task Preferred option Alternative Main cost

Transcribe a spoken question

whisper.cpp through JNI, using the layout of its whisper.android example

A Moonshine or Whisper-class model converted to LiteRT or ONNX

Audio capture and resampling to 16 kHz mono floats are yours to write

Answer it

LiteRT-LM with a .litertlm bundle such as Gemma 4 E2B

Gemini Nano through ML Kit GenAI (no model file to ship, limited device list)

A download measured in gigabytes; or device and feature availability you do not control

Speak the answer

Android TextToSpeech (system engine, offline voices depend on the device)

Kokoro or Piper exported to ONNX and run with ONNX Runtime

Phonemization and voice packs are not inside the ONNX file

Classify or describe a screenshot

A LiteRT image classifier run with CompiledModel

SigLIP 2 ONNX through ONNX Runtime, or a multimodal LLM bundle

Preprocessing must match the model card exactly

Run a PyTorch model that has no LiteRT path

ExecuTorch with a .pte file

ONNX Runtime Mobile

Smaller model zoo on the Hub than GGUF or ONNX

Memory sizing before you pick a model

A phone shares RAM with the operating system, the foreground app and every background process, and Android kills apps that exceed what it tolerates. The memory needed for a language model has three parts. With parameters stored at bits per weight, a context of tokens, layers, key-value heads of dimension and a cache element size of bytes, the working set is approximately

The first term is the download and the resident weights (memory-mapped bundles can be paged, but do not count on it), the second grows linearly with the context length you allow, and covers activations, the tokenizer and the runtime itself. The two-times factor is one key tensor and one value tensor per layer. Gemma 4 E2B illustrates why "effective" parameters matter: its per-layer embeddings make the bundle larger than the compute-relevant size, see Small LLMs & Embeddings for the catalogue and Hardware Sizing and Quantization for Inference for the full budget. Read the model card for the real , and , pick the context length your app needs rather than the maximum, and measure on a mid-range device rather than the newest flagship.

Reading the available memory on a device
import android.app.ActivityManager
import android.content.Context

fun memoryBudgetBytes(context: Context): Long {
    val am = context.getSystemService(Context.ACTIVITY_SERVICE) as ActivityManager
    val info = ActivityManager.MemoryInfo().also { am.getMemoryInfo(it) }
    // totalMem is the device RAM; availMem is what is free right now; lowMemory flags pressure.
    return if (info.lowMemory) 0L else info.availMem
}

Reference: ActivityManager.getMemoryInfo. Check the budget before offering to download or load a large model, and offer the smaller model when it does not fit.

LiteRT 2.x and the CompiledModel API

LiteRT is the new name of TensorFlow Lite: the runtime was renamed, the repository is google-ai-edge/LiteRT, and the documentation moved from ai.google.dev/edge to developers.google.com/edge. Model files keep the .tflite extension. The 2.x line is built around the CompiledModel API, which replaces the older Interpreter API with explicit input and output buffers and a list of preferred accelerators. The PyTorch conversion tool that was called ai-edge-torch is now litert-torch, covered in Conversion & Quantization.

Adding LiteRT to an app module
dependencies {
    implementation("com.google.ai.edge.litert:litert:2.2.0")
}

Reference: LiteRT on Android. The minimum SDK level stated there is 23. Gradle setup in general is in Project Structure & Gradle.

The CompiledModel workflow is: create the model with a list of accelerators, allocate input and output buffers, write the input, run and read the output. LiteRT falls back down the list when an accelerator is not available, so the same code runs on devices without a usable GPU or NPU.

Classifying a screenshot with CompiledModel (the "classify" step of DocsAssistant)
import android.content.Context
import com.google.ai.edge.litert.Accelerator
import com.google.ai.edge.litert.CompiledModel

class ScreenshotClassifier(context: Context) {
    // "screen_classifier.tflite" is a converted Hub image classifier kept in app assets
    private val model = CompiledModel.create(
        context.assets,
        "screen_classifier.tflite",
        CompiledModel.Options(Accelerator.GPU, Accelerator.CPU),
    )
    private val inputs = model.createInputBuffers()
    private val outputs = model.createOutputBuffers()

    fun classify(pixels: FloatArray): FloatArray {
        inputs[0].writeFloat(pixels)   // preprocessed exactly as the model card specifies
        model.run(inputs, outputs)
        return outputs[0].readFloat()  // class scores; take the arg-max
    }

    fun close() = model.close()
}

Reference: LiteRT on Android. The documentation shows CompiledModel.create(path, CompiledModel.Options(Accelerator.CPU)) with createInputBuffers(), createOutputBuffers(), writeFloat(), run() and readFloat(); the asset-based overload and the list of several accelerators are shown on the NPU page below.

GPU and NPU acceleration

The accelerators are Accelerator.CPU (the default), Accelerator.GPU and Accelerator.NPU. List them in order of preference. The NPU path supports Qualcomm and MediaTek hardware (the LiteRT documentation lists further vendors, so check the current support matrix for your target chips), and LiteRT offers two compilation strategies: ahead-of-time compilation, best for large models whose target chip is known, which lowers start-up cost and memory, and on-device compilation, which needs no preparation but pays a higher first-run cost. Distribution of NPU-specific model variants and runtime libraries is done through Play for On-device AI, described later on this page.

Preferring the NPU and falling back to the GPU
val model = CompiledModel.create(
    context.assets,
    "model/mymodel.tflite",
    CompiledModel.Options(Accelerator.NPU, Accelerator.GPU),
)

Reference: LiteRT — NPU acceleration. Older material describes a GPU delegate for the Interpreter API (GpuDelegate, CompatibilityList) and a Play services variant of the runtime; that is the pre-2.x route and is not needed for new CompiledModel code.

LiteRT-LM for language models

LiteRT-LM is Google’s runtime for generative models on top of LiteRT. It loads a single .litertlm file that bundles the quantized graphs, tokenizer and model metadata, and runs it on the CPU, GPU or NPU. The Kotlin API targets Android and the JVM. Pre-converted bundles are published under the litert-community organisation on the Hub. The repository litert-community/gemma-4-E2B-it-litert-lm holds the Gemma 4 E2B bundle (the catalogue page lists it with the Gemma 4 licence; on 2026-09-30 the Hub listed the repository as Apache-2.0 and not gated, and it is the same ID used in Small LLMs & Embeddings; re-read the model card before shipping).

Downloading the bundle on a development machine
hf download litert-community/gemma-4-E2B-it-litert-lm --local-dir ./gemma-4-e2b
Adding LiteRT-LM to an app module
dependencies {
    implementation("com.google.ai.edge.litertlm:litertlm-android:0.17.1")
}

The API has an Engine configured with the model path and a Backend (Backend.CPU(), Backend.GPU() or Backend.NPU(nativeLibraryDir = …​)), and Conversation objects created from it. Initialisation can take several seconds, so it belongs on a background dispatcher, never on the main thread.

The "answer" step of DocsAssistant with LiteRT-LM and Gemma 4 E2B
import com.google.ai.edge.litertlm.Backend
import com.google.ai.edge.litertlm.Engine
import com.google.ai.edge.litertlm.EngineConfig
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.flow.Flow
import kotlinx.coroutines.flow.map
import kotlinx.coroutines.withContext
import java.io.File

class LocalAnswerer(modelFile: File) {
    private val engine = Engine(
        EngineConfig(modelPath = modelFile.absolutePath, backend = Backend.GPU()),
    )

    // Loading the model takes seconds: call once, off the main thread.
    suspend fun start() = withContext(Dispatchers.Default) { engine.initialize() }

    // Streams the answer token by token; collect it from a ViewModel.
    fun answer(question: String, passages: String): Flow<String> {
        val conversation = engine.createConversation()
        val prompt = "Answer using only these documentation passages:\n$passages\n\nQuestion: $question"
        return conversation.sendMessageAsync(prompt).map { it.toString() }
    }
}

Reference: LiteRT-LM — Android (Kotlin) and the LiteRT-LM repository. The guide shows EngineConfig(modelPath, backend), engine.initialize(), engine.createConversation(conversationConfig), conversation.sendMessage(…​) and conversation.sendMessageAsync(…​) as a Flow. System prompt and sampling settings belong in a conversation configuration object; the snippet above relies on its defaults, so take the exact parameters from the guide for the version you pin. Close conversations and the engine when the screen is destroyed, because the weights stay resident until then.

Migrating from MediaPipe LLM Inference

The MediaPipe LLM Inference API (Android, iOS and web) is in maintenance-only mode: new features and optimisations go to LiteRT-LM. A project that uses com.google.mediapipe:tasks-genai keeps working, but new code should target LiteRT-LM: replace the MediaPipe task options with an EngineConfig, move to .litertlm bundles (see the export path in Conversion & Quantization) and use the streaming Conversation.sendMessageAsync call in place of the MediaPipe response callbacks. The other MediaPipe Tasks (vision, text and audio classifiers) are separate and are not covered by this notice.

Reference: MediaPipe LLM Inference API.

Gemini Nano with ML Kit GenAI and AICore

Gemini Nano is Google’s on-device model, served by the AICore system service. The app does not ship or download the weights: AICore manages distribution and updates and runs the model on the device’s accelerators. Apps reach it through the ML Kit GenAI APIs: Summarization, Proofreading, Rewriting and Image Description (beta), Speech Recognition (alpha) and the Prompt API (beta), which accepts custom text or multimodal prompts. Beta means the Prompt API is offered without an SLA or a deprecation policy, so keep a fallback path. Availability is limited to supported devices (Pixel, Samsung Galaxy, OnePlus and others) and changes over time; this page does not enumerate them. Check the device list linked from the ML Kit GenAI overview and always query the runtime status.

Adding the Prompt API
dependencies {
    implementation("com.google.mlkit:genai-prompt:1.0.0-beta4")
}
The "answer" step with Gemini Nano, falling back when the feature is unavailable
import com.google.mlkit.genai.common.FeatureStatus
import com.google.mlkit.genai.prompt.Generation

suspend fun answerWithGeminiNano(question: String, onChunk: (String) -> Unit): Boolean {
    val model = Generation.getClient()
    val status = model.checkStatus()
    // UNAVAILABLE, DOWNLOADABLE and DOWNLOADING all mean "not usable right now": return false so
    // the caller switches to the LiteRT-LM path. For DOWNLOADABLE you can also start
    // model.download().collect { ... } in the background and use Gemini Nano next time.
    if (status != FeatureStatus.AVAILABLE) return false
    model.generateContentStream(question).collect { chunk ->
        onChunk(chunk.candidates[0].text)
    }
    return true
}

Reference: ML Kit GenAI — Prompt API. The page requires API level 26 or higher. The documented calls include Generation.getClient(), checkStatus(), download() (a flow of started, progress, completed and failed events), generateContent(…​) and generateContentStream(…​); checkStatus() returns one of the FeatureStatus constants UNAVAILABLE, DOWNLOADABLE, DOWNLOADING or AVAILABLE (see the FeatureStatus reference), and the snippet only generates when it is AVAILABLE. The API is still in beta, so re-check the constants on upgrade. An overview of the Android side is in Gemini Nano on Android. Gemini Nano is Google’s model: nothing from the Hub runs through it, so it complements rather than replaces the Hub-based options on this page.

ONNX Runtime Mobile

ONNX Runtime runs ONNX graphs, the format most Transformers.js and optimum-onnx exports use, so it is the natural Android route for a Hub model that publishes ONNX weights: the SigLIP 2 classifier, Kokoro or Piper text-to-speech, and small embedding or ASR models. The Android package is com.microsoft.onnxruntime:onnxruntime-android.

Adding ONNX Runtime to an app module
dependencies {
    implementation("com.microsoft.onnxruntime:onnxruntime-android:1.30.0")
}
Running an ONNX model with an execution provider on Android
import ai.onnxruntime.OnnxTensor
import ai.onnxruntime.OrtEnvironment
import ai.onnxruntime.OrtSession
import java.nio.FloatBuffer

class OnnxModel(modelBytes: ByteArray) {
    private val env = OrtEnvironment.getEnvironment()
    private val session: OrtSession = env.createSession(
        modelBytes,
        OrtSession.SessionOptions().apply {
            addNnapi()   // optional: see the deprecation note in the text below
        },
    )

    fun run(input: FloatArray, shape: LongArray): FloatArray {
        OnnxTensor.createTensor(env, FloatBuffer.wrap(input), shape).use { tensor ->
            session.run(mapOf(session.inputNames.first() to tensor)).use { result ->
                val out = result[0].value as Array<FloatArray>
                return out[0]
            }
        }
    }
}

Reference: ONNX Runtime — Java API (the same API serves Android) and Execution providers. The shape and type of result[0].value depend on the model; read the graph outputs from session.outputInfo.

Execution providers select the hardware. The NNAPI execution provider needs Android 8.1 or higher. NNAPI itself is deprecated as of Android 15: the Android NDK documentation says it still works but expects most future devices to use the CPU backend, and recommends migrating performance-critical work to the GPU runtime of the LiteRT family or other maintained solutions (see NNAPI). The ONNX Runtime NNAPI page does not itself state the deprecation, so treat NNAPI as a legacy option and benchmark it against the plain CPU provider on your target devices. The QNN execution provider targets Qualcomm NPUs; its documentation focuses on Windows on Arm and lists few Android details, so check the current QNN EP page and whether a prebuilt Android package exists before designing around it. For LLMs, ONNX Runtime GenAI is the generation loop built on the runtime; its Android packaging is covered in its own documentation, see Java & Spring Boot for the JVM API. Producing ONNX files is in Conversion & Quantization.

ExecuTorch

ExecuTorch is PyTorch’s on-device runtime: models are exported to a .pte file with a backend such as XNNPACK, Vulkan or Qualcomm (see Conversion & Quantization for the export pipeline and optimum-executorch). On Android the runtime is an AAR with a Java and Kotlin API.

Adding ExecuTorch to an app module
dependencies {
    implementation("org.pytorch:executorch-android:1.4.0")
}
Loading a .pte file and running its forward method
import org.pytorch.executorch.EValue
import org.pytorch.executorch.Module
import org.pytorch.executorch.Tensor

class PteModel(ptePath: String) {
    private val module = Module.load(ptePath)   // a file on disk, for example from an AI pack

    fun classify(chw: FloatArray): FloatArray {
        val input = Tensor.fromBlob(chw, longArrayOf(1, 3, 224, 224))
        val output = module.forward(EValue.from(input))[0].toTensor()
        return output.dataAsFloatArray
    }
}

Reference: ExecuTorch — Using ExecuTorch on Android. The documentation shows Module.load, Tensor.fromBlob, EValue.from and forward; the output accessor names are taken from the Java API and should be checked against the reference for your version. For large language models the project points to the LlamaDemo app in the meta-pytorch/executorch-examples repository rather than documenting a separate artifact.

llama.cpp and whisper.cpp on Android

Both projects ship a native library and an Android example app that loads a GGUF (or ggml) file through JNI. They are the route when the model is published as GGUF, which is the widest-published format on the Hub (see llama.cpp and Model Formats). The examples are starting points to copy, not published libraries with a stable API, so pin a commit.

Table 2. Where the Android examples live
Project Example directory What it shows

llama.cpp

examples/llama.android in ggml-org/llama.cpp

A Gradle project with app and lib modules that wraps the native library for chat

whisper.cpp

examples/whisper.android and examples/whisper.android.java in ggml-org/whisper.cpp

The same JNI wrapper pattern in Kotlin and in Java, for speech-to-text

The whisper.android library exposes a WhisperContext class created from a file or from an asset and a suspending transcribeData function that takes 16 kHz mono float samples. The "transcribe" step of DocsAssistant is then audio capture plus one call.

The "transcribe" step with the whisper.cpp Android example library
import android.annotation.SuppressLint
import android.media.AudioFormat
import android.media.AudioRecord
import android.media.MediaRecorder
import com.whispercpp.whisper.WhisperContext

// Requires the RECORD_AUDIO permission to have been granted at run time.
@SuppressLint("MissingPermission")
suspend fun transcribe(whisper: WhisperContext, seconds: Int = 8): String {
    val rate = 16_000
    val shorts = ShortArray(rate * seconds)
    val recorder = AudioRecord(
        MediaRecorder.AudioSource.MIC, rate,
        AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT, shorts.size * 2,
    )
    recorder.startRecording()
    var read = 0
    while (read < shorts.size) {
        val n = recorder.read(shorts, read, shorts.size - read)
        if (n <= 0) break
        read += n
    }
    recorder.stop(); recorder.release()

    val floats = FloatArray(read) { shorts[it] / 32768f }
    return whisper.transcribeData(floats, printTimestamp = false)
}

Reference: whisper.cpp — whisper.android example. Pick the model by size: a ggml Whisper model is published in several precisions by the whisper.cpp author on the Hub, and the choice trades accuracy against the memory formula above, see Speech Models: STT & TTS. A Moonshine model converted to LiteRT or ONNX is the alternative for short utterances; its Android packaging was not verified for this page, so read the current Moonshine repository before committing to it. The capture and permission side is in Device Features: Camera, Location, Sensors & Media.

Speaking the answer

Android ships a text-to-speech engine, so the zero-download option for the "speak" step is the system TextToSpeech class. Offline availability of a voice depends on the device and the installed engine. When you need a consistent, Hub-sourced voice, run Kokoro or Piper exported to ONNX with the ONNX Runtime code above; both need a phonemizer in front of the model, which is not part of the ONNX file, and Piper’s current repository is under a GPL-3.0 licence, see Speech Models: STT & TTS.

The "speak" step with the system TextToSpeech engine
import android.content.Context
import android.speech.tts.TextToSpeech
import java.util.Locale

class Speaker(context: Context) {
    private var ready = false
    private val tts: TextToSpeech = TextToSpeech(context) { status ->
        if (status == TextToSpeech.SUCCESS) {
            ready = tts.setLanguage(Locale.US) >= TextToSpeech.LANG_AVAILABLE
        }
    }

    fun speak(text: String) {
        if (ready) tts.speak(text, TextToSpeech.QUEUE_FLUSH, null, "docs-answer")
    }

    fun close() { tts.stop(); tts.shutdown() }
}

Reference: TextToSpeech API reference.

Delivering model files

Weights of hundreds of megabytes to several gigabytes do not belong in the base APK. Three mechanisms exist, and they can be mixed.

Table 3. Model delivery options
Mechanism How it works Use when

Play for On-device AI (beta)

Models go into AI packs inside the app bundle, delivered install-time, fast-follow or on-demand, with Play’s test tracks and staged rollouts; device-targeted variants are possible

The model is large and you publish on Google Play; also the documented channel for NPU-specific variants

Play Asset Delivery

Asset packs with the same three modes; install-time packs ship as split APKs, fast-follow and on-demand packs do not count towards the initial install size

You already use asset packs, or need the same mechanism for non-model assets

Your own download

The app fetches the file (for example from the Hub’s resolve URL or your CDN) into app storage, verifies a checksum and resumes on failure

The app is not published on Google Play, or the model is user-selectable

Play for On-device AI is labelled beta on the Android developer site; check its current terms and limits there rather than relying on this page, because size limits are set by Google Play and change. The LiteRT NPU documentation describes the same mechanism for distributing NPU model variants and runtime libraries.

Fetching an on-demand pack and locating its files
import android.content.Context
import com.google.android.play.core.assetpacks.AssetPackManagerFactory
import com.google.android.play.core.ktx.requestFetch

suspend fun modelFile(context: Context): String? {
    val manager = AssetPackManagerFactory.getInstance(context)
    val pack = "gemma_model"                       // name declared in the asset pack's build.gradle
    manager.requestFetch(listOf(pack))             // suspends until the download is requested; observe progress with a listener
    return manager.getPackLocation(pack)?.assetsPath()?.let { "$it/gemma-4-E2B-it.litertlm" }
}

Reference: Play Asset Delivery and its integration guide. The documentation lists com.google.android.play:asset-delivery and asset-delivery-ktx at 2.3.0, requestFetch, registerListener for progress, getPackLocation (which returns null when the pack is not present) and assetsPath(). Install-time packs need about twice their size free on the device, whereas fast-follow and on-demand packs need only a few hundred extra megabytes. Packs of the on-demand kind must be fetched before getPackLocation returns a path, so show a progress screen and a "model not downloaded yet" state in the UI. Whatever the mechanism, verify the licence of a model before redistributing it inside your app, see Choosing Models & Licences.

Putting it together in a ViewModel

The pieces are independent, so a ViewModel only orchestrates them: transcribe, answer with a fallback, speak, and keep the heavy objects alive across configuration changes. Coroutine and lifecycle details are in ViewModel, Lifecycle & Coroutines.

One question end to end (the offline DocsAssistant loop)
import androidx.lifecycle.ViewModel
import androidx.lifecycle.viewModelScope
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.launch

class AssistantViewModel(
    private val whisper: WhisperContext,
    private val answerer: LocalAnswerer,
    private val speaker: Speaker,
) : ViewModel() {
    val answer = MutableStateFlow("")

    fun ask(passages: String) = viewModelScope.launch {
        val question = transcribe(whisper)
        val sb = StringBuilder()
        answerer.answer(question, passages).collect { sb.append(it); answer.value = sb.toString() }
        speaker.speak(sb.toString())
    }

    override fun onCleared() = speaker.close()
}

Reference: ViewModel overview. Model loading, downloads and long transcriptions should not be tied to an activity: for work that must survive the app going to the background see Background Work & Services, for store policy on large downloads and data-safety declarations see Publishing on Google Play, and for the microphone and storage permissions see Permissions, Privacy & Security. The wider Android documentation starts at Android.

Checklist for shipping a Hub model in an Android app

  • Read the model card: licence, gating, the intended use and the terms that travel with redistribution.

  • Prefer formats that cannot run code on load: .tflite, .litertlm, ONNX, .pte and GGUF are data files; do not load pickle checkpoints on device, and treat trust_remote_code as a server-side concern only.

  • Compute the memory budget with the formula above for the smallest device you support, and test on it.

  • Choose one accelerator order (for example NPU, GPU, CPU) and keep a CPU fallback.

  • Keep a fallback model or path: the feature may be unavailable (Gemini Nano on an unsupported device), the download may not have finished, or the device may be low on memory.

  • Deliver weights outside the base APK and show download state in the UI.

  • Pin runtime versions: LiteRT-LM, ML Kit GenAI Prompt and Play for On-device AI are young or in beta, and the llama.cpp and whisper.cpp Android examples are not versioned libraries.

  • Never embed a Hub token in the app; if a gated model must be fetched, do it from a server you control and pass the file on, using $HF_TOKEN only in CI or on that server.

References