Dynamic Resource Allocation and Devices

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production; see the section bibliography for the reference material consulted while preparing these pages.

CPU and memory are understood natively by the kubelet and scheduler. Everything else — GPUs, FPGAs, high-speed NICs, hardware security modules — reaches Pods through extension points. This page covers the two mechanisms, the older device plugin framework and the newer Dynamic Resource Allocation (DRA), and then RuntimeClass, which selects how a Pod’s containers are isolated rather than what hardware they get. It is written against Kubernetes 1.37; see the section index for the version baseline. The basic request/limit model is in Resource Management, and node placement in Scheduling and Disruptions.

The Bookshelf scenario rarely needs accelerators, so the examples here are an optional extension — for instance an embedding job for a recommendations feature. They need a node with the vendor driver installed; on kind use the example DRA driver from the official tutorial, which advertises fake devices.

Device plugins

A device plugin is a per-node gRPC service that registers with the kubelet and advertises an extended resource such as nvidia.com/gpu or amd.com/gpu. The scheduler treats it as an opaque integer counter; the kubelet asks the plugin to prepare the device (environment variables, device nodes, mounts) when the container starts. Containers request it under limits:

apiVersion: v1
kind: Pod
metadata:
  name: embedder
  namespace: bookshelf
spec:
  restartPolicy: Never
  containers:
    - name: embedder
      image: ghcr.io/example/bookshelf-embedder:1.0.0
      resources:
        limits:
          nvidia.com/gpu: 1        # whole units only, no fractions, no overcommit

Source: Kubernetes Docs — Device Plugins.

Limits of the model: devices are whole, interchangeable units, so you cannot ask for "a GPU with at least 40 GiB of memory", share one device between Pods except by plugin-specific tricks (time slicing, MIG partitions), or express topology such as "two GPUs on the same NVLink". Those gaps motivated DRA.

kubectl describe node gpu-node-1 | grep -A3 "Allocatable:"
kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\\.com/gpu

Source: Kubernetes Docs — Schedule GPUs.

Dynamic Resource Allocation

DRA is to devices what the PersistentVolumeClaim / StorageClass model is to storage (see Persistent Volumes and Storage Classes): workloads claim devices by characteristics, and the scheduler and a vendor DRA driver cooperate to allocate and prepare a concrete one. DRA reached stable (GA) in Kubernetes 1.34 under the resource.k8s.io/v1 API group; verify the API version against your cluster with kubectl api-resources --api-group=resource.k8s.io.

Object Role

ResourceSlice

Published by the driver on each node (or for network-attached pools): the inventory of devices with their attributes (model, memory, driver version) and capacities. Not written by users.

DeviceClass

Cluster-scoped, admin-defined category of device, with a selector and optional configuration. Comparable to a StorageClass.

ResourceClaim

A request for devices: which DeviceClass, how many, with which extra CEL selector. Allocated once, it can be shared by several Pods.

ResourceClaimTemplate

A template from which Kubernetes creates a per-Pod ResourceClaim, mirroring volume claim templates.

Selection uses CEL expressions evaluated against each device’s attributes and capacity, so a claim can say "a GPU with at least 40 GiB" without knowing node names.

apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: gpu.example.com
spec:
  selectors:
    - cel:
        expression: device.driver == "gpu.example.com"
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: large-gpu
  namespace: bookshelf
spec:
  spec:
    devices:
      requests:
        - name: gpu
          exactly:
            deviceClassName: gpu.example.com
            count: 1
            selectors:
              - cel:
                  expression: device.capacity["gpu.example.com"].memory.compareTo(quantity("40Gi")) >= 0
apiVersion: v1
kind: Pod
metadata:
  name: embedder
  namespace: bookshelf
spec:
  restartPolicy: Never
  resourceClaims:
    - name: gpu
      resourceClaimTemplateName: large-gpu
  containers:
    - name: embedder
      image: ghcr.io/example/bookshelf-embedder:1.0.0
      resources:
        claims:
          - name: gpu

Source: Kubernetes Docs — Dynamic Resource Allocation.

The field names above follow the resource.k8s.io/v1 schema at the time of writing; consult the API reference for your cluster’s exact shape, since DRA request syntax evolved across its beta releases.

kubectl get deviceclass
kubectl get resourceslices
kubectl get resourceclaims -n bookshelf
kubectl describe resourceclaim -n bookshelf
NAME              NODE          DRIVER            POOL          AGE
kind-worker-gpu   kind-worker   gpu.example.com   kind-worker   3m

Source: Kubernetes Docs — Allocate devices to workloads with DRA.

Trying it on kind

The official tutorial installs an example DRA driver that publishes fake devices, so the whole flow — ResourceSlice inventory, a claim, scheduling, and device preparation — can be observed without hardware. Create a kind cluster whose nodes enable container device interface (CDI) in containerd, install the driver chart, then apply the claim and Pod manifests above, adapted to the driver’s class name.

kind create cluster --name dra --image kindest/node:v1.37.0
# install the example driver as instructed by the tutorial, then:
kubectl get resourceslices
kubectl apply -f embedder.yaml
kubectl logs embedder -n bookshelf

Source: Kubernetes Docs — Install Drivers and Allocate Devices with DRA.

RuntimeClass

By default every Pod runs through the node’s default OCI runtime (runc), which shares the host kernel. A RuntimeClass names an alternative runtime handler configured in the node’s CRI implementation (containerd or CRI-O), such as gVisor (runsc, a user-space kernel) or Kata Containers (a lightweight VM per Pod). Pods opt in with runtimeClassName; the trade-off is stronger isolation for untrusted or multi-tenant code against some performance and compatibility cost, and possible Pod overhead (accounted via overhead.podFixed, see Resource Management). RuntimeClass can also carry a scheduling.nodeSelector and tolerations so sandboxed Pods land only on nodes that have the handler.

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
overhead:
  podFixed:
    memory: 64Mi
    cpu: 100m
scheduling:
  nodeSelector:
    sandbox.example.com/runtime: gvisor
apiVersion: v1
kind: Pod
metadata:
  name: untrusted-import
  namespace: bookshelf
spec:
  runtimeClassName: gvisor
  containers:
    - name: importer
      image: ghcr.io/example/bookshelf-importer:1.0.0

Source: Kubernetes Docs — Runtime Class.

Sandboxing complements, not replaces, the controls in Pod Security, and Pod-level isolation is a recurring theme in the multi-tenancy discussion in Admission Control and Policy.

References