Custom Resources and Operators

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production; see the section bibliography for the reference material consulted while preparing these pages.

Kubernetes is built so that its own API can be extended. Everything seen so far — Deployments, Services, Jobs — is a resource served by the API server and reconciled by a controller (Objects and the API). A CustomResourceDefinition (CRD) teaches the API server a new kind, and a controller you write gives it behaviour; together they are the operator pattern. This page is written against Kubernetes 1.37; see the section index for the version baseline. Helm-packaged operators and GitOps-delivered CRDs tie back to Helm and GitOps and CI/CD.

The 2015 book generation did this with ThirdPartyResource, which was deprecated in 1.7 and removed in 1.8; CRDs replaced it and are now a stable apiextensions.k8s.io/v1 API. Treat any book instruction using ThirdPartyResources as history.

CustomResourceDefinitions

A CRD declares a group, a kind, a scope and one or more versions, each with an OpenAPI v3 schema. Once it is created, kubectl can create, list and watch the new resource exactly like a built-in one, with RBAC (Authentication, Authorization and RBAC) and admission applying as usual. Here is a BookshelfBackup resource that asks for a backup of the Bookshelf database:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: bookshelfbackups.bookshelf.example.com
spec:
  group: bookshelf.example.com
  scope: Namespaced
  names:
    plural: bookshelfbackups
    singular: bookshelfbackup
    kind: BookshelfBackup
    shortNames: [bkp]
  versions:
    - name: v1
      served: true
      storage: true
      subresources:
        status: {}
      additionalPrinterColumns:
        - name: Schedule
          type: string
          jsonPath: .spec.schedule
        - name: Phase
          type: string
          jsonPath: .status.phase
        - name: Age
          type: date
          jsonPath: .metadata.creationTimestamp
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [schedule, retentionDays]
              properties:
                schedule:
                  type: string
                retentionDays:
                  type: integer
                  minimum: 1
                  maximum: 365
                target:
                  type: string
              x-kubernetes-validations:
                - rule: "self.retentionDays >= 7 || self.schedule.startsWith('@')"
                  message: "retention under 7 days is only allowed with an @-style schedule"
            status:
              type: object
              properties:
                phase:
                  type: string
                lastBackupTime:
                  type: string
                  format: date-time
apiVersion: bookshelf.example.com/v1
kind: BookshelfBackup
metadata:
  name: nightly
  namespace: bookshelf
spec:
  schedule: "0 2 * * *"
  retentionDays: 14
  target: postgres
kubectl apply -f bookshelfbackup-crd.yaml
kubectl apply -f nightly.yaml
kubectl get bkp -n bookshelf
NAME      SCHEDULE    PHASE   AGE
nightly   0 2 * * *           5s

Schema, CEL, versions and subresources

  • Schema. A structural OpenAPI v3 schema is required in apiextensions.k8s.io/v1; unknown fields are pruned. Defaults (default:) and format markers are part of it.

  • CEL validation rules. x-kubernetes-validations holds Common Expression Language rules evaluated by the API server (as in the example above), where self is the object at that schema level and oldSelf enables transition rules on updates. They replace many validating webhooks without running any extra service. The same language powers ValidatingAdmissionPolicy in Admission Control and Policy.

  • Versions and conversion. A CRD can serve several versions (v1alpha1, v1) while storing one. With identical schemas, conversion.strategy: None only rewrites apiVersion; when fields differ, use a Webhook strategy that runs a conversion service.

  • Subresources. status: {} splits spec and status, so users write spec while the controller writes status through /status with separate RBAC. scale exposes specReplicaPath, statusReplicaPath and a selector path so kubectl scale and the HPA (Autoscaling) can drive the resource.

  • Printer columns. additionalPrinterColumns decide what kubectl get shows.

The aggregation layer

When a CRD is not enough — custom storage, non-CRUD behaviour, or different request handling — an aggregated API server can register an APIService and have the main API server proxy a whole API group to it. metrics.k8s.io served by metrics-server is the familiar example. It is considerably more work than a CRD, so use it only when CRD semantics are insufficient.

Controllers and the operator pattern

A controller is a loop that watches objects, compares desired state (spec) with observed state and acts to close the gap, then writes the outcome to status. An operator is a controller plus CRDs that encode the operational knowledge of running one application — install, backup, upgrade, failover — so it happens automatically. Key building blocks:

  • Reconcile loop. Level-triggered and idempotent: the function receives a name, reads the current state and makes it right, so it must be safe to run repeatedly, after missed events or after a restart.

  • Watches. The controller subscribes to its own kind and to owned kinds (for example the Jobs or Pods it creates) so a change in any of them enqueues the owner.

  • Status. Report progress in status (phase, conditions, observed generation), not in logs.

  • Owner references. Children carry an ownerReferences entry pointing at the parent, so the garbage collector deletes them with it, and a watch on the child can be mapped to the parent.

  • Finalizers. A name in metadata.finalizers blocks deletion until the controller has cleaned up external resources (for example a backup in object storage) and removed it.

Kubebuilder scaffold

Kubebuilder, built on controller-runtime, generates the project, CRD manifests and controller skeleton:

kubebuilder init --domain example.com --repo github.com/example/bookshelf-operator
kubebuilder create api --group bookshelf --version v1 --kind BookshelfBackup --resource --controller
make manifests      # regenerate the CRD from Go type markers
make install        # apply the CRDs to the current cluster
make run            # run the controller locally against that cluster

An abbreviated reconcile function:

func (r *BookshelfBackupReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	var backup bookshelfv1.BookshelfBackup
	if err := r.Get(ctx, req.NamespacedName, &backup); err != nil {
		return ctrl.Result{}, client.IgnoreNotFound(err) // deleted: nothing to do
	}

	// Deletion: clean up external state, then release the finalizer.
	if !backup.DeletionTimestamp.IsZero() {
		if controllerutil.ContainsFinalizer(&backup, finalizerName) {
			if err := r.deleteRemoteBackups(ctx, &backup); err != nil {
				return ctrl.Result{}, err
			}
			controllerutil.RemoveFinalizer(&backup, finalizerName)
			return ctrl.Result{}, r.Update(ctx, &backup)
		}
		return ctrl.Result{}, nil
	}

	// Register the finalizer so the deletion branch above can run later.
	if controllerutil.AddFinalizer(&backup, finalizerName) {
		if err := r.Update(ctx, &backup); err != nil {
			return ctrl.Result{}, err
		}
	}

	// Ensure the CronJob that performs the backup exists and is owned by this resource.
	// CreateOrUpdate first reads the live object into cron, so the desired state is set in the mutate function.
	cron := &batchv1.CronJob{ObjectMeta: metav1.ObjectMeta{Name: backup.Name + "-backup", Namespace: backup.Namespace}}
	if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, cron, func() error {
		cron.Spec = desiredCronJobSpec(&backup)
		return ctrl.SetControllerReference(&backup, cron, r.Scheme)
	}); err != nil {
		return ctrl.Result{}, err
	}

	backup.Status.Phase = "Scheduled"
	return ctrl.Result{}, r.Status().Update(ctx, &backup)
}

The operator creates a CronJob (see Jobs and CronJobs) and owns it, which is the common shape: higher-level custom resources compose the built-in ones.

Other languages and well-known operators

Go is not mandatory. The Java Operator SDK offers the same reconcile model for Java, and Quarkus integrates it as an extension; see Quarkus on Kubernetes and OpenShift. Before writing one, check for an existing operator: cert-manager (certificates), Prometheus Operator (monitoring), CloudNativePG and other database operators, Strimzi (Kafka) and the Argo CD operator. Operators for stateful systems are discussed in StatefulSets.

kubectl plugins: a lighter extension

If you only need a new command rather than new resources, a kubectl plugin is any executable named kubectl-<name> on PATH; kubectl <name> runs it. The krew plugin manager indexes community plugins. Plugins add client-side convenience only and have no controller or server component. See kubectl Essentials.

The reconcile loop

flowchart TD EV["Event: create / update / delete\non BookshelfBackup or an owned CronJob"] --> Q["Work queue\n(namespace/name)"] Q --> R["Reconcile(name)"] R --> GET{"Resource\nexists?"} GET -->|no| DONE["Return: nothing to do"] GET -->|yes| DEL{"Being\ndeleted?"} DEL -->|yes| FIN["Run cleanup,\nremove finalizer"] DEL -->|no| OBS["Observe children;\ncreate or update owned CronJob"] OBS --> ST["Write status\n(phase, conditions)"] ST --> DONE R -->|error| RETRY["Requeue with backoff"] RETRY --> Q