Custom Resources and Operators
|
This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production; see the section bibliography for the reference material consulted while preparing these pages. |
Kubernetes is built so that its own API can be extended. Everything seen so far — Deployments, Services, Jobs — is a resource served by the API server and reconciled by a controller (Objects and the API). A CustomResourceDefinition (CRD) teaches the API server a new kind, and a controller you write gives it behaviour; together they are the operator pattern. This page is written against Kubernetes 1.37; see the section index for the version baseline. Helm-packaged operators and GitOps-delivered CRDs tie back to Helm and GitOps and CI/CD.
The 2015 book generation did this with ThirdPartyResource, which was deprecated in 1.7 and removed in 1.8;
CRDs replaced it and are now a stable apiextensions.k8s.io/v1 API. Treat any book instruction using
ThirdPartyResources as history.
CustomResourceDefinitions
A CRD declares a group, a kind, a scope and one or more versions, each with an OpenAPI v3 schema. Once it is
created, kubectl can create, list and watch the new resource exactly like a built-in one, with RBAC
(Authentication, Authorization and RBAC) and
admission applying as usual. Here is a BookshelfBackup resource that asks for a backup of the Bookshelf
database:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: bookshelfbackups.bookshelf.example.com
spec:
group: bookshelf.example.com
scope: Namespaced
names:
plural: bookshelfbackups
singular: bookshelfbackup
kind: BookshelfBackup
shortNames: [bkp]
versions:
- name: v1
served: true
storage: true
subresources:
status: {}
additionalPrinterColumns:
- name: Schedule
type: string
jsonPath: .spec.schedule
- name: Phase
type: string
jsonPath: .status.phase
- name: Age
type: date
jsonPath: .metadata.creationTimestamp
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
required: [schedule, retentionDays]
properties:
schedule:
type: string
retentionDays:
type: integer
minimum: 1
maximum: 365
target:
type: string
x-kubernetes-validations:
- rule: "self.retentionDays >= 7 || self.schedule.startsWith('@')"
message: "retention under 7 days is only allowed with an @-style schedule"
status:
type: object
properties:
phase:
type: string
lastBackupTime:
type: string
format: date-time
apiVersion: bookshelf.example.com/v1
kind: BookshelfBackup
metadata:
name: nightly
namespace: bookshelf
spec:
schedule: "0 2 * * *"
retentionDays: 14
target: postgres
kubectl apply -f bookshelfbackup-crd.yaml
kubectl apply -f nightly.yaml
kubectl get bkp -n bookshelf
NAME SCHEDULE PHASE AGE
nightly 0 2 * * * 5s
Schema, CEL, versions and subresources
-
Schema. A structural OpenAPI v3 schema is required in
apiextensions.k8s.io/v1; unknown fields are pruned. Defaults (default:) and format markers are part of it. -
CEL validation rules.
x-kubernetes-validationsholds Common Expression Language rules evaluated by the API server (as in the example above), whereselfis the object at that schema level andoldSelfenables transition rules on updates. They replace many validating webhooks without running any extra service. The same language powers ValidatingAdmissionPolicy in Admission Control and Policy. -
Versions and conversion. A CRD can serve several versions (
v1alpha1,v1) while storing one. With identical schemas,conversion.strategy: Noneonly rewritesapiVersion; when fields differ, use aWebhookstrategy that runs a conversion service. -
Subresources.
status: {}splits spec and status, so users writespecwhile the controller writesstatusthrough/statuswith separate RBAC.scaleexposesspecReplicaPath,statusReplicaPathand a selector path sokubectl scaleand the HPA (Autoscaling) can drive the resource. -
Printer columns.
additionalPrinterColumnsdecide whatkubectl getshows.
The aggregation layer
When a CRD is not enough — custom storage, non-CRUD behaviour, or different request handling — an aggregated API
server can register an APIService and have the main API server proxy a whole API group to it. metrics.k8s.io
served by metrics-server is the familiar example. It is considerably more work than a CRD, so use it only when CRD
semantics are insufficient.
Controllers and the operator pattern
A controller is a loop that watches objects, compares desired state (spec) with observed state and acts to
close the gap, then writes the outcome to status. An operator is a controller plus CRDs that encode the
operational knowledge of running one application — install, backup, upgrade, failover — so it happens
automatically. Key building blocks:
-
Reconcile loop. Level-triggered and idempotent: the function receives a name, reads the current state and makes it right, so it must be safe to run repeatedly, after missed events or after a restart.
-
Watches. The controller subscribes to its own kind and to owned kinds (for example the Jobs or Pods it creates) so a change in any of them enqueues the owner.
-
Status. Report progress in
status(phase, conditions, observed generation), not in logs. -
Owner references. Children carry an
ownerReferencesentry pointing at the parent, so the garbage collector deletes them with it, and a watch on the child can be mapped to the parent. -
Finalizers. A name in
metadata.finalizersblocks deletion until the controller has cleaned up external resources (for example a backup in object storage) and removed it.
Kubebuilder scaffold
Kubebuilder, built on controller-runtime, generates the project, CRD manifests and controller skeleton:
kubebuilder init --domain example.com --repo github.com/example/bookshelf-operator
kubebuilder create api --group bookshelf --version v1 --kind BookshelfBackup --resource --controller
make manifests # regenerate the CRD from Go type markers
make install # apply the CRDs to the current cluster
make run # run the controller locally against that cluster
An abbreviated reconcile function:
func (r *BookshelfBackupReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
var backup bookshelfv1.BookshelfBackup
if err := r.Get(ctx, req.NamespacedName, &backup); err != nil {
return ctrl.Result{}, client.IgnoreNotFound(err) // deleted: nothing to do
}
// Deletion: clean up external state, then release the finalizer.
if !backup.DeletionTimestamp.IsZero() {
if controllerutil.ContainsFinalizer(&backup, finalizerName) {
if err := r.deleteRemoteBackups(ctx, &backup); err != nil {
return ctrl.Result{}, err
}
controllerutil.RemoveFinalizer(&backup, finalizerName)
return ctrl.Result{}, r.Update(ctx, &backup)
}
return ctrl.Result{}, nil
}
// Register the finalizer so the deletion branch above can run later.
if controllerutil.AddFinalizer(&backup, finalizerName) {
if err := r.Update(ctx, &backup); err != nil {
return ctrl.Result{}, err
}
}
// Ensure the CronJob that performs the backup exists and is owned by this resource.
// CreateOrUpdate first reads the live object into cron, so the desired state is set in the mutate function.
cron := &batchv1.CronJob{ObjectMeta: metav1.ObjectMeta{Name: backup.Name + "-backup", Namespace: backup.Namespace}}
if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, cron, func() error {
cron.Spec = desiredCronJobSpec(&backup)
return ctrl.SetControllerReference(&backup, cron, r.Scheme)
}); err != nil {
return ctrl.Result{}, err
}
backup.Status.Phase = "Scheduled"
return ctrl.Result{}, r.Status().Update(ctx, &backup)
}
The operator creates a CronJob (see Jobs and CronJobs) and owns it, which is the common shape: higher-level custom resources compose the built-in ones.
Other languages and well-known operators
Go is not mandatory. The Java Operator SDK offers the same reconcile model for Java, and Quarkus integrates it as an extension; see Quarkus on Kubernetes and OpenShift. Before writing one, check for an existing operator: cert-manager (certificates), Prometheus Operator (monitoring), CloudNativePG and other database operators, Strimzi (Kafka) and the Argo CD operator. Operators for stateful systems are discussed in StatefulSets.
kubectl plugins: a lighter extension
If you only need a new command rather than new resources, a kubectl plugin is any executable named
kubectl-<name> on PATH; kubectl <name> runs it. The krew plugin manager indexes
community plugins. Plugins add client-side convenience only and have no controller or server component. See
kubectl Essentials.