Jobs and CronJobs

A Job’s Pod template may only set restartPolicy to OnFailure or Never — Always, the Deployment default, is rejected by the API server, because a Pod that always restarts can never reach a terminal state for the Job controller to count. Both retry a failure, but at a different layer, and Book 2’s ch. 3 (book ch. 10) treats the choice as a small but consequential one:

Book 2’s sampler default was OnFailure for the in-place-retry case, reserving Never for a Job whose failed attempts are worth keeping around individually — both patterns below use OnFailure, since neither needs to inspect an old failed attempt once a new Pod has already retried it.

Source: Kubernetes Docs — Job.

As kubectl Essentials already introduced, the Job controller stamps every Pod it creates with two labels of its own: job-name (the owning Job’s name) and controller-uid (a value unique to this particular Job object, so a second Job later created with the same name never accidentally adopts Pods left over from the first one). spec.selector is filled in automatically from these at creation time and, unlike a ReplicaSet's selector, is immutable once set. manualSelector: true opts out of the auto-generated selector in favour of a hand-written one, which is possible but discouraged — it is easy to write a selector that a second Job also matches, and nothing in Bookshelf needs it.

The simplest shape: completions: 1, parallelism: 1, exactly the manifest shown at the top of this page. One Pod runs, and the Job is Complete the moment it exits 0.

Book 2’s own way of creating this same one-shot Job was kubectl run notifier --image=ghcr.io/example/bookshelf-notifier:1.0.0 --restart=OnFailure — --mode=once — as kubectl Essentials covers in full, kubectl run’s Job/Deployment generators were removed in Kubernetes 1.18, and `kubectl run today always creates a bare Pod regardless of --restart. The direct replacement is an imperative kubectl create job, or the declarative manifest above:

Source: Kubernetes Docs — Job and Kubernetes Docs — Running Jobs.

Book 2’s second pattern split a fixed amount of work into completions equal-sized pieces, run up to parallelism at a time. On a 1.37 cluster the same pattern is written with completionMode: Indexed, so each Pod can read its own index and process only its slice instead of every Pod polling a shared queue for work:

Source: Kubernetes Docs — Indexed Job for Parallel Processing with Static Work Assignment.

Five completions, three Pods running at once: each Pod’s SHARD_INDEX (0-4) picks a disjoint slice of the catalog to notify about, with no coordination needed between the Pods — the split is entirely static, decided by completions and the shard count baked into the image’s arguments, unlike the work-queue pattern below where the split happens at runtime against shared state.

backoffLimitPerIndex: 2 and maxFailedIndexes: 1 together mean any single shard may retry up to twice on its own before that shard is given up on, and a persistently broken shard 2 does not consume retry budget that shards 0, 1, 3 and 4 never needed — they keep running to completion. Once every index has finished, the Job is still marked Failed if any index failed; maxFailedIndexes only ends it early, terminating the remaining Pods as soon as more than one shard has exhausted its own retries.

Source: Kubernetes Docs — Fine Parallel Processing Using a Work Queue and Kubernetes Docs — Job.

Book 2’s third pattern had producers push work items onto a queue Service and a pool of consumer Pods — created as a single Job with parallelism set but no fixed completions — pop items off it until the queue is empty, each Pod deciding for itself when there is no more work left rather than being handed a pre-assigned share. Rebuilt here with Redis as the queue, matching the Bookshelf app’s existing Redis instance (Persistent Volumes and Storage Classes covers how it is deployed):

Source: Kubernetes Docs — Fine Parallel Processing Using a Work Queue.

Each of the four consumer Pods loops on a blocking BRPOP notifier:pending 5 against Redis: whichever Pod happens to pop an item processes it, and a Pod that gets an empty result after its own timeout budget simply exits 0, counting as a success. Because completions is left unset, no new Pods are created once any consumer has exited successfully, and the Job is Complete once at least one Pod has succeeded and all Pods have terminated — there is no fixed total to reach, only "every consumer eventually found the queue empty and stopped". Producers (the catalog-api and reviews-api Deployments, on a catalog update) push with LPUSH notifier:pending <payload>, entirely decoupled from how many consumer Pods happen to be running at that moment.

Book 2’s own queue was a small HTTP work-queue Deployment reached through kubectl port-forward from the consumer Job’s Pods; the shape above is functionally identical — a shared, external store that decouples producers from however many consumers currently exist — swapped for Redis, since Bookshelf already runs one, and using a cluster-internal Service address instead of port-forward, which is a debugging tool (kubectl Essentials) rather than how Pods reach each other in production. The REDIS_PASSWORD above is never a literal in the manifest: it comes from a Secret created out of band, e.g. kubectl create secret generic bookshelf-redis --from-literal=password="$(openssl rand -base64 24)", exactly as ConfigMaps describes for Secret values in general.

Finally, once the consumer Job has run, Book 2’s way of checking on it was kubectl get jobs -a (--show-all) to see completed Jobs alongside active ones — as kubectl Essentials already covers, that flag was removed once kubectl get stopped hiding completed and failed objects by default, so a plain kubectl get jobs on 1.37 already shows notifier-queue-consumer once it finishes.