Secrets

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production; see the section bibliography for the reference material consulted while preparing these pages.

A Secret is the Kubernetes object for small pieces of sensitive data — a database credential, a TLS key pair, a registry login — so that they are kept out of container images and Pod specs. Structurally it is almost identical to a ConfigMap: a namespaced map of keys to values, consumed as environment variables or files. What differs is the handling around it: Secrets can be encrypted at rest, are not written to disk on nodes (volumes are backed by tmpfs), get their own RBAC resource, and are sent only to nodes that run a Pod using them. This page is written against Kubernetes 1.37; see the section index for the full version baseline.

All values on this page are placeholders. Replace <base64-encoded-value> and changeme with real values supplied from a secret manager or a pipeline; never commit real ones.

Source: Kubernetes Docs — Secrets.

What a Secret is, and what it is not

Secret values in the manifest’s data map are base64-encoded. That is an encoding for carrying arbitrary bytes in YAML/JSON, not a protection: anyone who can read the object can run base64 --decode. The stringData field is a write-only convenience that accepts plain text and is converted to data on write. By default the API server stores Secrets in etcd unencrypted, so real protection comes from three layers: RBAC limiting who may read them, encryption at rest in etcd, and keeping the authoritative copy in an external store.

apiVersion: v1
kind: Secret
metadata:
  name: catalog-db
  namespace: bookshelf
type: Opaque
stringData:                       # write-only, stored as base64 in data
  DB_USER: bookshelf
  DB_PASSWORD: changeme
# equivalent, pre-encoded form:
# data:
#   DB_PASSWORD: <base64-encoded-value>

Source: Kubernetes Docs — Secrets.

Secret types

The type field tells tooling what keys to expect, and the API server validates the keys of built-in types.

Type Purpose

Opaque (default)

Arbitrary user-defined data.

kubernetes.io/tls

A certificate and key, with required keys tls.crt and tls.key; used by Ingress and Gateway listeners (Ingress, Gateway API).

kubernetes.io/dockerconfigjson

Registry credentials in a .dockerconfigjson key, referenced from imagePullSecrets.

kubernetes.io/basic-auth, kubernetes.io/ssh-auth

Conventional key names (username/password, ssh-privatekey) for those credentials.

kubernetes.io/service-account-token

Legacy long-lived ServiceAccount token; prefer short-lived projected tokens.

bootstrap.kubernetes.io/token

Node bootstrap tokens used by kubeadm join.

Source: Kubernetes Docs — Secret types.

Creating Secrets

# Generic: literal values or files
kubectl create secret generic catalog-db \
  --from-literal=DB_USER=bookshelf \
  --from-literal=DB_PASSWORD=changeme \
  --namespace bookshelf

# TLS: from a certificate/key pair you already hold
kubectl create secret tls bookshelf-tls \
  --cert=./tls.crt --key=./tls.key --namespace bookshelf

# Registry credentials for imagePullSecrets
kubectl create secret docker-registry regcred \
  --docker-server=ghcr.io \
  --docker-username=<registry-user> \
  --docker-password=<registry-credential> \
  --namespace bookshelf

# Inspect without printing values
kubectl describe secret catalog-db -n bookshelf
kubectl get secret catalog-db -n bookshelf -o jsonpath='{.data.DB_USER}' | base64 --decode

Source: Kubernetes Docs — Managing Secrets using kubectl.

Literals typed on a command line land in shell history and process listings; prefer --from-file or a pipeline that pulls the value from a secret manager. Declarative management with kubectl apply and Kustomize’s secretGenerator is covered in the Kubernetes docs; committing the resulting YAML is only safe when it is encrypted (see Encrypted in Git: Sealed Secrets and SOPS).

Consuming Secrets

Both forms mirror ConfigMaps.

apiVersion: v1
kind: Pod
metadata:
  name: catalog-api
  namespace: bookshelf
spec:
  imagePullSecrets:
    - name: regcred
  containers:
    - name: catalog-api
      image: ghcr.io/example/bookshelf-catalog-api:1.0.0
      env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: catalog-db
              key: DB_PASSWORD
      volumeMounts:
        - name: db-credentials
          mountPath: /etc/secrets/db
          readOnly: true
  volumes:
    - name: db-credentials
      secret:
        secretName: catalog-db
        defaultMode: 0400

Source: Kubernetes Docs — Using Secrets.

Environment variables are simple but leak easily (child processes, crash dumps, kubectl describe of a Pod spec that inlines values, debug endpoints) and never refresh. Volume mounts are tmpfs-backed, can have restrictive file modes, and update in place about a minute after the Secret changes (except with subPath), so they are the better default. The propagation and restart-on-change rules are the same as for ConfigMaps.

Immutable Secrets

immutable: true freezes the object: nobody, including a mistaken kubectl apply, can alter it, and the kubelet stops watching it. Rotation then means creating a new Secret name and rolling the workload.

apiVersion: v1
kind: Secret
metadata:
  name: catalog-db-v2
  namespace: bookshelf
immutable: true
type: Opaque
stringData:
  DB_PASSWORD: changeme

Source: Kubernetes Docs — Immutable Secrets.

Protecting Secrets in the cluster

Encryption at rest

The API server can encrypt selected resources before writing to etcd, configured through an EncryptionConfiguration file passed with --encryption-provider-config. Providers are tried in order: the first encrypts new writes, all can decrypt reads. Two families matter:

  • Local keys (aescbc, aesgcm, secretbox): the key lives in a file on each control-plane node. Easy, but anyone with the file and an etcd backup can decrypt.

  • KMS v2 (kms with apiVersion: v2, stable since 1.29): the API server calls a plugin that wraps a data encryption key with a key held in an external KMS or HSM, caching it, so the key-encryption key never sits on the control plane. KMS v1 is deprecated.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          apiVersion: v2
          name: example-kms
          endpoint: unix:///var/run/kmsplugin/socket.sock
          timeout: 3s
      - identity: {}        # last: lets objects written before encryption still be read

Source: Kubernetes Docs — Using a KMS provider for data encryption.

After enabling it, existing Secrets stay plaintext until rewritten, for example with kubectl get secrets -A -o json | kubectl replace -f -. On managed clusters this is usually a switch in the provider’s console rather than a file you edit; see Cluster Setup and Administration.

Source: Kubernetes Docs — Encrypting Confidential Data at Rest.

RBAC and good practices

get, list and watch on Secrets are equivalent to reading them: a list returns every value in the namespace, and anyone who can create a Pod in a namespace can mount any Secret there. The official good-practices guide boils down to: enable encryption at rest; grant least-privilege RBAC, avoiding wildcards and list/watch on Secrets; restrict which Pods can use which Secrets with distinct ServiceAccounts; prefer short-lived credentials; and audit access. See Authentication, Authorization and RBAC.

Source: Kubernetes Docs — Good practices for Kubernetes Secrets.

External secret stores

Because a Secret in etcd is a copy, many teams keep the source of truth outside the cluster (a cloud secret manager, HashiCorp Vault) and sync or mount it in.

External Secrets Operator

The External Secrets Operator (ESO) is a controller with custom resources (Custom Resources and Operators). A SecretStore (or cluster-wide ClusterSecretStore) describes how to reach a backend; an ExternalSecret says which remote keys to fetch and what native Secret to create. The controller refreshes it on an interval, so rotation in the backend propagates automatically.

sequenceDiagram participant ES as ExternalSecret participant C as ESO controller participant SS as SecretStore participant B as External backend participant K as Kubernetes Secret participant P as Pod ES->>C: desired: remote key -> Secret catalog-db C->>SS: read provider config and auth C->>B: fetch remote value B-->>C: value C->>K: create or update Secret K-->>P: mounted or injected loop every refreshInterval C->>B: re-fetch C->>K: update if changed end
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: bookshelf-store
  namespace: bookshelf
spec:
  provider:
    aws:                           # one of many providers
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: bookshelf-eso    # IRSA / workload identity, no static credentials
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: catalog-db
  namespace: bookshelf
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: bookshelf-store
    kind: SecretStore
  target:
    name: catalog-db
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: bookshelf/catalog-db
        property: password

Source: External Secrets Operator — Overview. Check the current apiVersion for your installed release.

Secrets Store CSI driver

The Secrets Store CSI driver mounts secrets from Vault, Azure Key Vault, AWS Secrets Manager or Google Secret Manager as files through a CSI volume (see Persistent Volumes and Storage Classes for CSI). A SecretProviderClass describes what to fetch; the secret need never exist as a Kubernetes Secret unless secretObjects syncing is enabled.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: catalog-db-provider
  namespace: bookshelf
spec:
  provider: azure
  parameters:
    keyvaultName: <key-vault-name>
    tenantId: <tenant-id>
    objects: |
      array:
        - |
          objectName: catalog-db-credential
          objectType: secret

Source: Secrets Store CSI Driver documentation. The Azure side, including workload identity, is in Azure Key Vault and Secrets.

Encrypted in Git: Sealed Secrets and SOPS

GitOps (GitOps and CI/CD) wants every manifest in Git, but Secrets cannot be committed in clear. Two approaches store them encrypted:

  • Sealed Secrets: kubeseal encrypts a Secret with the cluster controller’s public key into a SealedSecret custom resource that is safe to commit; only the controller in that cluster can decrypt it into a native Secret.

  • SOPS (Mozilla SOPS, used with Flux or Argo CD plugins and Kustomize): encrypts only the values of a YAML file with age, PGP or a cloud KMS key, leaving keys readable so diffs stay useful; the GitOps tool decrypts at apply time.

# Sealed Secrets: encrypt a generated Secret manifest, commit only the sealed output
kubectl create secret generic catalog-db --from-literal=DB_PASSWORD=changeme \
  --dry-run=client -o yaml | kubeseal --format yaml > catalog-db-sealed.yaml

# SOPS: encrypt values in place with a key you control
sops --encrypt --age <age-public-key> --encrypted-regex '^(data|stringData)$' \
  --in-place catalog-db.yaml

Source: Sealed Secrets and SOPS documentation.

References