Secret Manager and Cloud KMS

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

See the section bibliography for the reference material consulted while preparing these pages.

Commands and limits below are current for the gcloud CLI 586.x line (gcloud secrets and gcloud kms command groups), the Secret Manager and Cloud KMS documentation and release notes published through 2026-09-29, the Google Cloud Client Libraries libraries-bom, Spring Cloud GCP 8.x for Spring Boot 4 (7.4.x for Spring Boot 3) and Terraform hashicorp/google v8 (checked 2026-09-30); re-verify against gcloud version, gcloud secrets create --help, gcloud kms keys create --help and the linked reference pages before relying on them in a pipeline.

Applications need two kinds of sensitive material: secrets (database passwords, API keys, tokens, certificates that the application must read back in clear text) and encryption keys (material that is used to encrypt and decrypt data but should never leave a hardened boundary). Google Cloud handles them with two services. Secret Manager stores, versions, rotates and controls access to small pieces of sensitive data. Cloud KMS (Key Management Service) creates and uses cryptographic keys, in software, in Cloud HSM hardware modules or in an external key manager, and lets other services encrypt your data with keys you control (customer-managed encryption keys, CMEK).

In the Bookshelf scenario (see Google Cloud) Secret Manager holds catalog-db-password (used by catalog-api to reach the Cloud SQL instance of Cloud SQL) and sendgrid-api-key (used by the notifier worker to send order-confirmation emails). Cloud KMS provides a key that encrypts the bookshelf-uploads-$PROJECT_ID bucket and the secrets themselves. Older guides still show these values as plain environment variables, in source control or in downloaded service-account key files; the current approach is a secret reference resolved at run time with an attached service account and Application Default Credentials. Non-secret settings (feature flags, endpoints, tuning values) belong in the configuration service described in Runtime Configuration, not here.

Prerequisites

PROJECT_ID=$(gcloud config get-value project)
PROJECT_NUMBER=$(gcloud projects describe "$PROJECT_ID" --format='value(projectNumber)')
REGION=us-central1

gcloud services enable \
  secretmanager.googleapis.com \
  cloudkms.googleapis.com \
  pubsub.googleapis.com

In the console, select the project in the project picker at the top of the page, open Security → Secret Manager (or Security → Key Management) and click Enable when prompted. Cloud Shell has the gcloud CLI preinstalled and authenticated as your user. The Pub/Sub API is only needed for the rotation topics later on this page.

Secret Manager concepts

Term Meaning

Secret

A named container (projects/PROJECT_ID/secrets/NAME) with its replication policy, labels, rotation and expiration settings, and IAM policy. It holds no payload by itself.

Secret version

An immutable payload (…​/versions/1, …​/versions/2, …​). Adding a value always creates a new version. A version is enabled, disabled (payload kept, access blocked) or destroyed (payload erased, irreversible).

Alias

A friendly name such as prod that points to one version number, so consumers can follow a pointer you move. The special name latest always means the highest enabled version number.

Replication policy

Where the payload is stored: automatic (Google chooses, global), user-managed (you list locations) or a regional secret (one location, regional endpoint).

Labels, annotations, tags

Labels organise and filter and show up in billing; annotations hold free-form metadata; resource tags drive conditional IAM and organisation policy.

Delayed destruction

A version destroy request is held for a configurable time (the secret’s version destroy TTL) before the payload is actually erased, so an accidental destroy can be undone.

A secret payload can be up to 64 KiB. It is bytes, not text, so a certificate bundle or a JSON document is fine. Treat the version number as the deployment unit: changing a password means adding a version, pointing consumers at it, and then disabling and eventually destroying the old one.

Replication and regional secrets

The default replication policy is automatic: Google replicates the payload to locations it chooses and the secret is reachable from the global endpoint secretmanager.googleapis.com. Choose user-managed replication when data residency rules require named locations. A user-managed policy lists one or more regions or multi-regions, and each location can use its own CMEK key.

Regional secrets (generally available since 2024-10-14) are a separate resource type pinned to a single location and served by the regional endpoint secretmanager.REGION.rep.googleapis.com. They suit workloads that must keep both the payload and the API traffic in one region. Multi-region locations for secrets have also been available since 2026-06. Regional and global secrets are different resources with different names (projects/PROJECT_ID/locations/REGION/secrets/NAME), and not every integration supports the regional form. In particular, Cloud Run does not support regional secrets, so the Bookshelf secrets used from Cloud Run stay global with automatic or user-managed replication. See Create a regional secret.

Expiration, rotation and delayed destruction

  • Expiration (--expire-time or --ttl) deletes the whole secret at a given time. It suits short-lived credentials issued for a demo or a migration, and it is irreversible.

  • Rotation (--next-rotation-time and --rotation-period, with one or more --topics) makes Secret Manager publish a SECRET_ROTATE event to Pub/Sub at the scheduled time. Secret Manager does not generate a new password or change the system that uses it. A subscriber, typically a Cloud Run function or a Cloud Run job, must create the new credential at the target system and call versions add. Treat the notification as the alarm clock, not the rotation.

  • Delayed destruction (--version-destroy-ttl) turns versions destroy into a scheduled destroy that can be cancelled with versions enable during the delay.

Creating and reading secrets

In the Google Cloud console

  1. Open Security → Secret Manager and click Create secret.

  2. Enter the name catalog-db-password.

  3. Paste the value into Secret value, or click Upload to read it from a file. Prefer upload so the value never sits in your clipboard history.

  4. Under Replication policy, keep Automatically replicate or select Manually manage locations and tick the regions. To create a regional secret, tick Use a regional secret and select the location instead.

  5. Optionally expand Encryption to pick a customer-managed key, Rotation to set a period and Pub/Sub topics, Expiration to set a deadline, and Labels to add app=bookshelf.

  6. Click Create secret. The secret page shows the versions table with Add new version, and a Permissions tab for IAM.

With the gcloud CLI

SECRET=catalog-db-password

# 1. The container: automatic replication, labels, delayed destruction of old versions
gcloud secrets create "$SECRET" \
  --replication-policy=automatic \
  --labels=app=bookshelf,env=dev \
  --version-destroy-ttl=7d

# 2. A version, read from a file (never type a value on the command line)
printf '%s' "REPLACE_ME" > /tmp/catalog-db-password.txt
gcloud secrets versions add "$SECRET" --data-file=/tmp/catalog-db-password.txt
rm /tmp/catalog-db-password.txt

# 3. A version from standard input, as a pipeline step would do
echo -n "REPLACE_ME" | gcloud secrets versions add "$SECRET" --data-file=-

# 4. Read it back
gcloud secrets versions access latest --secret="$SECRET"
gcloud secrets versions list "$SECRET"

REPLACE_ME is a placeholder. In practice the value comes from a password generator, an upstream system or a file produced by a pipeline, and it must not be committed or left in shell history. The --data-file=- form reads standard input, which keeps the value out of the process list. Reading with versions access prints the payload to the terminal, so use it for diagnosis, and pipe it to a consumer rather than saving it in a script.

User-managed and regional variants:

# User-managed replication in two regions
gcloud secrets create sendgrid-api-key \
  --replication-policy=user-managed \
  --locations=us-central1,us-east1

# A regional secret (different resource type, regional endpoint)
gcloud secrets create regional-demo \
  --location="$REGION"
echo -n "REPLACE_ME" | gcloud secrets versions add regional-demo \
  --secret=regional-demo --location="$REGION" --data-file=-

Versions, aliases and states

# Disable a version (reversible), re-enable it, then destroy it
gcloud secrets versions disable 1 --secret="$SECRET"
gcloud secrets versions enable 1 --secret="$SECRET"
gcloud secrets versions destroy 1 --secret="$SECRET"

# Aliases: make "prod" follow version 2
gcloud secrets update "$SECRET" --add-version-aliases=prod=2
gcloud secrets update "$SECRET" --remove-version-aliases=prod

A disabled version cannot be accessed but can be re-enabled, which makes disabling the right first step when a credential is suspected to be unused. Destroying is irreversible once the delay elapses. Aliases are resolved by the Secret Manager API and client libraries; integrations that take a version number (such as the Cloud Run flag below) accept a number or latest, so check the integration before relying on an alias.

Labels, annotations and tags

gcloud secrets update "$SECRET" \
  --update-labels=cost-center=books \
  --update-annotations=owner=catalog-team

# A resource tag lets IAM conditions and organisation policy target the secret
gcloud resource-manager tags bindings create \
  --tag-value="$PROJECT_ID/environment/dev" \
  --parent="//secretmanager.googleapis.com/projects/$PROJECT_NUMBER/secrets/$SECRET"

IAM for secrets

Grant access on the secret, not the project. A project-level secretAccessor binding lets the identity read every secret now and in the future, which defeats least privilege.

Role What it allows

roles/secretmanager.secretAccessor

Read the payload of versions (versions.access). What a running application needs.

roles/secretmanager.secretVersionAdder

Add new versions. For a rotation function, which should not be able to read old values.

roles/secretmanager.secretVersionManager

Add, enable, disable and destroy versions, without changing the secret’s settings.

roles/secretmanager.viewer

See secret metadata and the list of versions, but not payloads.

roles/secretmanager.admin

Everything, including IAM on the secret and deleting it. For automation that provisions secrets.

RUNTIME_SA="catalog-api-sa@$PROJECT_ID.iam.gserviceaccount.com"

gcloud iam service-accounts create catalog-api-sa --display-name="catalog-api runtime"

gcloud secrets add-iam-policy-binding catalog-db-password \
  --member="serviceAccount:$RUNTIME_SA" \
  --role=roles/secretmanager.secretAccessor

# Review the policy
gcloud secrets get-iam-policy catalog-db-password

The notifier worker gets secretAccessor on sendgrid-api-key only; catalog-api gets it on catalog-db-password only. Neither service account needs secretmanager.admin, and none needs a downloaded key: the service account is attached to the Cloud Run service or GKE workload, and the client library obtains a token through Application Default Credentials. On a workstation, impersonate the service account instead of exporting a key:

gcloud auth application-default login \
  --impersonate-service-account="$RUNTIME_SA"

For identities, workload identity and deny policies see Identity and Access. IAM conditions can narrow a binding to secrets with a given tag or name prefix, and a deny policy can block secretmanager.versions.access for everyone except a named group.

Consuming secrets

flowchart LR SM[("Secret Manager
catalog-db-password")] subgraph CR["Cloud Run: catalog-api"] ENV["Env var
pinned version"] VOL["Volume file
latest, refreshed"] end subgraph GKE["GKE: bookshelf-gke"] CSI["Secret Manager add-on
SecretProviderClass"] K8S["Kubernetes Secret
(optional sync)"] end APP["Java app
client library or
Spring Cloud GCP"] SM -->|"--set-secrets"| ENV SM -->|"--set-secrets path"| VOL SM -->|"CSI mount"| CSI CSI -.->|"sync"| K8S SM -->|"accessSecretVersion"| APP

Four consumption paths, from least to most code in the application: Cloud Run injects the secret before the container starts, the GKE add-on mounts it as a file (and optionally as a Kubernetes Secret), the application reads it through Spring configuration, or the application calls the client library directly.

Cloud Run

Cloud Run resolves a secret reference when an instance starts. There are two ways to expose it, with different update behaviour.

# As an environment variable: pin an explicit version
gcloud run services update catalog-api --region="$REGION" \
  --set-secrets=DB_PASSWORD=catalog-db-password:2

# As a file in a volume: reads the version again each time the file is opened,
# so "latest" follows rotation without a new revision
gcloud run services update catalog-api --region="$REGION" \
  --set-secrets=/secrets/db/password=catalog-db-password:latest
Environment variable Volume mount

Resolved

Once, when the instance starts

On each file read, so a new latest is picked up while the instance runs

Version

Pin a number for repeatable deployments; latest is read at startup only

Use latest to follow rotation, or a number to stay fixed

Restart needed after rotation

Yes (new revision or new instance)

No, if the application re-reads the file

Risk

Visible to anything that dumps the environment, such as a crash report

Path must not collide with another mount

The Cloud Run runtime service account needs secretAccessor on the secret. Deployment fails if it does not. Cloud Run only supports global secrets, as noted above. --update-secrets and --remove-secrets change one binding without replacing the rest. The mechanics of services, revisions and traffic splits are in Cloud Run and Cloud Run Scaling, Traffic and Jobs; the related environment-variable and volume handling for plain configuration is in Runtime Configuration.

GKE: the Secret Manager add-on

The managed Secret Manager add-on for GKE (a CSI driver that Google installs and upgrades) mounts secrets into pods as files and can mirror them into Kubernetes Secrets. It reached general availability on 2024-08-20, gained automatic rotation of mounted secrets on 2025-09-01 and the sync to Kubernetes Secrets on 2026-04-16. It requires Workload Identity Federation for GKE, which Autopilot clusters have on by default.

CLUSTER=bookshelf-gke

# Enable the add-on on an existing cluster (also --enable-secret-manager at create time)
gcloud container clusters update "$CLUSTER" \
  --location="$REGION" \
  --enable-secret-manager

# Optional: rotate mounted files automatically (check the flag names with --help)
gcloud container clusters update "$CLUSTER" \
  --location="$REGION" \
  --enable-secret-manager-rotation \
  --secret-manager-rotation-interval=120s

# Let the Kubernetes service account read the secret, with no keys involved
gcloud secrets add-iam-policy-binding catalog-db-password \
  --role=roles/secretmanager.secretAccessor \
  --member="principal://iam.googleapis.com/projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$PROJECT_ID.svc.id.goog/subject/ns/bookshelf/sa/catalog-api"

The workload declares what to mount with a SecretProviderClass whose provider is gke, then references it from a CSI volume:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: catalog-secrets
  namespace: bookshelf
spec:
  provider: gke
  parameters:
    secrets: |
      - resourceName: "projects/my-project-id/secrets/catalog-db-password/versions/latest"
        path: "db-password"
  # Optional: also create a Kubernetes Secret from the mounted file
  secretObjects:
    - secretName: catalog-db
      type: Opaque
      data:
        - objectName: "db-password"
          key: password
---
apiVersion: v1
kind: Pod
metadata:
  name: catalog-api
  namespace: bookshelf
spec:
  serviceAccountName: catalog-api
  containers:
    - name: app
      image: us-central1-docker.pkg.dev/my-project-id/bookshelf/catalog-api:1.0.0
      volumeMounts:
        - name: secrets
          mountPath: /var/run/secrets/bookshelf
          readOnly: true
  volumes:
    - name: secrets
      csi:
        driver: secrets-store-gke.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: catalog-secrets

Replace my-project-id with the project ID (the manifest cannot read shell variables). A synced Kubernetes Secret exists only while at least one pod mounts the class, and its value lives in etcd, so the file mount is the safer default; use the sync only for software that insists on a secretKeyRef or an envFrom. Mounting by file, a pod sees the new content after rotation within the configured interval, whereas an environment variable populated from a synced Secret needs a pod restart. More on GKE workloads, namespaces and Gateway API is in Deploying Microservices on GKE and GKE Clusters.

External Secrets Operator

The open-source External Secrets Operator is the portable alternative. A ClusterSecretStore with provider.gcpsm points to Secret Manager, and ExternalSecret objects create ordinary Kubernetes Secrets on a refresh interval. It works on any cluster, including non-GKE clusters and multi-cloud setups, at the cost of running and upgrading the operator yourself. Choose the add-on for GKE-only platforms and the operator when the same manifests must run elsewhere.

Java client library

Import the com.google.cloud:libraries-bom in dependencyManagement and add com.google.cloud:google-cloud-secretmanager without a version; see Google Cloud Libraries BOM and Secret Manager client libraries.

package com.example.bookshelf.config;

import com.google.cloud.secretmanager.v1.SecretManagerServiceClient;
import com.google.cloud.secretmanager.v1.SecretVersionName;

import java.io.IOException;

public class SecretReader {

    /** Reads a secret version using Application Default Credentials. */
    public String read(String projectId, String secretId, String version) throws IOException {
        try (SecretManagerServiceClient client = SecretManagerServiceClient.create()) {
            SecretVersionName name = SecretVersionName.of(projectId, secretId, version);
            return client.accessSecretVersion(name).getPayload().getData().toStringUtf8();
        }
    }
}

Derived from the access secret version sample.

Creating a client opens a channel, so create one per application and reuse it, and cache the value for a short time instead of calling the API for every request: access operations are billed and quota-limited. Run with a version of latest only when the code can tolerate a changing value.

Spring Boot and Spring Cloud GCP

Spring Cloud GCP’s Secret Manager support turns secrets into configuration properties, so catalog-api reads its password like any other Spring property. The starter is spring-cloud-gcp-starter-secretmanager (version managed by spring-cloud-gcp-dependencies 8.x for Spring Boot 4, 7.4.x for Spring Boot 3).

spring:
  config:
    import: "sm@"
  cloud:
    gcp:
      secretmanager:
        project-id: my-project-id
  datasource:
    url: jdbc:postgresql:///catalog?cloudSqlInstance=my-project-id:us-central1:bookshelf-catalog&socketFactory=com.google.cloud.sql.postgres.SocketFactory
    username: catalog-app
    password: ${sm@catalog-db-password}

The sm@ import enables the property source; ${sm@secret-id} resolves the latest version and ${sm@secret-id/version} pins one. The values are loaded when the application context starts, so a rotated secret needs a restart unless refresh is configured. Keep the property placeholders in committed files and never the values. Profiles and layered configuration are covered in Configuration and Profiles. For a Cloud SQL password specifically, prefer IAM database authentication (see Cloud SQL) so that no password exists at all; the secret remains the fallback for engines that lack it.

Cloud Build and CI

Cloud Build reads a secret during a build without writing it to the build config or logs: declare it under availableSecrets.secretManager with a versionName and an env name, list the name in a step’s secretEnv, and grant the build’s service account secretAccessor on that secret (see Use secrets in Cloud Build and Container Registry and Builds). GitHub’s encrypted secrets suit a token only GitHub uses; a value that production also needs should live in one place. A workflow can authenticate with Workload Identity Federation (no key file) through google-github-actions/auth and fetch it with get-secretmanager-secrets, using an identity that holds only secretAccessor. Federation setup is in Identity and Access; the CI side is in Docker CI/CD with GitHub Actions.

Rotation in practice

A rotation topic and a function make the schedule real:

gcloud pubsub topics create secret-rotation

# The Secret Manager service agent must be allowed to publish to the topic
gcloud pubsub topics add-iam-policy-binding secret-rotation \
  --member="serviceAccount:service-$PROJECT_NUMBER@gcp-sa-secretmanager.iam.gserviceaccount.com" \
  --role=roles/pubsub.publisher

gcloud secrets update sendgrid-api-key \
  --add-topics="projects/$PROJECT_ID/topics/secret-rotation" \
  --next-rotation-time="2026-12-01T03:00:00Z" \
  --rotation-period=90d

When the schedule fires, the topic receives a message with the event type SECRET_ROTATE and the secret name in its attributes. A subscriber (an Eventarc-triggered Cloud Run function, see Cloud Run Functions and Events) should:

  1. Call the third-party API to issue a new key (SendGrid, in the Bookshelf case).

  2. Add it as a new secret version with a service account holding only secretVersionAdder.

  3. Verify the new key works, then disable the previous version.

  4. Destroy the old version after a grace period, using the delayed destruction setting as a safety net.

Services that resolve latest at start-up need a restart or a new revision to pick up the change; the Cloud Run volume mount and the GKE add-on with rotation follow it live.

Auditing secret access

Admin Activity audit logs (creating secrets, changing IAM, adding versions) are always on. Reads of payloads are Data Access logs, which are off by default for most services and must be enabled for Secret Manager:

gcloud projects get-iam-policy "$PROJECT_ID" --format=json > /tmp/policy.json
# Add to "auditConfigs":
# { "service": "secretmanager.googleapis.com",
#   "auditLogConfigs": [ { "logType": "DATA_READ" }, { "logType": "ADMIN_READ" } ] }
gcloud projects set-iam-policy "$PROJECT_ID" /tmp/policy.json

Then query who read what:

gcloud logging read \
  'protoPayload.serviceName="secretmanager.googleapis.com" AND protoPayload.methodName="google.cloud.secretmanager.v1.SecretManagerService.AccessSecretVersion"' \
  --limit=10 \
  --format='table(timestamp, protoPayload.authenticationInfo.principalEmail, protoPayload.resourceName)'

Derived from Logs Explorer and the audit logging page above.

Data Access logs are charged by log volume; a busy service that reads a secret on every request will produce a lot of them, another reason to cache the value. Logs, metrics and alerting are covered in Monitoring and Observability.

Cloud KMS

Cloud KMS organises keys in a hierarchy that mirrors the way you want to grant access:

Object Meaning

Key ring

A named group of keys in one location (us-central1, a multi-region, or global). Key rings cannot be renamed or deleted, so name them for stable purposes, such as bookshelf.

Key (CryptoKey)

A logical key with a purpose, a protection level, a rotation schedule and an IAM policy. Purposes include symmetric encryption, asymmetric encryption, asymmetric signing, MAC signing and raw encryption.

Key version

The actual key material. Rotation creates a new primary version used for new encryptions; older enabled versions still decrypt. Versions are enabled, disabled, scheduled for destruction or destroyed.

Protection levels

Level Where key material lives Choose it when

Software

Cloud KMS software, FIPS 140-3 validated modules

The default. Lowest cost, all key purposes, enough for most workloads.

Cloud HSM

FIPS 140-3 Level 3 hardware security modules operated by Google

A compliance regime requires hardware-protected keys.

Cloud External Key Manager (EKM)

A key manager outside Google Cloud that you or a partner operate

Key custody must stay outside Google; keys are used over the network, so availability depends on the external system.

Single-tenant HSM options and EKM through a VPC are variations of the last two rows. The protection level is fixed when the key is created. Software and HSM keys can be used in the same CMEK configurations.

In the Google Cloud console

  1. Open Security → Key Management and click Create key ring.

  2. Enter bookshelf, choose the location us-central1 (it must match the resources the key will protect) and click Create.

  3. On the key ring page click Create key, name it bookshelf-data, choose Generated key, protection level Software, purpose Symmetric encrypt/decrypt.

  4. Under Rotation period choose 90 days and leave the first rotation date at its default, then click Create.

  5. Open the key to see its versions, the Permissions tab, and Usage for the services already using it.

With the gcloud CLI

gcloud kms keyrings create bookshelf --location="$REGION"

gcloud kms keys create bookshelf-data \
  --keyring=bookshelf --location="$REGION" \
  --purpose=encryption \
  --protection-level=software \
  --rotation-period=90d \
  --next-rotation-time="2026-12-01T00:00:00Z" \
  --destroy-scheduled-duration=30d

gcloud kms keys list --keyring=bookshelf --location="$REGION"
gcloud kms keys versions list --key=bookshelf-data --keyring=bookshelf --location="$REGION"

--destroy-scheduled-duration is set at creation (default 30 days) and cannot be changed afterwards; it is the period a destroyed-by-request key version waits before it is erased. Use --protection-level=hsm for Cloud HSM. Encrypting and decrypting small payloads directly:

echo -n "REPLACE_ME" > /tmp/plain.txt
gcloud kms encrypt \
  --key=bookshelf-data --keyring=bookshelf --location="$REGION" \
  --plaintext-file=/tmp/plain.txt --ciphertext-file=/tmp/cipher.bin

gcloud kms decrypt \
  --key=bookshelf-data --keyring=bookshelf --location="$REGION" \
  --ciphertext-file=/tmp/cipher.bin --plaintext-file=/tmp/plain-again.txt
rm -f /tmp/plain.txt /tmp/cipher.bin /tmp/plain-again.txt

Direct encryption is limited to 64 KiB of plaintext; for larger data use envelope encryption (the data is encrypted with a data key, which KMS wraps), which is what CMEK-enabled Google services do for you.

KMS IAM

Role Use

roles/cloudkms.admin

Create and manage keys and rings, without being able to use them to encrypt or decrypt.

roles/cloudkms.cryptoKeyEncrypterDecrypter

Use a key to encrypt and decrypt. Granted to applications and to service agents for CMEK.

roles/cloudkms.cryptoKeyEncrypter / roles/cloudkms.cryptoKeyDecrypter

Split the two operations, for example a producer that may only encrypt.

roles/cloudkms.signerVerifier, roles/cloudkms.publicKeyViewer

Asymmetric signing and fetching public keys.

roles/cloudkms.viewer

Read key metadata, not use it.

Separate the people who manage keys (cloudkms.admin) from the identities that use them (cryptoKeyEncrypterDecrypter). Grant on the key rather than on the project:

gcloud kms keys add-iam-policy-binding bookshelf-data \
  --keyring=bookshelf --location="$REGION" \
  --member="serviceAccount:$RUNTIME_SA" \
  --role=roles/cloudkms.cryptoKeyEncrypterDecrypter

Customer-managed encryption keys (CMEK)

Google encrypts data at rest by default with Google-managed keys. CMEK lets a supported service wrap its data keys with a KMS key in your project, so you control rotation, can disable access, and appear in the audit trail when the service uses the key. The pattern is always the same:

  1. Create the key in the same location as the resource it protects (a regional bucket in us-central1 needs a us-central1 key; global or multi-region resources need a matching multi-region or global key).

  2. Give the service’s service agent cloudkms.cryptoKeyEncrypterDecrypter on the key.

  3. Create or reconfigure the resource with the key name.

KEY="projects/$PROJECT_ID/locations/$REGION/keyRings/bookshelf/cryptoKeys/bookshelf-data"

# Cloud Storage: the service agent exists once the project has used Cloud Storage
GCS_AGENT=$(gcloud storage service-agent --project="$PROJECT_ID")
gcloud kms keys add-iam-policy-binding bookshelf-data \
  --keyring=bookshelf --location="$REGION" \
  --member="serviceAccount:$GCS_AGENT" \
  --role=roles/cloudkms.cryptoKeyEncrypterDecrypter

gcloud storage buckets update "gs://bookshelf-uploads-$PROJECT_ID" \
  --default-encryption-key="$KEY"

# Secret Manager: create the service agent, grant it the key, then use the key on a secret
gcloud beta services identity create --service=secretmanager.googleapis.com
gcloud kms keys add-iam-policy-binding bookshelf-data \
  --keyring=bookshelf --location="$REGION" \
  --member="serviceAccount:service-$PROJECT_NUMBER@gcp-sa-secretmanager.iam.gserviceaccount.com" \
  --role=roles/cloudkms.cryptoKeyEncrypterDecrypter

gcloud secrets create regional-cmek-demo \
  --location="$REGION" --kms-key-name="$KEY"

For Cloud SQL, create the service agent with gcloud beta services identity create --service=sqladmin.googleapis.com, grant it the key, then create the instance with --disk-encryption-key; CMEK on a Cloud SQL instance must be chosen at creation and cannot be added later (see Cloud SQL). Cloud Storage CMEK applies to new objects; existing objects keep their original key until rewritten (see Cloud Storage Security and Static Sites). Disks, GKE node boot disks, BigQuery and Pub/Sub have their own options. Organisation policies (constraints/gcp.restrictNonCmekServices and constraints/gcp.restrictCmekCryptoKeyProjects) can require CMEK and restrict which projects hold the keys.

Disabling or destroying a key makes the data unreadable. That is the point of CMEK ("crypto-shredding") and also its main operational risk, so keep keys in a dedicated project with restricted access and a documented procedure.

Cloud KMS Autokey

Autokey creates keys on demand so that platform teams do not have to pre-provision key rings for every service. A folder is configured with an Autokey key project; when a developer creates a protected resource, the service asks Autokey for a key handle, and Autokey creates a key with the right location, rotation and protection level, grants the service agent, and returns it. The setup is performed by a folder administrator and works for the supported services listed in the documentation:

An administrator with roles/cloudkms.autokeyAdmin on the folder runs gcloud kms autokey-config update with the folder and key project; a developer then runs gcloud kms key-handles create with a --resource-type-selector. See Cloud KMS Autokey overview and Enable Autokey.

The supported resource types and the exact flags change as services are added, so confirm them on the linked pages. Autokey suits organisations with many teams; for a single Bookshelf project a manually created key as above is simpler. Resource hierarchy and folder concepts are in Resource Hierarchy and Projects.

Terraform

resource "google_kms_key_ring" "bookshelf" {
  name     = "bookshelf"
  location = var.region
}

resource "google_kms_crypto_key" "data" {
  name            = "bookshelf-data"
  key_ring        = google_kms_key_ring.bookshelf.id
  rotation_period = "7776000s" # 90 days
  lifecycle {
    prevent_destroy = true
  }
}

resource "google_secret_manager_secret" "db_password" {
  secret_id = "catalog-db-password"
  labels    = { app = "bookshelf" }
  replication {
    auto {}
  }
}

resource "google_secret_manager_secret_iam_member" "catalog_api" {
  secret_id = google_secret_manager_secret.db_password.id
  role      = "roles/secretmanager.secretAccessor"
  member    = "serviceAccount:${google_service_account.catalog_api.email}"
}

Derived from the Terraform registry pages google_secret_manager_secret, google_kms_crypto_key.

Do not define the secret version in Terraform with a literal value: the payload would end up in the state file. Create the container in Terraform and add versions out of band, or use the write-only attribute that the provider documents for secret data, so the value is not stored. State handling and modules are in Infrastructure as Code.

Destroying keys and key versions

A Cloud KMS key ring and a key can never be deleted. Only key versions are destroyed, which erases the key material.

# Schedule destruction of version 1 (state becomes DESTROY_SCHEDULED)
gcloud kms keys versions destroy 1 \
  --key=bookshelf-data --keyring=bookshelf --location="$REGION"

# Changed your mind within the scheduled duration
gcloud kms keys versions restore 1 \
  --key=bookshelf-data --keyring=bookshelf --location="$REGION"

# Disable a version first, which is reversible and blocks use immediately
gcloud kms keys versions disable 1 \
  --key=bookshelf-data --keyring=bookshelf --location="$REGION"

Disabling is the first step; scheduling destruction waits for the key’s destroy-scheduled duration before the material is erased. Everything encrypted with that version is lost once it is. Billing for a version stops when it is destroyed. Before destroying, check usage in the console key page, the Cloud Asset Inventory and the Key Access Justifications or audit logs.

Application-level encryption with KMS

Services that offer CMEK encrypt on your behalf. Application-level encryption of fields (for example client-side field-level encryption in MongoDB) uses a KMS key as the master key that wraps data encryption keys. The MongoDB version of that pattern, including the driver configuration for a Google Cloud KMS provider, is in Authentication, authorization & encryption; on Google Cloud the master key is a symmetric KMS key in your project, accessible through the same Application Default Credentials and cryptoKeyEncrypterDecrypter role as above. Document databases are compared in MongoDB on Google Cloud.

Security checklist

  • One secret per credential, bound to the single service account that needs it, with secretAccessor on the secret and not on the project.

  • No service-account key files, and no secrets in committed manifests, container images or build logs.

  • Enable Data Access logs for Secret Manager, and use delayed destruction so an accidental destroy can be undone.

  • Keep CMEK keys in a separate project with distinct admins and users, and prevent_destroy on key resources.

Pricing model

Pricing changes, so no figures are quoted here; use the dated pricing pages (checked 2026-09-30).

  • Secret Manager charges for each active secret version per location per month, for access operations (AccessSecretVersion and similar) in blocks, and for rotation notifications. A small monthly free allotment of versions and operations is included. A user-managed secret replicated to several locations counts a version once per location. Disabled and destroyed versions are not billed as active.

  • Cloud KMS charges per key version per month, with higher rates for Cloud HSM and EKM, and per cryptographic operation, with a free allotment for software keys. Many versions that are rotated but never destroyed accumulate cost; a destroyed version stops billing.

  • Log volume for Data Access audit logs, Pub/Sub messages for rotation and the function that rotates the secret are billed through their own services.

Cost drivers to watch: lots of secrets with several old versions, an application that calls AccessSecretVersion on each request, and HSM keys with frequent rotation. The pricing pages are Secret Manager pricing and Cloud KMS pricing. The general cost tools are in Getting Started with Google Cloud.

Quotas and limits

A secret version payload is limited to 64 KiB, direct KMS encryption to 64 KiB of plaintext, and both services apply per-project, per-minute request quotas (see Secret Manager quotas and Cloud KMS quotas). Key rings and keys cannot be deleted, so plan names.

Launch stages and recent changes

Date Change

2024-08-20

Secret Manager add-on for GKE generally available.

2024-10-14

Regional secrets generally available.

2025-09-01

Automatic rotation for the GKE add-on’s mounted secrets generally available.

2026-04-16

Sync of add-on secrets to Kubernetes Secrets generally available.

2026-06

Multi-region locations for secrets.

The release notes linked below are the authority for newer dates. Older guides still show the Secret Manager integration with Cloud Functions as the way to read secrets, or the Kubernetes CSI driver you installed yourself; today the first-class paths are the Cloud Run flag and the managed GKE add-on.

Troubleshooting

Symptom Likely cause and fix

PERMISSION_DENIED on secretmanager.versions.access

The runtime service account lacks secretAccessor on that secret, or an IAM deny policy applies. Check the policy with gcloud secrets get-iam-policy, and the audit log entry.

Cloud Run deployment fails with "Permission denied on secret"

Grant secretAccessor to the service’s runtime service account, not to your user.

Cloud Run cannot find a regional secret

Cloud Run supports only global secrets. Create a global secret or read the regional one from application code.

NOT_FOUND for versions/latest

No enabled version exists (all were disabled or destroyed), or the secret and the version are in different locations or projects.

A rotated value is not picked up

The application resolved the secret at start-up (environment variable or Spring property). Restart, or switch to a volume mount, or refresh in code.

GKE pod stuck in ContainerCreating with a mount error

Workload Identity Federation is missing, the principal lacks secretAccessor, or the resourceName in the SecretProviderClass is wrong. Check kubectl describe pod events.

CMEK resource creation fails with a key permission error

The service agent was not created or lacks cryptoKeyEncrypterDecrypter on the key, or the key location does not match the resource.

Data became unreadable

The key (or the version that encrypted it) was disabled or destroyed. Re-enable it within the scheduled destruction period; after that the data cannot be recovered.

A quota error (RESOURCE_EXHAUSTED)

Too many access calls. Cache values in the client and reuse the client object.

Clean up

Delete in dependency order: consumers first, then secrets, then keys. Secret versions bill while active, and KMS key versions bill until they are destroyed, so a lab that forgets this keeps paying a little. Skip the commands for resources you did not create.

# Consumers: stop referencing the secret
gcloud run services update catalog-api --region="$REGION" \
  --remove-secrets=DB_PASSWORD
gcloud run services update catalog-api --region="$REGION" \
  --remove-secrets=/secrets/db/password

# Secrets (this deletes all versions and cannot be undone)
gcloud secrets delete catalog-db-password --quiet
gcloud secrets delete sendgrid-api-key --quiet
gcloud secrets delete regional-demo --location="$REGION" --quiet
gcloud secrets delete regional-cmek-demo --location="$REGION" --quiet

# Rotation plumbing and service account
gcloud pubsub topics delete secret-rotation
gcloud iam service-accounts delete "$RUNTIME_SA" --quiet

# GKE add-on (only if you enabled it for this lab)
gcloud container clusters update "$CLUSTER" --location="$REGION" --no-enable-secret-manager

# KMS: schedule destruction of every version; the key and key ring remain
for V in $(gcloud kms keys versions list --key=bookshelf-data --keyring=bookshelf \
    --location="$REGION" --filter='state=ENABLED OR state=DISABLED' --format='value(name.basename())'); do
  gcloud kms keys versions destroy "$V" --key=bookshelf-data --keyring=bookshelf --location="$REGION"
done

Before destroying the KMS versions, make sure that no bucket, instance or secret still depends on the key: re-key or delete them first, because their data becomes unreadable once the scheduled period passes. Key rings and keys cannot be deleted, only emptied, and cost nothing without active versions. Cloud Storage CMEK settings are removed with gcloud storage buckets update --clear-default-encryption-key. If you deleted a secret by mistake, there is no undelete; delayed destruction applies to versions, not to the secret itself. For a throwaway project, deleting the project schedules everything for deletion with the usual 30-day recovery window (see Resource Hierarchy and Projects).