Secret Manager and Cloud KMS
|
This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production. See the section bibliography for the reference material consulted while preparing these pages. |
Commands and limits below are current for the gcloud CLI 586.x line (gcloud secrets and gcloud kms command
groups), the Secret Manager and Cloud KMS documentation and release notes published through 2026-09-29, the Google
Cloud Client Libraries libraries-bom, Spring Cloud GCP 8.x for Spring Boot 4 (7.4.x for Spring Boot 3) and
Terraform hashicorp/google v8 (checked 2026-09-30); re-verify against gcloud version,
gcloud secrets create --help, gcloud kms keys create --help and the linked reference pages before relying on them
in a pipeline.
Applications need two kinds of sensitive material: secrets (database passwords, API keys, tokens, certificates that the application must read back in clear text) and encryption keys (material that is used to encrypt and decrypt data but should never leave a hardened boundary). Google Cloud handles them with two services. Secret Manager stores, versions, rotates and controls access to small pieces of sensitive data. Cloud KMS (Key Management Service) creates and uses cryptographic keys, in software, in Cloud HSM hardware modules or in an external key manager, and lets other services encrypt your data with keys you control (customer-managed encryption keys, CMEK).
In the Bookshelf scenario (see Google Cloud) Secret Manager holds catalog-db-password
(used by catalog-api to reach the Cloud SQL instance of Cloud SQL) and
sendgrid-api-key (used by the notifier worker to send order-confirmation emails). Cloud KMS provides a key that
encrypts the bookshelf-uploads-$PROJECT_ID bucket and the secrets themselves. Older guides still show these values
as plain environment variables, in source control or in downloaded service-account key files; the current approach
is a secret reference resolved at run time with an attached service account and Application Default Credentials.
Non-secret settings (feature flags, endpoints, tuning values) belong in the configuration service described in
Runtime Configuration, not here.
Prerequisites
PROJECT_ID=$(gcloud config get-value project)
PROJECT_NUMBER=$(gcloud projects describe "$PROJECT_ID" --format='value(projectNumber)')
REGION=us-central1
gcloud services enable \
secretmanager.googleapis.com \
cloudkms.googleapis.com \
pubsub.googleapis.com
Derived from Configure Secret Manager and Create a key.
In the console, select the project in the project picker at the top of the page, open Security → Secret Manager (or Security → Key Management) and click Enable when prompted. Cloud Shell has the gcloud CLI preinstalled and authenticated as your user. The Pub/Sub API is only needed for the rotation topics later on this page.
Secret Manager concepts
| Term | Meaning |
|---|---|
Secret |
A named container ( |
Secret version |
An immutable payload ( |
Alias |
A friendly name such as |
Replication policy |
Where the payload is stored: automatic (Google chooses, global), user-managed (you list locations) or a regional secret (one location, regional endpoint). |
Labels, annotations, tags |
Labels organise and filter and show up in billing; annotations hold free-form metadata; resource tags drive conditional IAM and organisation policy. |
Delayed destruction |
A version destroy request is held for a configurable time (the secret’s version destroy TTL) before the payload is actually erased, so an accidental destroy can be undone. |
A secret payload can be up to 64 KiB. It is bytes, not text, so a certificate bundle or a JSON document is fine. Treat the version number as the deployment unit: changing a password means adding a version, pointing consumers at it, and then disabling and eventually destroying the old one.
Replication and regional secrets
The default replication policy is automatic: Google replicates the payload to locations it chooses and the secret
is reachable from the global endpoint secretmanager.googleapis.com. Choose user-managed replication when data
residency rules require named locations. A user-managed policy lists one or more regions or multi-regions, and each
location can use its own CMEK key.
Regional secrets (generally available since 2024-10-14) are a separate resource type pinned to a single location and
served by the regional endpoint secretmanager.REGION.rep.googleapis.com. They suit workloads that must keep both
the payload and the API traffic in one region. Multi-region locations for secrets have also been available since
2026-06. Regional and global secrets are different resources with different names (projects/PROJECT_ID/locations/REGION/secrets/NAME),
and not every integration supports the regional form. In particular, Cloud Run does not support regional
secrets, so the Bookshelf secrets used from Cloud Run stay global with automatic or user-managed replication. See
Create a regional secret.
Expiration, rotation and delayed destruction
-
Expiration (
--expire-timeor--ttl) deletes the whole secret at a given time. It suits short-lived credentials issued for a demo or a migration, and it is irreversible. -
Rotation (
--next-rotation-timeand--rotation-period, with one or more--topics) makes Secret Manager publish aSECRET_ROTATEevent to Pub/Sub at the scheduled time. Secret Manager does not generate a new password or change the system that uses it. A subscriber, typically a Cloud Run function or a Cloud Run job, must create the new credential at the target system and callversions add. Treat the notification as the alarm clock, not the rotation. -
Delayed destruction (
--version-destroy-ttl) turnsversions destroyinto a scheduled destroy that can be cancelled withversions enableduring the delay.
Creating and reading secrets
In the Google Cloud console
-
Open Security → Secret Manager and click Create secret.
-
Enter the name
catalog-db-password. -
Paste the value into Secret value, or click Upload to read it from a file. Prefer upload so the value never sits in your clipboard history.
-
Under Replication policy, keep Automatically replicate or select Manually manage locations and tick the regions. To create a regional secret, tick Use a regional secret and select the location instead.
-
Optionally expand Encryption to pick a customer-managed key, Rotation to set a period and Pub/Sub topics, Expiration to set a deadline, and Labels to add
app=bookshelf. -
Click Create secret. The secret page shows the versions table with Add new version, and a Permissions tab for IAM.
With the gcloud CLI
SECRET=catalog-db-password
# 1. The container: automatic replication, labels, delayed destruction of old versions
gcloud secrets create "$SECRET" \
--replication-policy=automatic \
--labels=app=bookshelf,env=dev \
--version-destroy-ttl=7d
# 2. A version, read from a file (never type a value on the command line)
printf '%s' "REPLACE_ME" > /tmp/catalog-db-password.txt
gcloud secrets versions add "$SECRET" --data-file=/tmp/catalog-db-password.txt
rm /tmp/catalog-db-password.txt
# 3. A version from standard input, as a pipeline step would do
echo -n "REPLACE_ME" | gcloud secrets versions add "$SECRET" --data-file=-
# 4. Read it back
gcloud secrets versions access latest --secret="$SECRET"
gcloud secrets versions list "$SECRET"
Derived from Create and access a secret, Add a secret version and gcloud secrets create.
REPLACE_ME is a placeholder. In practice the value comes from a password generator, an upstream system or a file
produced by a pipeline, and it must not be committed or left in shell history. The --data-file=- form reads standard
input, which keeps the value out of the process list. Reading with versions access prints the payload to the
terminal, so use it for diagnosis, and pipe it to a consumer rather than saving it in a script.
User-managed and regional variants:
# User-managed replication in two regions
gcloud secrets create sendgrid-api-key \
--replication-policy=user-managed \
--locations=us-central1,us-east1
# A regional secret (different resource type, regional endpoint)
gcloud secrets create regional-demo \
--location="$REGION"
echo -n "REPLACE_ME" | gcloud secrets versions add regional-demo \
--secret=regional-demo --location="$REGION" --data-file=-
Derived from Choose a replication policy and Create a regional secret.
Versions, aliases and states
# Disable a version (reversible), re-enable it, then destroy it
gcloud secrets versions disable 1 --secret="$SECRET"
gcloud secrets versions enable 1 --secret="$SECRET"
gcloud secrets versions destroy 1 --secret="$SECRET"
# Aliases: make "prod" follow version 2
gcloud secrets update "$SECRET" --add-version-aliases=prod=2
gcloud secrets update "$SECRET" --remove-version-aliases=prod
Derived from Manage secret versions and Add an alias to a secret version.
A disabled version cannot be accessed but can be re-enabled, which makes disabling the right first step when a
credential is suspected to be unused. Destroying is irreversible once the delay elapses. Aliases are resolved by the
Secret Manager API and client libraries; integrations that take a version number (such as the Cloud Run flag below)
accept a number or latest, so check the integration before relying on an alias.
Labels, annotations and tags
gcloud secrets update "$SECRET" \
--update-labels=cost-center=books \
--update-annotations=owner=catalog-team
# A resource tag lets IAM conditions and organisation policy target the secret
gcloud resource-manager tags bindings create \
--tag-value="$PROJECT_ID/environment/dev" \
--parent="//secretmanager.googleapis.com/projects/$PROJECT_NUMBER/secrets/$SECRET"
Derived from Create and manage labels and Tags for Secret Manager.
IAM for secrets
Grant access on the secret, not the project. A project-level secretAccessor binding lets the identity read every
secret now and in the future, which defeats least privilege.
| Role | What it allows |
|---|---|
|
Read the payload of versions ( |
|
Add new versions. For a rotation function, which should not be able to read old values. |
|
Add, enable, disable and destroy versions, without changing the secret’s settings. |
|
See secret metadata and the list of versions, but not payloads. |
|
Everything, including IAM on the secret and deleting it. For automation that provisions secrets. |
RUNTIME_SA="catalog-api-sa@$PROJECT_ID.iam.gserviceaccount.com"
gcloud iam service-accounts create catalog-api-sa --display-name="catalog-api runtime"
gcloud secrets add-iam-policy-binding catalog-db-password \
--member="serviceAccount:$RUNTIME_SA" \
--role=roles/secretmanager.secretAccessor
# Review the policy
gcloud secrets get-iam-policy catalog-db-password
Derived from Access control with IAM and gcloud secrets add-iam-policy-binding.
The notifier worker gets secretAccessor on sendgrid-api-key only; catalog-api gets it on
catalog-db-password only. Neither service account needs secretmanager.admin, and none needs a downloaded key:
the service account is attached to the Cloud Run service or GKE workload, and the client library obtains a token
through Application Default Credentials. On a workstation, impersonate the service account instead of exporting a key:
gcloud auth application-default login \
--impersonate-service-account="$RUNTIME_SA"
Derived from Use service account impersonation.
For identities, workload identity and deny policies see Identity and Access. IAM
conditions can narrow a binding to secrets with a given tag or name prefix, and a deny policy can block
secretmanager.versions.access for everyone except a named group.
Consuming secrets
catalog-db-password")] subgraph CR["Cloud Run: catalog-api"] ENV["Env var
pinned version"] VOL["Volume file
latest, refreshed"] end subgraph GKE["GKE: bookshelf-gke"] CSI["Secret Manager add-on
SecretProviderClass"] K8S["Kubernetes Secret
(optional sync)"] end APP["Java app
client library or
Spring Cloud GCP"] SM -->|"--set-secrets"| ENV SM -->|"--set-secrets path"| VOL SM -->|"CSI mount"| CSI CSI -.->|"sync"| K8S SM -->|"accessSecretVersion"| APP
Four consumption paths, from least to most code in the application: Cloud Run injects the secret before the container starts, the GKE add-on mounts it as a file (and optionally as a Kubernetes Secret), the application reads it through Spring configuration, or the application calls the client library directly.
Cloud Run
Cloud Run resolves a secret reference when an instance starts. There are two ways to expose it, with different update behaviour.
# As an environment variable: pin an explicit version
gcloud run services update catalog-api --region="$REGION" \
--set-secrets=DB_PASSWORD=catalog-db-password:2
# As a file in a volume: reads the version again each time the file is opened,
# so "latest" follows rotation without a new revision
gcloud run services update catalog-api --region="$REGION" \
--set-secrets=/secrets/db/password=catalog-db-password:latest
Derived from Configure secrets for services and gcloud run services update.
| Environment variable | Volume mount | |
|---|---|---|
Resolved |
Once, when the instance starts |
On each file read, so a new |
Version |
Pin a number for repeatable deployments; |
Use |
Restart needed after rotation |
Yes (new revision or new instance) |
No, if the application re-reads the file |
Risk |
Visible to anything that dumps the environment, such as a crash report |
Path must not collide with another mount |
The Cloud Run runtime service account needs secretAccessor on the secret. Deployment fails if it does not. Cloud
Run only supports global secrets, as noted above. --update-secrets and --remove-secrets change one binding
without replacing the rest. The mechanics of services, revisions and traffic splits are in
Cloud Run and Cloud Run Scaling, Traffic and Jobs; the
related environment-variable and volume handling for plain configuration is in
Runtime Configuration.
GKE: the Secret Manager add-on
The managed Secret Manager add-on for GKE (a CSI driver that Google installs and upgrades) mounts secrets into pods as files and can mirror them into Kubernetes Secrets. It reached general availability on 2024-08-20, gained automatic rotation of mounted secrets on 2025-09-01 and the sync to Kubernetes Secrets on 2026-04-16. It requires Workload Identity Federation for GKE, which Autopilot clusters have on by default.
CLUSTER=bookshelf-gke
# Enable the add-on on an existing cluster (also --enable-secret-manager at create time)
gcloud container clusters update "$CLUSTER" \
--location="$REGION" \
--enable-secret-manager
# Optional: rotate mounted files automatically (check the flag names with --help)
gcloud container clusters update "$CLUSTER" \
--location="$REGION" \
--enable-secret-manager-rotation \
--secret-manager-rotation-interval=120s
# Let the Kubernetes service account read the secret, with no keys involved
gcloud secrets add-iam-policy-binding catalog-db-password \
--role=roles/secretmanager.secretAccessor \
--member="principal://iam.googleapis.com/projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$PROJECT_ID.svc.id.goog/subject/ns/bookshelf/sa/catalog-api"
Derived from Use the Secret Manager add-on for GKE.
The workload declares what to mount with a SecretProviderClass whose provider is gke, then references it from a
CSI volume:
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: catalog-secrets
namespace: bookshelf
spec:
provider: gke
parameters:
secrets: |
- resourceName: "projects/my-project-id/secrets/catalog-db-password/versions/latest"
path: "db-password"
# Optional: also create a Kubernetes Secret from the mounted file
secretObjects:
- secretName: catalog-db
type: Opaque
data:
- objectName: "db-password"
key: password
---
apiVersion: v1
kind: Pod
metadata:
name: catalog-api
namespace: bookshelf
spec:
serviceAccountName: catalog-api
containers:
- name: app
image: us-central1-docker.pkg.dev/my-project-id/bookshelf/catalog-api:1.0.0
volumeMounts:
- name: secrets
mountPath: /var/run/secrets/bookshelf
readOnly: true
volumes:
- name: secrets
csi:
driver: secrets-store-gke.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: catalog-secrets
Derived from Use the Secret Manager add-on for GKE and the Secrets Store CSI Driver documentation.
Replace my-project-id with the project ID (the manifest cannot read shell variables). A synced Kubernetes Secret
exists only while at least one pod mounts the class, and its value lives in etcd, so the file mount is the safer
default; use the sync only for software that insists on a secretKeyRef or an envFrom. Mounting by file, a pod
sees the new content after rotation within the configured interval, whereas an environment variable populated from a
synced Secret needs a pod restart. More on GKE workloads, namespaces and Gateway API is in
Deploying Microservices on GKE and GKE Clusters.
- External Secrets Operator
-
The open-source External Secrets Operator is the portable alternative. A
ClusterSecretStorewithprovider.gcpsmpoints to Secret Manager, andExternalSecretobjects create ordinary Kubernetes Secrets on a refresh interval. It works on any cluster, including non-GKE clusters and multi-cloud setups, at the cost of running and upgrading the operator yourself. Choose the add-on for GKE-only platforms and the operator when the same manifests must run elsewhere.
Java client library
Import the com.google.cloud:libraries-bom in dependencyManagement and add com.google.cloud:google-cloud-secretmanager
without a version; see Google Cloud Libraries BOM and
Secret Manager client libraries.
package com.example.bookshelf.config;
import com.google.cloud.secretmanager.v1.SecretManagerServiceClient;
import com.google.cloud.secretmanager.v1.SecretVersionName;
import java.io.IOException;
public class SecretReader {
/** Reads a secret version using Application Default Credentials. */
public String read(String projectId, String secretId, String version) throws IOException {
try (SecretManagerServiceClient client = SecretManagerServiceClient.create()) {
SecretVersionName name = SecretVersionName.of(projectId, secretId, version);
return client.accessSecretVersion(name).getPayload().getData().toStringUtf8();
}
}
}
Derived from the access secret version sample.
Creating a client opens a channel, so create one per application and reuse it, and cache the value for a short time
instead of calling the API for every request: access operations are billed and quota-limited. Run with a version of
latest only when the code can tolerate a changing value.
Spring Boot and Spring Cloud GCP
Spring Cloud GCP’s Secret Manager support turns secrets into configuration properties, so catalog-api reads its
password like any other Spring property. The starter is spring-cloud-gcp-starter-secretmanager (version managed by
spring-cloud-gcp-dependencies 8.x for Spring Boot 4, 7.4.x for Spring Boot 3).
spring:
config:
import: "sm@"
cloud:
gcp:
secretmanager:
project-id: my-project-id
datasource:
url: jdbc:postgresql:///catalog?cloudSqlInstance=my-project-id:us-central1:bookshelf-catalog&socketFactory=com.google.cloud.sql.postgres.SocketFactory
username: catalog-app
password: ${sm@catalog-db-password}
Derived from Spring Cloud GCP Secret Manager.
The sm@ import enables the property source; ${sm@secret-id} resolves the latest version and
${sm@secret-id/version} pins one. The values are loaded when the application context starts, so a rotated secret
needs a restart unless refresh is configured. Keep the property placeholders in committed files and never the
values. Profiles and layered configuration are covered in
Configuration and Profiles. For a Cloud SQL password specifically, prefer IAM database
authentication (see Cloud SQL) so that no password exists at all; the secret remains
the fallback for engines that lack it.
Cloud Build and CI
Cloud Build reads a secret during a build without writing it to the build config or logs: declare it under
availableSecrets.secretManager with a versionName and an env name, list the name in a step’s secretEnv, and
grant the build’s service account secretAccessor on that secret (see
Use secrets in Cloud Build and
Container Registry and Builds). GitHub’s encrypted secrets suit a token only GitHub
uses; a value that production also needs should live in one place. A workflow can authenticate with Workload Identity
Federation (no key file) through google-github-actions/auth and fetch it
with get-secretmanager-secrets, using an identity
that holds only secretAccessor. Federation setup is in Identity and Access; the CI
side is in Docker CI/CD with GitHub Actions.
Rotation in practice
A rotation topic and a function make the schedule real:
gcloud pubsub topics create secret-rotation
# The Secret Manager service agent must be allowed to publish to the topic
gcloud pubsub topics add-iam-policy-binding secret-rotation \
--member="serviceAccount:service-$PROJECT_NUMBER@gcp-sa-secretmanager.iam.gserviceaccount.com" \
--role=roles/pubsub.publisher
gcloud secrets update sendgrid-api-key \
--add-topics="projects/$PROJECT_ID/topics/secret-rotation" \
--next-rotation-time="2026-12-01T03:00:00Z" \
--rotation-period=90d
Derived from Create a rotation schedule and Event notifications.
When the schedule fires, the topic receives a message with the event type SECRET_ROTATE and the secret name in its
attributes. A subscriber (an Eventarc-triggered Cloud Run function, see
Cloud Run Functions and Events) should:
-
Call the third-party API to issue a new key (SendGrid, in the Bookshelf case).
-
Add it as a new secret version with a service account holding only
secretVersionAdder. -
Verify the new key works, then disable the previous version.
-
Destroy the old version after a grace period, using the delayed destruction setting as a safety net.
Services that resolve latest at start-up need a restart or a new revision to pick up the change; the Cloud Run
volume mount and the GKE add-on with rotation follow it live.
Auditing secret access
Admin Activity audit logs (creating secrets, changing IAM, adding versions) are always on. Reads of payloads are Data Access logs, which are off by default for most services and must be enabled for Secret Manager:
gcloud projects get-iam-policy "$PROJECT_ID" --format=json > /tmp/policy.json
# Add to "auditConfigs":
# { "service": "secretmanager.googleapis.com",
# "auditLogConfigs": [ { "logType": "DATA_READ" }, { "logType": "ADMIN_READ" } ] }
gcloud projects set-iam-policy "$PROJECT_ID" /tmp/policy.json
Derived from Secret Manager audit logging and Configure Data Access audit logs.
Then query who read what:
gcloud logging read \
'protoPayload.serviceName="secretmanager.googleapis.com" AND protoPayload.methodName="google.cloud.secretmanager.v1.SecretManagerService.AccessSecretVersion"' \
--limit=10 \
--format='table(timestamp, protoPayload.authenticationInfo.principalEmail, protoPayload.resourceName)'
Derived from Logs Explorer and the audit logging page above.
Data Access logs are charged by log volume; a busy service that reads a secret on every request will produce a lot of them, another reason to cache the value. Logs, metrics and alerting are covered in Monitoring and Observability.
Cloud KMS
Cloud KMS organises keys in a hierarchy that mirrors the way you want to grant access:
| Object | Meaning |
|---|---|
Key ring |
A named group of keys in one location ( |
Key (CryptoKey) |
A logical key with a purpose, a protection level, a rotation schedule and an IAM policy. Purposes include symmetric encryption, asymmetric encryption, asymmetric signing, MAC signing and raw encryption. |
Key version |
The actual key material. Rotation creates a new primary version used for new encryptions; older enabled versions still decrypt. Versions are enabled, disabled, scheduled for destruction or destroyed. |
Protection levels
| Level | Where key material lives | Choose it when |
|---|---|---|
Software |
Cloud KMS software, FIPS 140-3 validated modules |
The default. Lowest cost, all key purposes, enough for most workloads. |
Cloud HSM |
FIPS 140-3 Level 3 hardware security modules operated by Google |
A compliance regime requires hardware-protected keys. |
Cloud External Key Manager (EKM) |
A key manager outside Google Cloud that you or a partner operate |
Key custody must stay outside Google; keys are used over the network, so availability depends on the external system. |
Single-tenant HSM options and EKM through a VPC are variations of the last two rows. The protection level is fixed when the key is created. Software and HSM keys can be used in the same CMEK configurations.
In the Google Cloud console
-
Open Security → Key Management and click Create key ring.
-
Enter
bookshelf, choose the locationus-central1(it must match the resources the key will protect) and click Create. -
On the key ring page click Create key, name it
bookshelf-data, choose Generated key, protection level Software, purpose Symmetric encrypt/decrypt. -
Under Rotation period choose 90 days and leave the first rotation date at its default, then click Create.
-
Open the key to see its versions, the Permissions tab, and Usage for the services already using it.
With the gcloud CLI
gcloud kms keyrings create bookshelf --location="$REGION"
gcloud kms keys create bookshelf-data \
--keyring=bookshelf --location="$REGION" \
--purpose=encryption \
--protection-level=software \
--rotation-period=90d \
--next-rotation-time="2026-12-01T00:00:00Z" \
--destroy-scheduled-duration=30d
gcloud kms keys list --keyring=bookshelf --location="$REGION"
gcloud kms keys versions list --key=bookshelf-data --keyring=bookshelf --location="$REGION"
Derived from Create a key, Key rotation and gcloud kms keys create.
--destroy-scheduled-duration is set at creation (default 30 days) and cannot be changed afterwards; it is the
period a destroyed-by-request key version waits before it is erased. Use --protection-level=hsm for Cloud HSM.
Encrypting and decrypting small payloads directly:
echo -n "REPLACE_ME" > /tmp/plain.txt
gcloud kms encrypt \
--key=bookshelf-data --keyring=bookshelf --location="$REGION" \
--plaintext-file=/tmp/plain.txt --ciphertext-file=/tmp/cipher.bin
gcloud kms decrypt \
--key=bookshelf-data --keyring=bookshelf --location="$REGION" \
--ciphertext-file=/tmp/cipher.bin --plaintext-file=/tmp/plain-again.txt
rm -f /tmp/plain.txt /tmp/cipher.bin /tmp/plain-again.txt
Derived from Encrypt and decrypt data with a symmetric key.
Direct encryption is limited to 64 KiB of plaintext; for larger data use envelope encryption (the data is encrypted with a data key, which KMS wraps), which is what CMEK-enabled Google services do for you.
KMS IAM
| Role | Use |
|---|---|
|
Create and manage keys and rings, without being able to use them to encrypt or decrypt. |
|
Use a key to encrypt and decrypt. Granted to applications and to service agents for CMEK. |
|
Split the two operations, for example a producer that may only encrypt. |
|
Asymmetric signing and fetching public keys. |
|
Read key metadata, not use it. |
Separate the people who manage keys (cloudkms.admin) from the identities that use them
(cryptoKeyEncrypterDecrypter). Grant on the key rather than on the project:
gcloud kms keys add-iam-policy-binding bookshelf-data \
--keyring=bookshelf --location="$REGION" \
--member="serviceAccount:$RUNTIME_SA" \
--role=roles/cloudkms.cryptoKeyEncrypterDecrypter
Derived from Cloud KMS IAM permissions and roles.
Customer-managed encryption keys (CMEK)
Google encrypts data at rest by default with Google-managed keys. CMEK lets a supported service wrap its data keys with a KMS key in your project, so you control rotation, can disable access, and appear in the audit trail when the service uses the key. The pattern is always the same:
-
Create the key in the same location as the resource it protects (a regional bucket in
us-central1needs aus-central1key; global or multi-region resources need a matching multi-region or global key). -
Give the service’s service agent
cloudkms.cryptoKeyEncrypterDecrypteron the key. -
Create or reconfigure the resource with the key name.
KEY="projects/$PROJECT_ID/locations/$REGION/keyRings/bookshelf/cryptoKeys/bookshelf-data"
# Cloud Storage: the service agent exists once the project has used Cloud Storage
GCS_AGENT=$(gcloud storage service-agent --project="$PROJECT_ID")
gcloud kms keys add-iam-policy-binding bookshelf-data \
--keyring=bookshelf --location="$REGION" \
--member="serviceAccount:$GCS_AGENT" \
--role=roles/cloudkms.cryptoKeyEncrypterDecrypter
gcloud storage buckets update "gs://bookshelf-uploads-$PROJECT_ID" \
--default-encryption-key="$KEY"
# Secret Manager: create the service agent, grant it the key, then use the key on a secret
gcloud beta services identity create --service=secretmanager.googleapis.com
gcloud kms keys add-iam-policy-binding bookshelf-data \
--keyring=bookshelf --location="$REGION" \
--member="serviceAccount:service-$PROJECT_NUMBER@gcp-sa-secretmanager.iam.gserviceaccount.com" \
--role=roles/cloudkms.cryptoKeyEncrypterDecrypter
gcloud secrets create regional-cmek-demo \
--location="$REGION" --kms-key-name="$KEY"
Derived from Use customer-managed encryption keys (Cloud Storage), Secret Manager CMEK and gcloud storage buckets update.
For Cloud SQL, create the service agent with gcloud beta services identity create --service=sqladmin.googleapis.com,
grant it the key, then create the instance with --disk-encryption-key; CMEK on a Cloud SQL instance must be chosen
at creation and cannot be added later (see Cloud SQL). Cloud Storage CMEK applies to
new objects; existing objects keep their original key until rewritten (see
Cloud Storage Security and Static Sites). Disks, GKE node boot disks, BigQuery and
Pub/Sub have their own options. Organisation policies (constraints/gcp.restrictNonCmekServices and
constraints/gcp.restrictCmekCryptoKeyProjects) can require CMEK and restrict which projects hold the keys.
Disabling or destroying a key makes the data unreadable. That is the point of CMEK ("crypto-shredding") and also its main operational risk, so keep keys in a dedicated project with restricted access and a documented procedure.
Cloud KMS Autokey
Autokey creates keys on demand so that platform teams do not have to pre-provision key rings for every service. A folder is configured with an Autokey key project; when a developer creates a protected resource, the service asks Autokey for a key handle, and Autokey creates a key with the right location, rotation and protection level, grants the service agent, and returns it. The setup is performed by a folder administrator and works for the supported services listed in the documentation:
An administrator with roles/cloudkms.autokeyAdmin on the folder runs gcloud kms autokey-config update with the
folder and key project; a developer then runs gcloud kms key-handles create with a --resource-type-selector. See
Cloud KMS Autokey overview and
Enable Autokey.
The supported resource types and the exact flags change as services are added, so confirm them on the linked pages. Autokey suits organisations with many teams; for a single Bookshelf project a manually created key as above is simpler. Resource hierarchy and folder concepts are in Resource Hierarchy and Projects.
Terraform
resource "google_kms_key_ring" "bookshelf" {
name = "bookshelf"
location = var.region
}
resource "google_kms_crypto_key" "data" {
name = "bookshelf-data"
key_ring = google_kms_key_ring.bookshelf.id
rotation_period = "7776000s" # 90 days
lifecycle {
prevent_destroy = true
}
}
resource "google_secret_manager_secret" "db_password" {
secret_id = "catalog-db-password"
labels = { app = "bookshelf" }
replication {
auto {}
}
}
resource "google_secret_manager_secret_iam_member" "catalog_api" {
secret_id = google_secret_manager_secret.db_password.id
role = "roles/secretmanager.secretAccessor"
member = "serviceAccount:${google_service_account.catalog_api.email}"
}
Derived from the Terraform registry pages google_secret_manager_secret, google_kms_crypto_key.
Do not define the secret version in Terraform with a literal value: the payload would end up in the state file. Create the container in Terraform and add versions out of band, or use the write-only attribute that the provider documents for secret data, so the value is not stored. State handling and modules are in Infrastructure as Code.
Destroying keys and key versions
A Cloud KMS key ring and a key can never be deleted. Only key versions are destroyed, which erases the key material.
# Schedule destruction of version 1 (state becomes DESTROY_SCHEDULED)
gcloud kms keys versions destroy 1 \
--key=bookshelf-data --keyring=bookshelf --location="$REGION"
# Changed your mind within the scheduled duration
gcloud kms keys versions restore 1 \
--key=bookshelf-data --keyring=bookshelf --location="$REGION"
# Disable a version first, which is reversible and blocks use immediately
gcloud kms keys versions disable 1 \
--key=bookshelf-data --keyring=bookshelf --location="$REGION"
Derived from Destroy and restore key versions.
Disabling is the first step; scheduling destruction waits for the key’s destroy-scheduled duration before the material is erased. Everything encrypted with that version is lost once it is. Billing for a version stops when it is destroyed. Before destroying, check usage in the console key page, the Cloud Asset Inventory and the Key Access Justifications or audit logs.
Application-level encryption with KMS
Services that offer CMEK encrypt on your behalf. Application-level encryption of fields (for example client-side
field-level encryption in MongoDB) uses a KMS key as the master key that wraps data encryption keys. The MongoDB
version of that pattern, including the driver configuration for a Google Cloud KMS provider, is in
Authentication, authorization & encryption; on Google Cloud the master key is a symmetric KMS key in your project,
accessible through the same Application Default Credentials and cryptoKeyEncrypterDecrypter role as above. Document
databases are compared in MongoDB on Google Cloud.
Security checklist
-
One secret per credential, bound to the single service account that needs it, with
secretAccessoron the secret and not on the project. -
No service-account key files, and no secrets in committed manifests, container images or build logs.
-
Enable Data Access logs for Secret Manager, and use delayed destruction so an accidental destroy can be undone.
-
Keep CMEK keys in a separate project with distinct admins and users, and
prevent_destroyon key resources.
Pricing model
Pricing changes, so no figures are quoted here; use the dated pricing pages (checked 2026-09-30).
-
Secret Manager charges for each active secret version per location per month, for access operations (
AccessSecretVersionand similar) in blocks, and for rotation notifications. A small monthly free allotment of versions and operations is included. A user-managed secret replicated to several locations counts a version once per location. Disabled and destroyed versions are not billed as active. -
Cloud KMS charges per key version per month, with higher rates for Cloud HSM and EKM, and per cryptographic operation, with a free allotment for software keys. Many versions that are rotated but never destroyed accumulate cost; a destroyed version stops billing.
-
Log volume for Data Access audit logs, Pub/Sub messages for rotation and the function that rotates the secret are billed through their own services.
Cost drivers to watch: lots of secrets with several old versions, an application that calls AccessSecretVersion
on each request, and HSM keys with frequent rotation. The pricing pages are
Secret Manager pricing and
Cloud KMS pricing. The general cost tools are in
Getting Started with Google Cloud.
Quotas and limits
A secret version payload is limited to 64 KiB, direct KMS encryption to 64 KiB of plaintext, and both services apply per-project, per-minute request quotas (see Secret Manager quotas and Cloud KMS quotas). Key rings and keys cannot be deleted, so plan names.
Launch stages and recent changes
| Date | Change |
|---|---|
2024-08-20 |
Secret Manager add-on for GKE generally available. |
2024-10-14 |
Regional secrets generally available. |
2025-09-01 |
Automatic rotation for the GKE add-on’s mounted secrets generally available. |
2026-04-16 |
Sync of add-on secrets to Kubernetes Secrets generally available. |
2026-06 |
Multi-region locations for secrets. |
The release notes linked below are the authority for newer dates. Older guides still show the Secret Manager integration with Cloud Functions as the way to read secrets, or the Kubernetes CSI driver you installed yourself; today the first-class paths are the Cloud Run flag and the managed GKE add-on.
Troubleshooting
| Symptom | Likely cause and fix |
|---|---|
|
The runtime service account lacks |
Cloud Run deployment fails with "Permission denied on secret" |
Grant |
Cloud Run cannot find a regional secret |
Cloud Run supports only global secrets. Create a global secret or read the regional one from application code. |
|
No enabled version exists (all were disabled or destroyed), or the secret and the version are in different locations or projects. |
A rotated value is not picked up |
The application resolved the secret at start-up (environment variable or Spring property). Restart, or switch to a volume mount, or refresh in code. |
GKE pod stuck in |
Workload Identity Federation is missing, the principal lacks |
CMEK resource creation fails with a key permission error |
The service agent was not created or lacks |
Data became unreadable |
The key (or the version that encrypted it) was disabled or destroyed. Re-enable it within the scheduled destruction period; after that the data cannot be recovered. |
A quota error ( |
Too many access calls. Cache values in the client and reuse the client object. |
Clean up
Delete in dependency order: consumers first, then secrets, then keys. Secret versions bill while active, and KMS key versions bill until they are destroyed, so a lab that forgets this keeps paying a little. Skip the commands for resources you did not create.
# Consumers: stop referencing the secret
gcloud run services update catalog-api --region="$REGION" \
--remove-secrets=DB_PASSWORD
gcloud run services update catalog-api --region="$REGION" \
--remove-secrets=/secrets/db/password
# Secrets (this deletes all versions and cannot be undone)
gcloud secrets delete catalog-db-password --quiet
gcloud secrets delete sendgrid-api-key --quiet
gcloud secrets delete regional-demo --location="$REGION" --quiet
gcloud secrets delete regional-cmek-demo --location="$REGION" --quiet
# Rotation plumbing and service account
gcloud pubsub topics delete secret-rotation
gcloud iam service-accounts delete "$RUNTIME_SA" --quiet
# GKE add-on (only if you enabled it for this lab)
gcloud container clusters update "$CLUSTER" --location="$REGION" --no-enable-secret-manager
# KMS: schedule destruction of every version; the key and key ring remain
for V in $(gcloud kms keys versions list --key=bookshelf-data --keyring=bookshelf \
--location="$REGION" --filter='state=ENABLED OR state=DISABLED' --format='value(name.basename())'); do
gcloud kms keys versions destroy "$V" --key=bookshelf-data --keyring=bookshelf --location="$REGION"
done
Derived from Delete a secret, Destroy and restore key versions and gcloud run services update.
Before destroying the KMS versions, make sure that no bucket, instance or secret still depends on the key: re-key or
delete them first, because their data becomes unreadable once the scheduled period passes. Key rings and keys cannot
be deleted, only emptied, and cost nothing without active versions. Cloud Storage CMEK settings are removed with
gcloud storage buckets update --clear-default-encryption-key. If you deleted a secret by mistake, there is no
undelete; delayed destruction applies to versions, not to the secret itself. For a throwaway project, deleting the
project schedules everything for deletion with the usual 30-day recovery window (see
Resource Hierarchy and Projects).
Related pages
-
Runtime Configuration — Parameter Manager, environment variables, ConfigMaps and secret references.
-
Identity and Access — service accounts, Workload Identity Federation and deny policies.
-
Cloud Run — deploying
catalog-apiwith secrets. -
Deploying Microservices on GKE — the Secret Manager add-on in a Bookshelf cluster.
-
Cloud SQL — the database that uses
catalog-db-password. -
Cloud Storage Security and Static Sites — CMEK on buckets.
-
Sending Email from Google Cloud — the
sendgrid-api-keyconsumer. -
Authentication, authorization & encryption — field-level encryption with a KMS master key.
-
Configuration and Profiles — Spring profiles and property sources.
-
AWS (Amazon Web Services) and Azure — the AWS and Azure secret and key services.