Secrets Manager, Parameter Store and KMS

Within the customer managed category, the key spec and key usage decide what the key can do:

The origin of the key material can be KMS itself (the default), imported material you supply, a CloudHSM key store (a single-tenant HSM cluster you own, for compliance that requires dedicated hardware) or an external key store (the key stays in an HSM outside AWS). CloudHSM and external key stores add cost and operational burden and are only worth it when a regulation names them; Bookshelf uses the default origin.

Three mechanisms authorise key use, and they combine differently from most AWS services:

A key policy for the Bookshelf data key alias/bookshelf-data that separates administrators (whose statement grants no kms:Decrypt) from users (the catalog-api task role and the Secrets Manager service acting for it):

The kms:ViaService condition means the role can use the key only when the request comes from Secrets Manager or Systems Manager on its behalf, never by calling kms:Decrypt directly. The root statement is what keeps the key manageable if the administrator role is deleted; removing it is how people lock themselves out of a key, and then need AWS Support to recover it. Do not put user-supplied data in the key policy; see Key policies in AWS KMS.

Encrypt accepts at most 4,096 bytes, and sending large payloads to a remote service is slow. KMS and the AWS Encryption SDK therefore use envelope encryption: ask KMS for a unique data key, encrypt the data locally with the plaintext copy, discard it, and store the encrypted copy of the data key next to the ciphertext. To decrypt, send the stored encrypted data key to KMS (Decrypt), receive the plaintext data key and decrypt locally. Only the small data key ever travels to KMS, and each object gets its own key.

The encryption context is a set of non-secret key/value pairs (for example app=bookshelf and tenant=<tenant-id>) that is cryptographically bound to the ciphertext, logged in CloudTrail in plaintext and required again on decryption. Use it to tie a ciphertext to its purpose and to write policies such as "decrypt only when kms:EncryptionContext:app equals `bookshelf`". AWS services do this for you: S3, EBS, RDS and Secrets Manager all use envelope encryption internally with their own data keys under your KMS key.

Rotation replaces the key material while the key ID, alias and policy stay the same; ciphertext made under older material still decrypts, because KMS keeps every version. Only symmetric keys with KMS-generated material support it; asymmetric, HMAC and imported-material keys do not, and you rotate those by creating a new key and updating the alias.

Rotation does not re-encrypt existing data: old ciphertexts keep working and new encryptions use the new material. Automatic rotations are included in the key price; on-demand rotations are charged like API requests. See Rotate AWS KMS keys.

By default a KMS key exists in one Region, and ciphertext from us-east-1 cannot be decrypted by a different key in eu-west-1. A multi-Region key is a set of interoperable keys with the same key ID (prefixed mrk-) and the same key material in different Regions, so data encrypted in one Region can be decrypted in another without a cross-Region call. Typical uses are global applications, disaster recovery, DynamoDB global tables, Aurora global databases, and Secrets Manager replicas. Each replica has its own key policy, alias and grants, and a single primary key controls shared properties such as rotation.

Multi-Region keys are not a way to share one key across Regions' IAM boundaries: you still manage each replica’s policy. See Multi-Region keys in AWS KMS.

Deleting a KMS key destroys its key material and makes everything encrypted under it unrecoverable (a deleted symmetric key cannot be recreated, even from the same material). Because of that, AWS never deletes a key immediately: you schedule deletion with a mandatory waiting period of 7 to 30 days (default 30; the real deletion can take up to 24 hours longer). While the key is PendingDeletion it cannot be used for any cryptographic operation, and you can cancel the deletion at any time before the period ends; afterwards the key, its aliases and metadata are gone. If you are unsure, disable the key instead (reversible).

Create a CloudWatch alarm on use attempts during the waiting period (any Decrypt of a pending key is evidence that something still needs it). A multi-Region primary key cannot be deleted until all its replicas are; a replica can be deleted at any time. See Delete an AWS KMS key.

Rotation changes the credential in both Secrets Manager and the service that checks it, using a four-step state machine run by a Lambda rotation function that Secrets Manager invokes:

Three rotation strategies exist, in order of preference:

The rotation function runs in your VPC when the database is private, so it needs a route to the Secrets Manager endpoint (a VPC interface endpoint or a NAT gateway, see VPC Networking) and permission to call Secrets Manager and decrypt the key. Applications must re-read the secret after a rotation (see the consumption patterns below); a connection pool that caches the old password for its lifetime fails at the next reconnect. Details are in Rotate AWS Secrets Manager secrets and the databases' own pages: Amazon RDS and Amazon Aurora.

Managed external secrets are a secret type, introduced in 2025, that stores and automatically rotates credentials for supported integration partners (SaaS vendors that publish a predefined secret format). Rotation runs without a Lambda function in your account, the partner defines the secret’s required JSON shape, and the full Secrets Manager feature set (IAM and resource policies, replication, CloudTrail, EventBridge) applies. In the console it is the Managed external secret option in Store a new secret; the partner list and programmatic creation steps are in Using Secrets Manager managed external secrets. Use it for partner API keys instead of hand-writing one more rotation function.

Like an S3 bucket policy, a resource-based policy attached to a secret says who may use it, which is how you share a secret across accounts, or add a guard rail on top of IAM. It is evaluated together with the caller’s identity policies; for a caller in another account both must allow the request, and the secret must use a customer managed key whose key policy also allows that account.

Resource in a secret’s resource policy is always "*", meaning "this secret". Test a draft with validate-resource-policy (it flags public access and syntax problems) before attaching it; a wrong deny can lock out even your administrators. See Attach a permissions policy to a secret.

A secret can be replicated to other Regions for disaster recovery or to serve workloads close to users. Each replica is a read-only copy, encrypted with a KMS key you choose in the destination Region, kept in sync with the primary (including rotation results). A replica is billed as a secret. To make a replica an independent writable secret (failover), promote it.

Use a multi-Region KMS key (Multi-Region keys) to keep the encryption consistent, or a per-Region key if the Regions' compliance rules differ. Reference: Replicate AWS Secrets Manager secrets to other Regions.

Every Secrets Manager and KMS call is recorded by CloudTrail, including who read GetSecretValue and the KMS Decrypt it triggered, which is the main reason to prefer Secrets Manager over a plain parameter for credentials. Secrets Manager also publishes events to EventBridge (rotation start/success/failure, value changes, deletion scheduled) that you can route to SNS or a Lambda function:

Monitoring and alerting patterns are on Monitoring and Observability. See Monitor AWS Secrets Manager secrets.

delete-secret does not remove a secret immediately. It marks the secret for deletion with a recovery window of 7 to 30 days (default 30) during which it cannot be read, its name cannot be reused, and restore-secret brings it back; afterwards it is deleted permanently. --force-delete-without-recovery skips the window and is immediate and irreversible, which is appropriate only for throwaway secrets (a test environment’s, or one created by mistake).

Replicas must be removed (remove-regions-from-replication) before the primary can be deleted. A common outage is deleting a secret that a running service still reads: the task or pod keeps working until its next restart or cache refresh, then fails. Check describe-secret and CloudTrail GetSecretValue activity first.

Reference: AWS Systems Manager Parameter Store and the AWS CLI Command Reference — ssm.

Parameter Store can also read a Secrets Manager secret through the reserved path /aws/reference/secretsmanager/<secret-name>, so one ssm:GetParameter call (with --with-decryption) works for both stores; a service that has only Parameter Store integration, such as an older CloudFormation or Lambda feature, can still get a rotated secret that way. Use IAM permissions on ssm:GetParameter and secretsmanager:GetSecretValue for that path.

AWS SDK for Java 2.x with the default credential chain; SecretsManagerClient calls cost money and add latency, so cache the value and refresh it:

For production, prefer the maintained caching library (com.amazonaws.secretsmanager:aws-secretsmanager-caching-java, whose 2.x line uses the SDK v2) with its SecretCache and SecretCacheConfiguration, which also handles version stages and refresh jitter, and read the AWS SDK for Java 2.x Get a Secrets Manager secret value using Java. Never log the secret JSON.

Bookshelf’s catalog-api is a Spring Boot service, so it imports secrets and parameters straight into the Environment with Spring Cloud AWS (4.x for Spring Boot 4, 3.4.x for Spring Boot 3.5), using the credentials of the task role. The mechanics of spring.config.import, profiles and precedence are on Configuration and Profiles; the Bookshelf-specific lines are:

The keys of the secret’s JSON (username, password, host, port, dbname) become properties, so the datasource above resolves from the secret; Parameter Store entries under the prefix become properties named by their last path segments. Spring Cloud AWS reads values at start-up; to pick up a rotated password at run time enable its reload support (spring.cloud.aws.secretsmanager.reload.strategy, see the Spring Cloud AWS reference) or restart the service after rotation. The optional: prefix keeps local development working without AWS credentials.

In a task definition, secrets entries (not environment) tell the ECS agent to fetch the value when the task starts and inject it as an environment variable. The task execution role (not the task role) needs the read permissions, because ECS resolves them before your code runs. See ECS and Fargate for the roles themselves.

The format for Secrets Manager is secret-arn:json-key:version-stage:version-id: the trailing :: means "`AWSCURRENT`, latest version", and omitting the :password:: suffix injects the whole JSON string. Parameter Store entries use the parameter ARN, or just the name when the parameter is in the same Region. The execution role needs secretsmanager:GetSecretValue, ssm:GetParameters and kms:Decrypt on the keys involved (the policy above, attached to the execution role). The value is resolved once, at task start: after a rotation (or a new put-secret-value) start a new deployment with aws ecs update-service --cluster bookshelf --service catalog-api --force-new-deployment so the tasks restart and re-read, or let the application read the secret itself through the SDK. Reference: Pass Secrets Manager secrets through Amazon ECS environment variables and Pass Systems Manager parameters.

Lambda’s own environment variables are encrypted but not rotated, so credentials usually come from a secret at run time. Add the AWS Parameters and Secrets Lambda Extension as a layer: it runs a local HTTP cache on port 2773 that calls Secrets Manager or Parameter Store with the function’s execution role, caches results (TTL up to 300 seconds, default 300) and spares your function an SDK dependency. The same local-endpoint design is also available as the AWS Workload Credentials Provider (formerly named the Secrets Manager Agent), a standalone binary for containers, EC2 and Lambda, which uses the same port and the X-Aws-Parameters-Secrets-Token header.

The function then calls the local endpoint with the session token as the header (the extension rejects requests without it; this is its SSRF protection):

Parameter Store values use /systemsmanager/parameters/get?name=<url-encoded-name>&withDecryption=true. The execution role needs the same secretsmanager:GetSecretValue / ssm:GetParameter and kms:Decrypt statements as above. With SnapStart the AWS_SESSION_TOKEN variable is not always set; read the token from the SDK credential provider chain instead. Because the extension has no cache invalidation, a rotated secret can be stale for up to the TTL; lower SECRETS_MANAGER_TTL or retry once after an authentication failure. See AWS Lambda, and the Use Secrets Manager secrets in Lambda functions page.

Kubernetes Secret objects are only base64-encoded; on EKS, the recommended pattern keeps the source of truth in Secrets Manager and brings values into pods in one of two ways. Both authenticate with EKS Pod Identity (see EKS Clusters and Deploying Microservices on EKS) so that only the service account that needs the secret can read it.

Secrets Store CSI Driver with the AWS Secrets and Configuration Provider (ASCP) mounts secrets as files in a volume and can sync them to a Kubernetes Secret:

The role’s trust policy uses the Pod Identity principal and the same permission policy as above:

The mount (and the synced Secret) exist only while a pod that mounts the volume runs. With enableSecretRotation the driver re-reads the secret periodically and updates the files, but environment variables from a synced Secret are fixed at pod start, so mount the file and re-read it, or restart the Deployment. Private clusters need a VPC endpoint for AWS STS and for Secrets Manager. See Use ASCP with Pod Identity for Amazon EKS and the Kubernetes primer in Orchestration: Swarm Mode and Kubernetes.

External Secrets Operator (ESO) is the alternative: a CNCF-hosted controller that polls Secrets Manager or Parameter Store and writes ordinary Kubernetes Secret objects, so pods consume them with the usual secretKeyRef or envFrom and no volume. It uses the default credential chain, so it works with the Pod Identity or IRSA role of the operator itself:

Choose ASCP when you want values to exist only as files in running pods and to keep Kubernetes Secret objects out of etcd; choose ESO when you want GitOps-friendly declarative ExternalSecret objects, support for several backends (Vault, Azure Key Vault, Google Secret Manager) or a cluster-wide store. Both bill you for Secrets Manager API calls, so set a sensible refresh interval. Runtime configuration (ConfigMaps) is on Runtime Configuration; the Spring-side view of Kubernetes Secrets is on Configuration and Profiles.

CI pipelines authenticate to AWS with OIDC federation and a short-lived role (see Identity and Access and Docker CI/CD with GitHub Actions), so no AWS key needs to be stored in GitHub at all. Keep secrets that only the pipeline needs (a registry token, a signing passphrase) in GitHub Actions secrets, and secrets that the running application needs in Secrets Manager; a workflow that must read one can do so after assuming its role, for example with the aws-actions/aws-secretsmanager-get-secrets action or aws secretsmanager get-secret-value (mask the value with ::add-mask:: before it can reach a log). The CI/CD page is Developer Tools and DevOps.