Monitoring and Observability

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

See the section bibliography for the reference material consulted while preparing these pages.

Commands and behaviour on this page are current for AWS CLI v2 (2.37 line) and the CloudWatch, CloudTrail, Amazon Managed Service for Prometheus and Amazon Managed Grafana documentation as read on 2026-10-01; re-verify against aws --version and the linked reference pages before relying on them in a pipeline. Examples use us-east-1 and the placeholder account ID 111122223333; both are stand-ins for your own values.

Every other page in this section shows how to build one piece of Bookshelf — ECS services on Fargate, Lambda functions, an EKS cluster, Aurora, DynamoDB, CloudFront. This page answers the question that comes after all of that exists: "is it healthy, and what happened at 03:14 this morning?". It covers four different questions that AWS answers with four different services:

  • What is the system doing right now? — Amazon CloudWatch metrics, logs and alarms.

  • Why is one request slow? — traces, through CloudWatch Application Signals and OpenTelemetry.

  • Who changed what? — AWS CloudTrail.

  • Can I keep the Prometheus and Grafana tooling I already know? — Amazon Managed Service for Prometheus and Amazon Managed Grafana.

The application-side half of the story (what to log, which Micrometer meters to publish, how to expose them) lives in the Spring Boot pages on this site and is linked rather than repeated: Logging: Logback Defaults, Structured Output, and Container-Friendly Routing and Metrics and Observability: Micrometer, Actuator, and Distributed Tracing. The Prometheus data model, PromQL and Alertmanager are documented once in Prometheus & Monitoring; this page only shows how AWS runs them for you. For the Azure counterpart see Monitoring and Observability.

The AWS observability map

flowchart LR subgraph Sources["Bookshelf sources"] ECS["ECS tasks\ncatalog-api, orders-api"] LAM["Lambda\nnotifier, thumbnailer"] EKS["EKS pods\nbookshelf-eks"] ALB["ALB, CloudFront,\nAurora, DynamoDB"] end ECS -- "awslogs / stdout" --> CWL[("CloudWatch Logs\nlog groups")] LAM -- "automatic" --> CWL EKS -- "CloudWatch agent" --> CWL ALB -- "service metrics" --> CWM[("CloudWatch Metrics")] ECS -- "OTLP traces + metrics" --> AS["Application Signals\n+ X-Ray traces"] EKS -- "remote write" --> AMP[("Managed Prometheus\nworkspace")] CWL -- "metric filter / EMF" --> CWM CWL --> LI["Logs Insights\nLive Tail"] CWM --> ALM["Alarms\n(metric, composite)"] ALM --> SNS["SNS topic\nbookshelf-oncall"] AMP --> GRAF["Managed Grafana"] CWM --> GRAF API["Every API call"] --> CT[("CloudTrail\nevent history, trail")] CT --> EB["EventBridge rule"] EB --> SNS

Three ideas organise everything below. Metrics are cheap numeric time series that exist as soon as a resource exists. Logs are records you must send somewhere and pay to ingest and store. Traces connect one request across services. CloudTrail is separate on purpose: it records control-plane and data-plane API activity about your account, not what your application does.

CloudWatch metrics

A CloudWatch metric is identified by a namespace (AWS/ECS, AWS/Lambda, AWS/ApplicationELB), a metric name (CPUUtilization) and zero or more dimensions (ClusterName, ServiceName). Most AWS services publish metrics into their own AWS/…​ namespace automatically and for free; some, such as detailed EC2 monitoring or Container Insights, are opt-in and billed. Metrics are regional, and they are kept with decreasing resolution over time (per the CloudWatch concepts page, 1-minute data points are aggregated to 5-minute and then 1-hour ones as they age, for up to 15 months), so a graph of last quarter is coarser than one of last hour.

Custom metrics are anything you publish yourself. Two routes exist:

PutMetricData

One API call per batch of data points, in a namespace you choose (for example Bookshelf/Orders). Simple, but every unique combination of metric name and dimensions is a separately billed metric.

Embedded metric format (EMF)

You write a structured JSON log line containing an _aws block that tells CloudWatch which fields are metrics and which are dimensions; CloudWatch Logs extracts the metrics for you. It fits Lambda and containers well because the only thing the code needs is stdout.

{
  "_aws": {
    "Timestamp": 1790000000000,
    "CloudWatchMetrics": [
      {
        "Namespace": "Bookshelf/Orders",
        "Dimensions": [["Service"]],
        "Metrics": [{ "Name": "OrdersPlaced", "Unit": "Count" }]
      }
    ]
  },
  "Service": "orders-api",
  "OrdersPlaced": 1,
  "orderId": "ord-1001"
}

Derived from Amazon CloudWatch User Guide — Embedded metric format specification.

Dimensions are the cost lever. A dimension with unbounded values (a user ID, an order ID) creates one billed metric per value; keep such values in the log line as ordinary properties (as orderId above) and out of the dimension list.

For Java applications, Micrometer is the usual front end: Spring Cloud AWS ships a CloudWatch metrics starter that publishes the meters described on Metrics and Observability: Micrometer, Actuator, and Distributed Tracing, or you can keep the Prometheus endpoint and scrape it (see Managed Prometheus below).

In the AWS Management Console

  1. Open CloudWatch and pick the Region in the top-right selector (metrics do not cross Regions).

  2. Choose Metrics → All metrics, then a namespace such as AWS/ECS.

  3. Select ClusterName, ServiceName, tick CPUUtilization for catalog-api, and use the Graphed metrics tab to change the statistic (Average, p99) and period.

  4. Choose Add to dashboard to place the widget on a dashboard.

With the AWS CLI

export AWS_REGION=us-east-1

# What namespaces and metrics exist for the catalog-api service?
aws cloudwatch list-metrics \
  --namespace AWS/ECS \
  --dimensions Name=ServiceName,Value=catalog-api \
  --query 'Metrics[].MetricName' --output text

# Publish a custom data point
aws cloudwatch put-metric-data \
  --namespace Bookshelf/Orders \
  --metric-name OrdersPlaced \
  --dimensions Service=orders-api \
  --value 1 --unit Count

# Read the last hour of average CPU, one point per five minutes
aws cloudwatch get-metric-statistics \
  --namespace AWS/ECS --metric-name CPUUtilization \
  --dimensions Name=ClusterName,Value=bookshelf Name=ServiceName,Value=catalog-api \
  --start-time "$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ)" \
  --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --period 300 --statistics Average

The date -v-1H form is the macOS/BSD spelling; on GNU/Linux or in CloudShell use date -u -d '1 hour ago'.

Derived from AWS CLI Command Reference — cloudwatch put-metric-data and AWS CLI Command Reference — cloudwatch get-metric-statistics.

CloudWatch alarms

An alarm watches one metric (or a metric-math expression) and moves between OK, ALARM and INSUFFICIENT_DATA. When the state changes it runs actions: publish to an SNS topic, trigger an Auto Scaling policy (the ECS and EC2 pages show target-tracking policies that create their own alarms), or start a Systems Manager action. Three details decide whether an alarm is useful:

  • Evaluation periods and datapoints to alarm. "3 out of 5 one-minute periods" tolerates a single noisy minute.

  • Missing-data treatment. A service that stops emitting data is often the failure; choose breaching for counts that should never stop, notBreaching for sparse metrics.

  • Anomaly detection bands learn the metric’s usual shape and alarm on deviation, which suits traffic with daily and weekly rhythm better than a fixed threshold.

Composite alarms combine other alarms with AND, OR and NOT so that a page goes out only when the symptom and the cause both fire (for example "ALB 5xx high AND no healthy hosts", not either alone). A composite alarm can also suppress the notifications of its children, which is how a single incident stops producing twenty pages.

Bookshelf’s on-call alarm watches the target 5xx count behind alb-bookshelf:

aws sns create-topic --name bookshelf-oncall
aws sns subscribe \
  --topic-arn "arn:aws:sns:us-east-1:111122223333:bookshelf-oncall" \
  --protocol email --notification-endpoint oncall@example.com

aws cloudwatch put-metric-alarm \
  --alarm-name catalog-api-5xx-high \
  --namespace AWS/ApplicationELB \
  --metric-name HTTPCode_Target_5XX_Count \
  --dimensions Name=LoadBalancer,Value=app/alb-bookshelf/0123456789abcdef \
  --statistic Sum --period 60 \
  --evaluation-periods 5 --datapoints-to-alarm 3 \
  --threshold 10 --comparison-operator GreaterThanOrEqualToThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions "arn:aws:sns:us-east-1:111122223333:bookshelf-oncall" \
  --ok-actions "arn:aws:sns:us-east-1:111122223333:bookshelf-oncall"

aws cloudwatch put-composite-alarm \
  --alarm-name catalog-api-down \
  --alarm-rule 'ALARM("catalog-api-5xx-high") AND ALARM("catalog-api-no-healthy-hosts")' \
  --alarm-actions "arn:aws:sns:us-east-1:111122223333:bookshelf-oncall"

The load-balancer dimension value (app/<name>/<id>) is the tail of the load balancer ARN; read it from aws elbv2 describe-load-balancers. The SNS subscription must be confirmed from the email it sends before notifications flow. The second alarm (catalog-api-no-healthy-hosts) is created the same way on the UnHealthyHostCount metric.

Derived from AWS CLI Command Reference — cloudwatch put-metric-alarm, AWS CLI Command Reference — cloudwatch put-composite-alarm and Amazon CloudWatch User Guide — Using Amazon CloudWatch alarms.

In the AWS Management Console

  1. In CloudWatch, choose Alarms → All alarms → Create alarm.

  2. Choose Select metric, navigate to ApplicationELB → Per AppELB, per TG Metrics, and select HTTPCode_Target_5XX_Count for the catalog-api target group.

  3. Set statistic Sum, period 1 minute, threshold type Static, Greater/Equal 10; under Additional configuration set Datapoints to alarm to 3 out of 5 and Missing data treatment to Treat missing data as good.

  4. On the next page choose In alarm → Select an existing SNS topic → bookshelf-oncall, then name the alarm and create it.

  5. To combine alarms, choose Create alarm → Create composite alarm and write the rule expression.

CloudWatch Logs

Logs land in log groups (one per application or function, with shared retention, encryption and access settings) containing log streams (one per task, instance or function container). The services in this section reach a log group in different ways:

Source How logs arrive

Lambda

Automatic: every function writes to /aws/lambda/<function-name>; the execution role needs the logs:CreateLogStream and logs:PutLogEvents permissions (the managed AWSLambdaBasicExecutionRole has them)

ECS on Fargate

The task definition’s awslogs log driver, with the task execution role allowed to write; see ECS and Fargate

EKS

The Amazon CloudWatch Observability add-on (CloudWatch agent plus Fluent Bit) or your own log shipper; see EKS Clusters

EC2

The unified CloudWatch agent

Other AWS services

Their own switch: VPC Flow Logs, API Gateway access logs, CloudFront standard logging, Route 53 resolver query logs

Two settings matter more than the rest. Retention defaults to never expire, so a forgotten log group keeps billing storage forever; set it explicitly (the console and put-retention-policy offer fixed values from one day upwards). Log classes trade features for price: the Standard class supports everything, while a cheaper infrequent-access class suits logs you rarely query and does not support every feature; check the Logs User Guide for the current feature matrix before choosing.

Metric filters turn matching log events into metrics, which alarms can then watch:

aws logs create-log-group --log-group-name /bookshelf/catalog-api
aws logs put-retention-policy \
  --log-group-name /bookshelf/catalog-api --retention-in-days 30

aws logs put-metric-filter \
  --log-group-name /bookshelf/catalog-api \
  --filter-name ErrorCount \
  --filter-pattern '"ERROR"' \
  --metric-transformations \
      metricName=CatalogErrors,metricNamespace=Bookshelf/Catalog,metricValue=1,defaultValue=0

# Follow the log group live from the terminal
aws logs tail /bookshelf/catalog-api --follow --since 10m --format short

aws logs tail is the command-line cousin of Live Tail in the console, which streams matching events as they arrive and is handy while a deployment rolls out. Log data is encrypted at rest by default; you can attach a KMS key to a log group (see Secrets Manager, Parameter Store and KMS), and the data protection policy feature can mask sensitive patterns such as email addresses before they are stored.

Derived from AWS CLI Command Reference — logs put-metric-filter, AWS CLI Command Reference — logs tail and Amazon CloudWatch Logs User Guide — Working with log groups and log streams.

In the AWS Management Console

  1. In CloudWatch, choose Logs → Log management, then Create log group.

  2. Open the group, choose the Retention column value (initially Never expire) and set 30 days.

  3. Choose Actions → Create metric filter, enter the pattern "ERROR", name the metric CatalogErrors in the Bookshelf/Catalog namespace, and save.

  4. Choose Start tailing to open Live Tail on the group.

Logs Insights

CloudWatch Logs Insights queries one or more log groups interactively. It supports three query languages, chosen per query: the purpose-built Logs Insights QL (a pipe-based language with fields, filter, stats, sort, parse and limit), OpenSearch PPL and OpenSearch SQL. It discovers fields automatically in JSON logs and in the logs of services such as Lambda, CloudTrail, VPC Flow Logs and Route 53; field indexes let a query skip events that cannot match, which lowers both time and scanned-data cost.

Bookshelf’s recent catalog-api errors, with the Spring Boot JSON log layout from Logging: Logback Defaults, Structured Output, and Container-Friendly Routing:

fields @timestamp, @logStream, level, message
| filter level = "ERROR"
| sort @timestamp desc
| limit 50

Latency percentiles per five-minute bin for a Lambda function (Lambda writes REPORT lines with @duration):

filter @type = "REPORT"
| stats avg(@duration), pct(@duration, 95), max(@maxMemoryUsed / 1000 / 1000) as maxMemMB by bin(5m)

From the CLI a query is asynchronous: start it, then poll for results.

QUERY_ID=$(aws logs start-query \
  --log-group-names /bookshelf/catalog-api \
  --start-time "$(( $(date +%s) - 3600 ))" \
  --end-time "$(date +%s)" \
  --query-string 'fields @timestamp, message | filter level = "ERROR" | sort @timestamp desc | limit 20' \
  --query 'queryId' --output text)

aws logs get-query-results --query-id "$QUERY_ID"

Derived from AWS CLI Command Reference — logs start-query and Amazon CloudWatch Logs User Guide — Analyzing log data with CloudWatch Logs Insights.

Pricing is per gigabyte of uncompressed log data scanned, whatever the language, so narrow the time range and the list of log groups before you run an expensive query. Per the Logs User Guide, a query times out after 60 minutes, its results stay available for 7 days, and an account can run 100 concurrent Logs Insights QL queries (15 each for PPL and SQL). Logs Insights cannot see events older than the log group itself.

In the AWS Management Console

  1. In CloudWatch, choose Logs → Logs Insights.

  2. Choose the query language, select /bookshelf/catalog-api in Select log group(s), and set the time range.

  3. Paste the query and choose Run query; use Save to keep it and Add to dashboard to chart a stats result.

Tracing: Application Signals, OpenTelemetry and X-Ray

Tracing follows one request across catalog-api, orders-api and the databases behind them. AWS now builds this on OpenTelemetry (OTel), the vendor-neutral instrumentation standard, in two layers:

Instrumentation

The OpenTelemetry SDKs and agents in your process, or the AWS Distro for OpenTelemetry (ADOT), AWS’s supported distribution of them. For a Spring Boot service the lowest-effort option is a Java agent attached at start-up, so no code changes are needed.

Collection and backend

The CloudWatch agent or an OpenTelemetry Collector receives OTLP data and forwards traces to AWS X-Ray (the trace store behind CloudWatch’s Traces and Trace Map pages) and metrics to CloudWatch.

CloudWatch Application Signals sits on top: from the OpenTelemetry data it derives standard application metrics (call volume, availability, latency, faults, errors) per service and operation, draws an automatically discovered topology map, lets you define service level objectives (SLOs) and alarm on their burn, and shows change events (deployments, configuration changes) next to the metrics. Per the User Guide it supports Java, Python, Node.js and .NET applications on Amazon EKS, Amazon ECS, Amazon EC2 and Lambda, and works with CloudWatch RUM and Synthetics canaries. On EKS the CloudWatch Observability add-on can instrument workloads for you; on ECS and EC2 you run the CloudWatch agent beside the application and attach the auto-instrumentation; on Lambda you add an AWS-provided OpenTelemetry layer. Using Application Signals is billed; see the CloudWatch pricing page.

The X-Ray SDK and daemon are in maintenance mode

Older tutorials and the X-Ray developer guide chapters on its language SDKs show the X-Ray SDK and the X-Ray daemon as the way to trace an application. That is no longer the recommended path: according to the X-Ray documentation, on 2026-02-25 the X-Ray SDKs and daemon entered maintenance mode, in which AWS limits releases to security fixes and ships no new features. The X-Ray service itself, its console experience and its APIs are not going away; what changes is how data gets in. The migration guide recommends OpenTelemetry SDKs or ADOT for instrumentation and the CloudWatch agent or an OpenTelemetry Collector (with the X-Ray exporter) instead of the daemon. As of 2026-10-01 the official support-timeline page lists maintenance mode but no end-of-support date; earlier AWS communication gave 2027-02-25 as a planned end of support, so plan to be migrated before that date rather than waiting for confirmation.

The mapping is mechanical: an X-Ray segment becomes a server span, a subsegment becomes another span, annotations and metadata become span attributes, and the X-Ray daemon becomes the collector. Context propagation switches from the X-Amzn-Trace-Id header to W3C Trace Context (the OTel X-Ray propagator is available when a service in the path still needs the AWS header, such as API Gateway).

Bookshelf: tracing catalog-api on ECS

The service needs a collector reachable from the task and a Java agent on the command line. Conceptually:

# In the container definition of catalog-api (values, not a full task definition):
#   JAVA_TOOL_OPTIONS   = -javaagent:/otel-auto-instrument/javaagent.jar
#   OTEL_SERVICE_NAME   = catalog-api
#   OTEL_RESOURCE_ATTRIBUTES = deployment.environment.name=prod
#   OTEL_EXPORTER_OTLP_PROTOCOL = http/protobuf
#   OTEL_EXPORTER_OTLP_ENDPOINT = http://localhost:4318   # CloudWatch agent sidecar
#   OTEL_AWS_APPLICATION_SIGNALS_ENABLED = true

# The task role (not the execution role) needs permission to send telemetry:
aws iam attach-role-policy \
  --role-name bookshelf-catalog-api-task-role \
  --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
  --output text

aws iam attach-role-policy \
  --role-name bookshelf-catalog-api-task-role \
  --policy-arn arn:aws:iam::aws:policy/AWSXrayWriteOnlyAccess

The Application Signals "enable on ECS" page gives the exact sidecar, init container and agent paths, which change between releases; treat the variables above as the shape of the configuration and copy the current task-definition snippet from that page. The two managed policies are broad convenience policies; for production, scope an inline policy to xray:PutTraceSegments, xray:PutTelemetryRecords, xray:GetSamplingRules and xray:GetSamplingTargets (and cloudwatch:PutMetricData and logs:* writes only if the sidecar needs them).

Derived from Amazon CloudWatch User Guide — Application Signals, Amazon CloudWatch User Guide — Enable your applications on Amazon ECS and AWS X-Ray Developer Guide — Migrating from X-Ray instrumentation to OpenTelemetry instrumentation.

In the AWS Management Console

  1. In CloudWatch, choose Application Signals → Services and, the first time, Start discovering your Services (this creates the service-linked role and enables discovery).

  2. Follow Configure Application Signals for your platform (EKS, ECS, EC2 or Lambda); on EKS this installs the CloudWatch Observability add-on.

  3. After traffic flows, open Services, choose catalog-api, and use Service operations, Dependencies and the correlated Traces; open Application Map for the topology.

  4. Choose Service Level Objectives → Create SLO to set, for example, 99.9% availability over a rolling window.

With the AWS CLI

# Turn on service discovery (service-linked role + change-event detection)
aws application-signals start-discovery

aws application-signals list-services \
  --start-time "$(( $(date +%s) - 3600 ))" --end-time "$(date +%s)" \
  --query 'ServiceSummaries[].KeyAttributes.Name' --output text

Derived from AWS CLI Command Reference — application-signals start-discovery.

Other CloudWatch capabilities

Container Insights

Collects and aggregates cluster-, node-, task- and pod-level metrics and logs for ECS and EKS (CPU, memory, network, restarts) and presents curated dashboards. Enable it per ECS cluster with aws ecs update-cluster-settings --cluster bookshelf --settings name=containerInsights,value=enhanced (the value enabled selects the older, cheaper level), and on EKS through the CloudWatch Observability add-on described on EKS Clusters. Cost scales with the number of metrics and log volume, so enable it where you will use it.

Synthetics canaries

Scripted checks (Node.js or Python on Lambda, with a browser for UI flows) that run on a schedule against your endpoints and report success rate and latency. A canary that calls https://api.example.com/health for Bookshelf is the managed form of the hand-written "website health check" Lambda found in older tutorials, and it can alarm like any other metric.

CloudWatch RUM

Real-user monitoring: a small JavaScript snippet in the static site served from CloudFront reports page-load timings and front-end errors from real browsers.

Dashboards and cross-account observability

A dashboard is a JSON document of widgets, so it deploys from CloudFormation. With Observability Access Manager a central monitoring account can view the metrics, logs and traces of many source accounts (see Accounts and Organizations).

Investigations and automatic recovery

CloudWatch can propose root causes with AI-assisted investigations. For EC2, "simplified automatic recovery" is on by default for supported instances, so the older recipe of a CloudWatch alarm with a recover action is rarely needed (see Amazon EC2 and Auto Scaling).

Database insights

For Aurora and RDS, CloudWatch Database Insights replaces the standalone Performance Insights console experience; see Amazon RDS and Amazon Aurora.

AWS CloudTrail

CloudTrail records who did what, when, from where for activity in an AWS account — it is the audit log, not the application log. It logs four categories of event:

Management events

Control-plane operations such as RunInstances, CreateBucket, PutRolePolicy or console sign-in. Recorded by default.

Data events

High-volume data-plane operations such as S3 GetObject and PutObject, Lambda Invoke and DynamoDB item operations. Not recorded by default; you opt in per resource type, and they are billed per event.

Network activity events

Calls made through VPC endpoints, useful to prove that traffic stayed on the private path.

Insights events

Statistical detection of unusual API call rates or error rates.

There are three places to read them:

  • Event history is on in every account with no setup and shows the last 90 days of management events for the current Region (read-only lookups by attribute). It is the first stop in an investigation but is not an audit store: it is per Region and cannot be extended.

  • A trail delivers events continuously to an S3 bucket (and optionally to CloudWatch Logs and EventBridge) with no 90-day limit. Create a multi-Region trail, ideally an organization trail from the management or a delegated administrator account so every member account is covered (see Accounts and Organizations), turn on log file integrity validation, and protect the bucket (block public access, KMS encryption, restrictive policy, Object Lock or MFA delete for tamper resistance).

  • CloudTrail Lake stores events in event data stores and queries them with SQL. Per the CloudTrail User Guide, Lake is no longer open to new customers from 2026-05-31; existing customers can keep using it. For a new Bookshelf account, use a trail to S3 and query it with Amazon Athena, or send the trail to CloudWatch Logs and use Logs Insights.

A trail needs a bucket that trusts the CloudTrail service principal. The bucket policy below is the documented shape for one account’s trail; replace the bucket name and account ID:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AWSCloudTrailAclCheck",
      "Effect": "Allow",
      "Principal": { "Service": "cloudtrail.amazonaws.com" },
      "Action": "s3:GetBucketAcl",
      "Resource": "arn:aws:s3:::bookshelf-trail-111122223333",
      "Condition": {
        "StringEquals": {
          "aws:SourceArn": "arn:aws:cloudtrail:us-east-1:111122223333:trail/bookshelf-trail"
        }
      }
    },
    {
      "Sid": "AWSCloudTrailWrite",
      "Effect": "Allow",
      "Principal": { "Service": "cloudtrail.amazonaws.com" },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::bookshelf-trail-111122223333/AWSLogs/111122223333/*",
      "Condition": {
        "StringEquals": {
          "s3:x-amz-acl": "bucket-owner-full-control",
          "aws:SourceArn": "arn:aws:cloudtrail:us-east-1:111122223333:trail/bookshelf-trail"
        }
      }
    }
  ]
}
aws s3api create-bucket --bucket "bookshelf-trail-$ACCOUNT_ID" --region us-east-1
aws s3api put-public-access-block --bucket "bookshelf-trail-$ACCOUNT_ID" \
  --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
aws s3api put-bucket-policy --bucket "bookshelf-trail-$ACCOUNT_ID" --policy file://trail-bucket-policy.json

aws cloudtrail create-trail \
  --name bookshelf-trail \
  --s3-bucket-name "bookshelf-trail-$ACCOUNT_ID" \
  --is-multi-region-trail \
  --enable-log-file-validation
aws cloudtrail start-logging --name bookshelf-trail

# Opt in to S3 data events for the uploads bucket only
aws cloudtrail put-event-selectors --trail-name bookshelf-trail \
  --advanced-event-selectors '[{"Name":"Uploads data events","FieldSelectors":[
    {"Field":"eventCategory","Equals":["Data"]},
    {"Field":"resources.type","Equals":["AWS::S3::Object"]},
    {"Field":"resources.ARN","StartsWith":["arn:aws:s3:::bookshelf-uploads-111122223333/"]}]}]'

# Event history: who deleted something in the last day?
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteBucket \
  --max-results 5 --query 'Events[].[EventTime,Username,EventName]' --output table

Derived from AWS CLI Command Reference — cloudtrail create-trail, AWS CloudTrail User Guide — Amazon S3 bucket policy for CloudTrail and AWS CloudTrail User Guide — CloudTrail concepts. The S3 bucket stays private; the S3 pages (S3 Security and Static Hosting) show the full hardening list.

In the AWS Management Console

  1. Open CloudTrail → Event history to search the last 90 days with no setup (filter by Event name, User name or Resource name).

  2. Choose Trails → Create trail, name it bookshelf-trail, let CloudTrail create a new S3 bucket (or pick your own), and enable Log file validation.

  3. Under Choose log events keep Management events, and add Data events for the S3 bucket bookshelf-uploads-<account-id> only if you need object-level auditing.

  4. After creation confirm the trail Apply trail to all Regions setting is Yes.

Alerting on root sign-in

The root user should almost never sign in (see Identity and Access), so a root sign-in is a high-signal event. CloudTrail delivers console sign-ins to EventBridge as management events, so an EventBridge rule can notify the on-call topic. Create the rule in the same Region where the sign-in events are delivered (root sign-ins through the global console endpoint are recorded in us-east-1; check the CloudTrail documentation for your setup) and give SNS a resource policy that lets EventBridge publish.

aws events put-rule --name root-sign-in --event-pattern '{
  "detail-type": ["AWS Console Sign In via CloudTrail"],
  "detail": { "userIdentity": { "type": ["Root"] } }
}'
aws events put-targets --rule root-sign-in \
  --targets "Id=oncall,Arn=arn:aws:sns:us-east-1:111122223333:bookshelf-oncall"

aws sns set-topic-attributes \
  --topic-arn "arn:aws:sns:us-east-1:111122223333:bookshelf-oncall" \
  --attribute-name Policy --attribute-value file://sns-eventbridge-policy.json

Derived from Amazon EventBridge User Guide — Creating rules that react to AWS API calls via CloudTrail and AWS CLI Command Reference — events put-rule. Continuous detection beyond one rule (configuration drift, threat detection, compliance packs) belongs to AWS Config, AWS Health and Trusted Advisor, and to the services on Governance, Security and Compliance.

Amazon Managed Service for Prometheus and Amazon Managed Grafana

If your teams already speak Prometheus, AWS runs the storage and the dashboards for you; the query language, data model, alerting rules and Grafana dashboards are the ones documented in this site’s Prometheus reference, starting at Prometheus & Monitoring. Read Prometheus architecture & ecosystem for the components, Storage, federation & remote write for remote write, and Containers & Kubernetes monitoring for scraping pods. What changes on AWS is operations:

Amazon Managed Service for Prometheus (AMP)

A serverless, Prometheus-compatible store. You create a workspace; a scraper or your own Prometheus server sends samples to its remote-write endpoint using SigV4-signed requests; queries (PromQL), recording rules and alerting rules run against the workspace, and alerts flow to a managed alert manager (SNS receivers). Data is replicated across three Availability Zones, and per the User Guide samples are kept for 150 days by default, configurable up to 1,095 days. An AWS-managed collector (agentless scraper) can scrape an EKS cluster with no Prometheus pods of your own.

Amazon Managed Grafana (AMG)

A managed Grafana workspace with single sign-on through IAM Identity Center or a SAML 2.0 identity provider and built-in data sources for CloudWatch, X-Ray, OpenSearch Service, Timestream and AMP. It is priced per active user in the workspace.

Choose AMP and AMG when you run EKS and already have Prometheus rules and Grafana dashboards, when you need to correlate CloudWatch and Prometheus data on one dashboard, or when you want to stop operating Prometheus storage. Choose CloudWatch alone when the workload is mostly Lambda and managed services that already emit CloudWatch metrics. Application metrics from Spring Boot reach AMP either by scraping the Actuator Prometheus endpoint (see Metrics and Observability: Micrometer, Actuator, and Distributed Tracing) or through the OpenTelemetry Collector’s Prometheus remote-write exporter.

In the AWS Management Console

  1. Open Amazon Prometheus and choose Create → Create workspace; give it the alias bookshelf.

  2. For EKS metrics choose the workspace → Managed collectors → Create, pick bookshelf-eks, subnets and security groups, and a scrape configuration.

  3. Open Amazon Managed Grafana → Create workspace, choose AWS IAM Identity Center for authentication and Service managed permissions with Amazon Managed Service for Prometheus and Amazon CloudWatch as data sources.

  4. In the Grafana workspace URL, sign in with an Identity Center user, add the AMP workspace as a data source and import a dashboard.

With the AWS CLI

WORKSPACE_ID=$(aws amp create-workspace --alias bookshelf \
  --query workspaceId --output text)
aws amp describe-workspace --workspace-id "$WORKSPACE_ID" \
  --query 'workspace.prometheusEndpoint' --output text

GRAFANA_ID=$(aws grafana create-workspace \
  --workspace-name bookshelf-grafana \
  --account-access-type CURRENT_ACCOUNT \
  --authentication-providers AWS_SSO \
  --permission-type SERVICE_MANAGED \
  --workspace-data-sources PROMETHEUS CLOUDWATCH \
  --query 'workspace.id' --output text)

The remote-write URL is the workspace’s prometheusEndpoint followed by api/v1/remote_write, and the writer (Prometheus, the managed collector or an OpenTelemetry Collector) needs aps:RemoteWrite on the workspace through its IAM role; on EKS grant it with EKS Pod Identity as described on Deploying Microservices on EKS. A minimal identity policy for a writer:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["aps:RemoteWrite"],
      "Resource": "arn:aws:aps:us-east-1:111122223333:workspace/ws-EXAMPLE-1111-2222"
    }
  ]
}

Derived from AWS CLI Command Reference — amp create-workspace, AWS CLI Command Reference — grafana create-workspace, Amazon Managed Service for Prometheus User Guide and Amazon Managed Grafana User Guide.

Pricing model

Prices change and differ by Region; use the pricing pages and the AWS Pricing Calculator for numbers, and read the dimensions below as the cost drivers rather than as amounts.

  • CloudWatch metrics: AWS service metrics are free at their default resolution; custom metrics are billed per metric per month (each metric-name and dimension combination counts), plus API requests for GetMetricData and similar; detailed or high-resolution metrics and Container Insights cost more.

  • Alarms: billed per alarm per month, with different rates for standard, high-resolution, composite and anomaly-detection alarms.

  • Logs: billed for ingestion (per gigabyte, usually the largest line item), storage (per gigabyte-month, until retention deletes it) and Logs Insights queries (per gigabyte scanned). Log class, vended-log ingestion and data protection add their own dimensions.

  • Application Signals and Synthetics and RUM: billed per signal or telemetry volume, per canary run and per RUM event respectively.

  • X-Ray traces: billed per trace recorded and retrieved; sampling is the control.

  • CloudTrail: the first copy of management events delivered to a trail is free; additional trails, data events, network activity events and Insights are billed per event; S3 storage for the trail is separate.

  • Managed Prometheus: billed for ingested samples, storage, and query samples processed; the managed collector adds an hourly charge.

  • Managed Grafana: billed per active user per month, by editor or viewer role.

The hidden-cost patterns are always the same: debug-level logging left on in production, unbounded metric dimensions, never expire retention, and data events enabled on a busy bucket. The system-wide view is on Cost Management and Support.

Limits and quotas

The numbers below were read from the official pages on 2026-10-01; most can change, and the Service Quotas console (or aws service-quotas list-service-quotas --service-code monitoring) shows your applied values.

Item Limit (source page)

Logs Insights QL concurrent queries

100 per account; PPL and SQL 15 each (Logs User Guide)

Logs Insights query duration and result retention

Time out after 60 minutes; results kept 7 days (Logs User Guide)

PutMetricAlarm rate

3 requests per second per Region, adjustable (CloudWatch quotas)

PutMetricData rate

500 requests per second per Region, adjustable (CloudWatch quotas)

Application Signals SLOs

250 per Region, 100 per service, adjustable (CloudWatch quotas)

Synthetics canaries

500 per Region, adjustable (CloudWatch quotas)

CloudWatch RUM app monitors

20 per Region, adjustable (CloudWatch quotas)

CloudTrail event history

90 days of management events, per Region (CloudTrail User Guide)

Managed Prometheus retention

150 days default, up to 1,095 days (Managed Prometheus User Guide)

Recent changes

Date Change

2026-02-25

The X-Ray SDKs and daemon enter maintenance mode (security fixes only); OpenTelemetry and ADOT with the CloudWatch agent or an OpenTelemetry Collector are the recommended path

2026-05-31

CloudTrail Lake stops accepting new customers; existing customers are unaffected

Current

Logs Insights offers OpenSearch PPL and SQL besides Logs Insights QL, and field indexes to cut scan cost

Current

Application Signals adds SLOs, change events, audit findings and dynamic service grouping

Current

CloudWatch Database Insights supersedes the Performance Insights console experience for RDS and Aurora

Where older guides are outdated: tutorials that install the X-Ray daemon or add the X-Ray SDK to new code; posts that rely on an alarm with a recover action for EC2 failure (simplified automatic recovery is now the default); and any guidance that recommends CloudTrail Lake for a new account.

Troubleshooting

Symptom Likely cause and check

No logs from an ECS task

The task execution role lacks logs:CreateLogStream and logs:PutLogEvents, or the log group does not exist and awslogs-create-group is unset

No logs from Lambda

The execution role has no logging permissions, or a function in a VPC cannot reach the endpoint (not applicable to the built-in log delivery, which is service-side)

Alarm stuck in INSUFFICIENT_DATA

The metric is not published (wrong dimensions or Region), or the period is shorter than the publishing interval

Alarm never notifies

The SNS subscription is unconfirmed, the topic policy does not allow CloudWatch, or the topic is encrypted with a KMS key CloudWatch cannot use

Logs Insights returns nothing

Time range or log-group selection is wrong, or the events pre-date the log group

Application Signals shows no service

The agent or sidecar is not receiving OTLP data, the task role lacks permissions, or discovery was never started

No traces after removing the X-Ray daemon

The application still exports to the daemon port; point the OTLP exporter at the CloudWatch agent or collector

CloudTrail trail shows no events

Logging is stopped, the bucket policy denies the service principal, or the events are data events that were never enabled

aps remote write returns 403

The writer’s role lacks aps:RemoteWrite, or requests are not SigV4-signed

Clean up

Delete in dependency order; log groups and trail buckets keep billing until you remove them.

aws cloudwatch delete-alarms --alarm-names catalog-api-down catalog-api-5xx-high catalog-api-no-healthy-hosts
aws logs delete-metric-filter --log-group-name /bookshelf/catalog-api --filter-name ErrorCount
aws logs delete-log-group --log-group-name /bookshelf/catalog-api

aws events remove-targets --rule root-sign-in --ids oncall
aws events delete-rule --name root-sign-in
aws sns delete-topic --topic-arn "arn:aws:sns:us-east-1:111122223333:bookshelf-oncall"

aws cloudtrail stop-logging --name bookshelf-trail
aws cloudtrail delete-trail --name bookshelf-trail
aws s3 rm "s3://bookshelf-trail-$ACCOUNT_ID" --recursive
aws s3api delete-bucket --bucket "bookshelf-trail-$ACCOUNT_ID"

aws amp delete-workspace --workspace-id "$WORKSPACE_ID"
aws grafana delete-workspace --workspace-id "$GRAFANA_ID"

Delete composite alarms before the alarms they reference. If the trail bucket is versioned or has Object Lock, empty all versions first (see S3 Object Storage). Custom metrics cannot be deleted; they expire about 15 months after the last data point. Detach and delete the IAM policies attached to bookshelf-catalog-api-task-role only if the role is yours to remove; aws application-signals discovery uses a service-linked role that is safe to leave in place.

Related pages

References