Monitoring and Observability

Plain text lines become one entry each, with severity DEFAULT. A single-line JSON object is parsed into a structured payload, and a few well-known fields are promoted: severity, message, logging.googleapis.com/trace (links the log to a Cloud Trace span) and logging.googleapis.com/spanId. Since Spring Boot 3.4 the built-in structured logging can emit exactly that shape:

Derived from Spring Boot structured logging and Structured logging.

Logger configuration, MDC fields and per-environment log levels are covered in Logging: Logback Defaults, Structured Output, and Container-Friendly Routing; container stdout conventions are in Resources, Logging and Monitoring. For custom code that has to write entries directly, the Java client library (google-cloud-logging) or the Spring Cloud GCP spring-cloud-gcp-starter-logging starter exist, but stdout is simpler and cheaper to operate.

Every project has two buckets that you cannot delete: _Required (audit logs such as Admin Activity, retained 400 days, free, not editable) and _Default (everything else, retained 30 days by default). Two sinks, also named _Required and _Default, route entries into them. You can add user-defined buckets with their own region, retention (1 day to 3650 days), CMEK key and access control, and you can lock a bucket’s retention so that it cannot be shortened.

Derived from Routing and storage overview and Configure log buckets.

Derived from Using the gcloud CLI with Logging, gcloud logging read and gcloud logging buckets create.

Querying uses the Logging query language: field=value, comparison operators (>= works on severity), AND/OR/NOT, : for "contains" and =~ for regular expressions. Always include a timestamp bound or --freshness so that the query scans a small slice, because both speed and the quota on reads depend on it.

A sink needs a writer identity, a service account that Logging creates for it, and that identity needs write permission on the destination. The following archives the audit logs of the project into the bookshelf-audit bucket and exports ERROR entries to BigQuery for analysis.

Derived from Configure and manage sinks, Route logs to BigQuery and gcloud logging sinks create.

Sinks to a log bucket in the same project need no extra grant. For Cloud Storage use roles/storage.objectCreator, and for Pub/Sub use roles/pubsub.publisher, on the destination. For a whole organisation or folder, create an aggregated sink with --organization=ORG_ID or --folder=FOLDER_ID and --include-children, which is the standard way to centralise audit logs in a security project. Sinks only act on entries received after they are created, so nothing can be back-filled; plan the routing before the data you need arrives.

Exclusions reduce cost by dropping noisy entries before storage:

Derived from Exclusion filters.

Upgrading a log bucket to Log Analytics (--enable-analytics) lets you query it with SQL in the console, using BigQuery’s query engine, without exporting anything. A linked BigQuery dataset additionally exposes the bucket to BigQuery itself, so existing dashboards and joins with business data keep working. Depending on the console release the page is labelled Log Analytics or Observability Analytics; the feature is the same.

Derived from Log Analytics and Create a linked BigQuery dataset.

A query counts which principals changed IAM policies this week (open Logging → Log Analytics, choose the bucket’s _AllLogs view):

Derived from Sample SQL queries for Log Analytics.

A log-based metric turns log entries into a time series that Cloud Monitoring can chart and alert on. A counter metric counts matching entries, a distribution metric extracts a numeric field (for example a latency) into buckets. System-defined log-based metrics exist for platform events; user-defined ones are billed by the data they ingest as for any custom metric.

Derived from Log-based metrics and gcloud logging metrics create.

The metric appears in Monitoring as logging.googleapis.com/user/orders_failed. Metrics only count entries that arrive after they are created. Prefer an application metric (see Micrometer below) when you control the code; use log-based metrics for third-party components and for platform logs you cannot change.

Cloud Audit Logs answer "who did what, where and when" and are written for you:

Enable Data Access logs selectively (for Secret Manager and Cloud KMS they are worth the cost) in IAM & Admin → Audit Logs, or by editing the IAM policy’s auditConfigs. The full security-side treatment, including organisation-wide sinks and the policy that forbids disabling them, is in Governance, Security and Compliance.

Derived from Cloud Audit Logs overview and Configure Data Access audit logs.

An alerting policy tells a channel who to contact. Channels include email, SMS, Slack, PagerDuty, webhooks, Pub/Sub and the mobile app. Create the channel first and keep its resource name.

An alerting policy has one or more conditions (a metric threshold, metric absence, a PromQL query, or a matched log entry), optional documentation shown in the notification, and the notification channels. A condition has a duration, so a single bad sample does not page anyone. When the condition holds, Monitoring opens an incident, notifies the channels, and closes the incident automatically once the condition clears.

Choose alerts for symptoms (error rate, latency, saturation, queue backlog), not for causes, and make each one actionable by linking a runbook in the documentation field.

An uptime check probes an HTTP, HTTPS or TCP endpoint from several regions (public checks) or from inside a VPC (private checks, through a service directory endpoint) and records pass or fail as the metric monitoring.googleapis.com/uptime_check/check_passed. Alert on it so you hear about an outage from Monitoring before you hear about it from a customer.

Service monitoring models your application as services and lets you set a service level objective (SLO) on a service level indicator (SLI) such as request-based availability ("97% of requests are not 5xx") or latency ("99% of requests are faster than 500 ms") over a rolling or calendar compliance period. Cloud Run, GKE (Istio and Cloud Service Mesh), App Engine and Cloud Endpoints services are detected automatically; others are defined as custom services. The error budget is the allowed failure that remains, and a burn-rate alert fires when the budget is being consumed faster than the period allows, which pages for fast burns and creates a ticket for slow burns.

There is no gcloud monitoring slos command group; automate SLOs with the Service Monitoring API or Terraform (google_monitoring_slo):

Derived from Service monitoring and SLOs, Service Monitoring API: serviceLevelObjectives and SLO API structures.

Spring Boot Actuator and Micrometer collect JVM, HTTP and datasource metrics; the Prometheus registry exposes them for scraping, and the OpenTelemetry registry can push them with OTLP. The general model, naming, custom meters and dashboard ideas are in Metrics and Observability: Micrometer, Actuator, and Distributed Tracing. On Google Cloud you have two good routes: scrape the Prometheus endpoint with Managed Service for Prometheus (next section), or push OTLP metrics to the Telemetry API (see OpenTelemetry below). Avoid the old Stackdriver Micrometer registry for new work.

Derived from Spring Boot Actuator metrics.

Enable managed collection on an existing cluster and declare the scrape target:

Derived from Enable managed collection on GKE.

Derived from Set up managed collection: PodMonitoring.

Derived from Set up managed collection.

The Pod’s service account needs roles/monitoring.metricWriter, which on GKE with Workload Identity Federation is granted to the Kubernetes service account’s principal (see Deploying Microservices on GKE). Metrics appear in Monitoring → Metrics Explorer in PromQL mode within a minute or two.

Cloud Monitoring understands PromQL natively in Metrics Explorer, dashboards and alerting policies (conditionPrometheusQueryLanguage). External tools such as Grafana connect to the Prometheus-compatible query API with an access token or a service account:

Derived from Query using Cloud Monitoring and PromQL and Query using the Prometheus HTTP API.

Rule evaluation can be managed (Rules custom resource) or done in Cloud Monitoring alerting policies; recording rules, alert rules and Alertmanager routing are in Recording & alerting rules and Alertmanager. Kubernetes-specific monitoring patterns (kube-state-metrics, cAdvisor) are in Containers & Kubernetes monitoring. For Grafana on top of GMP see Grafana: Dashboards & visualization; Google Cloud Monitoring is also available as a Grafana data source.

Trace has no day-to-day CLI; the API is the programmable surface:

Derived from Cloud Trace overview and Cloud Trace API: traces.list.

Latency sampling is the cost lever: trace a fraction of requests (for example 10%) with Micrometer’s management.tracing.sampling.probability rather than all of them.

Spring Boot’s tracing support bridges Micrometer to OpenTelemetry. Add the bridge and the OTLP exporter, point it at the Telemetry API, and let Application Default Credentials provide the token:

Derived from Spring Boot tracing and opentelemetry-operations-java.

The exporter-auto module (from the Google Cloud OpenTelemetry operations library) supplies the credential handling; Google’s Telemetry API page documents the exact authenticator settings for the OTLP exporter in use, so follow it for the version you adopt. For a quick try without code changes, the OpenTelemetry Java agent attaches at start-up:

Derived from OpenTelemetry Java agent: getting started.

In the example the agent talks to a local Collector on port 4317, which is the safer pattern when several services share one exporter configuration. A minimal Collector configuration:

Derived from OpenTelemetry Collector with Google Cloud and Google-built OpenTelemetry Collector.

The workload’s service account needs roles/cloudtrace.agent for traces and roles/monitoring.metricWriter for metrics. Keep the log correlation intact: with Micrometer Tracing the trace and span ids are placed in the MDC, and the gcp structured log format writes them as logging.googleapis.com/trace so that Logs Explorer shows View trace details on each entry.

Derived from Formatting error messages and gcloud beta error-reporting events report.

Logging an exception with its stack trace (log.error("lookup failed", ex)) in the gcp structured format is enough for Spring Boot services; no extra library is required. Error Reporting alerts are free of charge and send mail rather than pages; for paging, create a log-based or metric alert on the same log filter. The error counts depend on the logs being stored, so an exclusion that drops ERROR entries also hides errors.