Deploying Microservices on GKE

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

See the section bibliography for the reference material consulted while preparing these pages.

Examples on this page target the gcloud CLI 586.x line, kubectl within one minor version of the cluster, Helm 4, Skaffold 2.x, the Jib Maven plugin 3.x, the Kubernetes Gateway API standard channel (v1 resources) as implemented by GKE, and a GKE cluster on the Regular release channel at Kubernetes 1.35 (all checked 2026-09-30). GKE ships new versions, GatewayClasses and policy fields frequently, so confirm the exact versions with gcloud container get-server-config and the GKE release notes before relying on a flag or field.

This page is scoped to workload deployment: the Bookshelf services (catalog-api, orders-api, notifier) running as Kubernetes resources on an already existing cluster. Creating bookshelf-gke itself — Autopilot versus Standard, zonal versus regional, release channels, node pools, private clusters, upgrades and cluster pricing — is on GKE Clusters. Generic Kubernetes concepts (Pods, Deployments, Services, probes, ConfigMaps, Secrets, Helm and Kustomize basics, the Gateway API model) live in the Kubernetes section and are not repeated here; only what is specific to running them on Google Cloud is. Useful companions are ReplicaSets and Deployments, Services, Gateway API, Helm, Kustomize, Autoscaling and Health Probes and Container Lifecycle. The images come from the Artifact Registry repository described in Container Registry and Builds. For the broader question of whether a workload belongs on GKE at all, see Choosing a Compute Service and the decision table near the end of this page.

Enable the APIs and point kubectl at the cluster first:

export PROJECT_ID=$(gcloud config get-value project)
export PROJECT_NUMBER=$(gcloud projects describe "$PROJECT_ID" --format='value(projectNumber)')
export REGION=us-central1
export CLUSTER=bookshelf-gke
export REGISTRY="$REGION-docker.pkg.dev/$PROJECT_ID/bookshelf"

gcloud services enable container.googleapis.com artifactregistry.googleapis.com \
  cloudbuild.googleapis.com certificatemanager.googleapis.com secretmanager.googleapis.com

gcloud container clusters get-credentials "$CLUSTER" --location="$REGION"
kubectl get nodes

Derived from Install kubectl and configure cluster access and gcloud container clusters get-credentials.

The project, project number and region above are placeholders; substitute your own. kubectl get nodes on an Autopilot cluster may list few or no nodes until the first Pod is scheduled, because Autopilot provisions capacity on demand.

Request path overview

A request to Bookshelf crosses Google’s edge, a Cloud Armor policy, one global external Application Load Balancer that GKE programs from Gateway API resources, and then goes straight to Pods through network endpoint groups (container-native load balancing), so there is no extra hop through node ports.

flowchart LR Client["Client"] DNS["Cloud DNS record"] Armor["Cloud Armor policy\nbookshelf-policy"] LB["Global external Application\nLoad Balancer\n(Gateway + HTTPRoute)"] Cat["catalog-api\nPods"] Ord["orders-api\nPods"] Not["notifier\nPods"] SQL[("Cloud SQL\nbookshelf-catalog")] FS[("Firestore\norders")] PS["Pub/Sub\ntopic orders"] Client --> DNS --> Armor --> LB LB -->|"/api/catalog"| Cat LB -->|"/api/orders"| Ord Cat --> SQL Ord --> FS Ord --> PS PS --> Not

The Pods reach Cloud SQL, Firestore and Pub/Sub with Workload Identity Federation for GKE (see Pod credentials: Workload Identity Federation for GKE), never with downloaded keys. Load balancer concepts, Cloud Armor and certificates are on Load Balancing and Cloud Armor.

Building and pushing the images

Three routes get the Bookshelf images into the bookshelf Artifact Registry repository. Which one to use depends on whether the project already has a Dockerfile and whether you want a container daemon on the build machine.

Jib for the Spring Boot services

Jib builds an optimized, layered image for a Java project straight from Maven or Gradle, with no Dockerfile and no local Docker daemon. This is the shortest path for catalog-api and orders-api:

<plugin>
  <groupId>com.google.cloud.tools</groupId>
  <artifactId>jib-maven-plugin</artifactId>
  <version>3.4.5</version><!-- check the latest 3.x -->
  <configuration>
    <from>
      <image>eclipse-temurin:21-jre</image>
    </from>
    <to>
      <image>us-central1-docker.pkg.dev/my-project-id/bookshelf/catalog-api</image>
      <tags>
        <tag>${image.tag}</tag>
      </tags>
    </to>
    <container>
      <ports>
        <port>8080</port>
      </ports>
    </container>
  </configuration>
</plugin>
gcloud auth configure-docker "$REGION-docker.pkg.dev"
./mvnw compile jib:build -Dimage.tag=1.0.0

Derived from the Jib Maven plugin documentation and Configure authentication to Artifact Registry for Docker. Jib pushes with the credentials Docker is configured with, which here is the credential helper using your own gcloud login; in CI use Workload Identity Federation instead (see CI/CD: GitHub Actions to Artifact Registry to Helm). The plugin version above is a placeholder: check the latest 3.x release.

Cloud Build or a Dockerfile

When a Dockerfile exists (the notifier or a polyglot service), let Cloud Build build and push it without a local daemon:

gcloud builds submit --tag "$REGISTRY/notifier:1.0.0" .

Derived from Build and push a Docker image with Cloud Build. Triggers, build configuration files and private pools are on Container Registry and Builds. The older gcr.io Container Registry host was shut down on 2025-03-18; guides that still push to gcr.io/PROJECT_ID/…​ describe a retired endpoint, and Artifact Registry (REGION-docker.pkg.dev) is the current registry.

Allowing the cluster to pull

Nodes pull images with the identity of the node service account. On Autopilot and on Standard clusters created with a dedicated node service account, grant that account read access to the repository rather than relying on the broad default Compute Engine account:

export NODE_SA="gke-nodes@$PROJECT_ID.iam.gserviceaccount.com"

gcloud artifacts repositories add-iam-policy-binding bookshelf \
  --location="$REGION" \
  --member="serviceAccount:$NODE_SA" \
  --role="roles/artifactregistry.reader"

Derived from Configure access control in Artifact Registry and gcloud artifacts repositories add-iam-policy-binding. The node service account is chosen when the cluster is created; see GKE Clusters.

The Bookshelf services as Deployments and Services

Each service is a Deployment (stateless, rolling updates) plus a ClusterIP Service. The manifest below is the catalog-api; orders-api differs only in names, image and port. Everything GKE-specific is limited to the image host, the resource requests and the ServiceAccount.

apiVersion: v1
kind: Namespace
metadata:
  name: bookshelf
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: catalog-api
  namespace: bookshelf
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: catalog-api
  namespace: bookshelf
  labels:
    app: catalog-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: catalog-api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: catalog-api
    spec:
      serviceAccountName: catalog-api
      containers:
        - name: catalog-api
          image: us-central1-docker.pkg.dev/my-project-id/bookshelf/catalog-api:1.0.0
          ports:
            - containerPort: 8080
          envFrom:
            - configMapRef:
                name: catalog-api-config
          resources:
            requests:
              cpu: 500m
              memory: 768Mi
            limits:
              memory: 768Mi
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            initialDelaySeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  name: catalog-api
  namespace: bookshelf
spec:
  selector:
    app: catalog-api
  ports:
    - port: 80
      targetPort: 8080

Reference: Deploying a stateless application and GKE Services. The semantics of every field are in ReplicaSets and Deployments and Services. The my-project-id project is a placeholder.

Autopilot specifics worth knowing before you write the numbers:

  • Resource requests are what you are billed for. Autopilot bills for the CPU, memory and ephemeral storage that Pods request, and it enforces minimum and maximum requests per compute class; requests you omit are defaulted, and requests outside the allowed ratios are adjusted. Set explicit requests on every container.

  • Requests equal limits for memory is the safe pattern. Autopilot sets limits equal to requests for burstable workloads in some compute classes, so design as if there is no memory headroom.

  • Spot and compute classes are chosen with node selectors or Pod annotations (for example Spot for tolerant workers); the supported classes and their request ranges are on the Autopilot resource requests page.

  • Security defaults are stricter: privileged containers, host paths and some capabilities are rejected. The generic background is on Pod Security.

Deploy with plain kubectl:

kubectl apply -f k8s/
kubectl -n bookshelf rollout status deploy/catalog-api
kubectl -n bookshelf get pods,svc

Derived from the kubectl reference; command patterns are in kubectl Essentials. The Autopilot request rules are in Resource requests in Autopilot.

Kustomize overlays per stage

Kustomize (built into kubectl) layers a stage-specific patch over the base manifests, which fits "one project per app per stage" from Resource Hierarchy and Projects: the base is identical and each stage’s overlay sets the image tag, replica count and registry project.

# k8s/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
images:
  - name: us-central1-docker.pkg.dev/my-project-id/bookshelf/catalog-api
    newTag: 1.0.0
replicas:
  - name: catalog-api
    count: 5
kubectl kustomize k8s/overlays/prod   # render
kubectl apply -k k8s/overlays/prod    # apply

Reference: Kustomize reference and Kustomize. Config Sync (below) can apply the same overlay from Git.

Packaging Bookshelf as a Helm chart

Helm bundles the Deployment, Service, ServiceAccount, HTTPRoute and policies into one versioned release, with values per stage. A minimal layout has Chart.yaml, values.yaml and one template per resource.

# values.yaml
image:
  repository: us-central1-docker.pkg.dev/my-project-id/bookshelf/catalog-api
  tag: "1.0.0"
replicaCount: 3
resources:
  requests:
    cpu: 500m
    memory: 768Mi
gateway:
  host: bookshelf.example.com
# templates/deployment.yaml (excerpt)
spec:
  replicas: {{ .Values.replicaCount }}
  template:
    spec:
      containers:
        - name: catalog-api
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
helm lint ./charts/catalog-api
helm upgrade --install catalog-api ./charts/catalog-api \
  --namespace bookshelf --create-namespace \
  --set image.tag=1.0.1 \
  --atomic --timeout 5m
helm history catalog-api -n bookshelf
helm rollback catalog-api 1 -n bookshelf

Derived from the helm upgrade and chart template guide pages, and Deploy an application with Helm on GKE style workflows. --atomic rolls the release back automatically if the Pods do not become ready, which is the property to want in CI. Chart structure and values design are in Helm. Helm charts can also be stored as OCI artifacts in Artifact Registry (helm push to a repository of format Docker), which keeps charts and images in one place with one IAM model.

Inner loop: Skaffold and Cloud Code

For day-to-day development the build-push-deploy cycle should be one command. Skaffold watches the source, rebuilds with Jib, pushes to a registry the cluster can reach and re-applies the manifests; Cloud Code puts the same loop into IntelliJ and VS Code, with debugger attach, log streaming and a Kubernetes explorer.

# skaffold.yaml
apiVersion: skaffold/v4beta11
kind: Config
metadata:
  name: catalog-api
build:
  artifacts:
    - image: us-central1-docker.pkg.dev/my-project-id/bookshelf/catalog-api
      jib:
        project: catalog-api
  tagPolicy:
    gitCommit: {}
manifests:
  kustomize:
    paths:
      - k8s/overlays/dev
deploy:
  kubectl: {}
portForward:
  - resourceType: service
    resourceName: catalog-api
    namespace: bookshelf
    port: 80
    localPort: 8080
skaffold dev --default-repo="$REGISTRY"   # build, deploy, tail logs, redeploy on change
skaffold run --default-repo="$REGISTRY"   # one-off build and deploy, as CI would
skaffold delete                           # remove what skaffold deployed

Derived from the Skaffold documentation and Cloud Code for IntelliJ: Kubernetes getting started; pick the apiVersion that matches skaffold version. Use a development cluster or a namespace per developer, never the production cluster. Cloud Code also bundles Gemini Code Assist in its current releases. Cloud Deploy is Google’s managed delivery service built on Skaffold (promotion across dev, staging and production targets with approvals and rollback); it is a good next step once manual skaffold run stops being enough, and it accepts the same skaffold.yaml. Its overview is in the Cloud Deploy documentation.

Exposing services with the Gateway API

GKE implements the Kubernetes Gateway API for its Cloud Load Balancing products: you create a Gateway that selects a GatewayClass, one or more HTTPRoute objects that attach to it, and optional Google policy resources for what the standard API does not cover. GKE installs the GatewayClasses and CRDs automatically on Autopilot clusters; on Standard clusters turn it on with --gateway-api=standard. The general model is on Gateway API.

GatewayClass Load balancer Use for

gke-l7-global-external-managed

Global external Application Load Balancer

Internet-facing Bookshelf, multi-region backends, Cloud CDN. Recommended over the legacy class below.

gke-l7-regional-external-managed

Regional external Application Load Balancer

Internet-facing traffic that must stay in one region (data residency, Standard network tier).

gke-l7-rilb

Regional internal Application Load Balancer

Internal service-to-service or on-premises-reachable APIs over private IPs (for example the orders-api admin path).

gke-l7-gxlb

Classic global external Application Load Balancer

Legacy; existing setups only. Prefer gke-l7-global-external-managed.

Multi-cluster variants (suffix -mc) exist for fleets and are covered below. List what your cluster offers with kubectl get gatewayclass. Reference: About Gateway API and Deploying Gateways.

Prerequisites for the managed proxies

The regional classes (external and internal) need a proxy-only subnet in the Gateway’s region and VPC network; the global class does not. Create it once per region and network:

gcloud compute networks subnets create proxy-only-us-central1 \
  --purpose=REGIONAL_MANAGED_PROXY --role=ACTIVE \
  --region="$REGION" --network=bookshelf-vpc --range=10.129.0.0/23

gcloud container clusters update "$CLUSTER" --location="$REGION" --gateway-api=standard  # Standard clusters only

Derived from Proxy-only subnets for Envoy-based load balancers and Enable Gateway API in a cluster. Clusters must be VPC-native and keep the HTTP load balancing add-on enabled. The VPC and subnet design is on VPC Networking on Google Cloud.

A global external Gateway with HTTPS

Certificates for a GKE Gateway come from Certificate Manager (referenced through a certificate map annotation, the recommended route, and the one that scales to many hostnames) or from a Kubernetes Secret. The reserved global IP address and the certificate are created once, outside Kubernetes:

gcloud services enable certificatemanager.googleapis.com

gcloud compute addresses create bookshelf-ip --global

gcloud certificate-manager dns-authorizations create bookshelf-auth --domain=bookshelf.example.com
# add the CNAME record that the command prints to your DNS zone, then:
gcloud certificate-manager certificates create bookshelf-cert \
  --domains=bookshelf.example.com --dns-authorizations=bookshelf-auth
gcloud certificate-manager maps create bookshelf-map
gcloud certificate-manager maps entries create bookshelf-entry \
  --map=bookshelf-map --hostname=bookshelf.example.com --certificates=bookshelf-cert

Derived from Deploy a Google-managed certificate with DNS authorization and Certificate maps. DNS authorization lets Google issue the certificate before the Gateway exists, so there is no downtime while it provisions. Certificates and DNS are on Cloud DNS and Certificate Manager.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: bookshelf-gateway
  namespace: bookshelf
  annotations:
    networking.gke.io/certmap: bookshelf-map
spec:
  gatewayClassName: gke-l7-global-external-managed
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      allowedRoutes:
        kinds:
          - kind: HTTPRoute
  addresses:
    - type: NamedAddress
      value: bookshelf-ip
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: bookshelf-routes
  namespace: bookshelf
spec:
  parentRefs:
    - name: bookshelf-gateway
  hostnames:
    - bookshelf.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api/catalog
      backendRefs:
        - name: catalog-api
          port: 80
    - matches:
        - path:
            type: PathPrefix
            value: /api/orders
      backendRefs:
        - name: orders-api
          port: 80
kubectl apply -f gateway.yaml
kubectl -n bookshelf describe gateway bookshelf-gateway   # wait for Programmed=True
kubectl -n bookshelf get gateway bookshelf-gateway -o jsonpath='{.status.addresses[0].value}'

Derived from Deploying Gateways and Secure a Gateway. Provisioning the load balancer takes several minutes. gateway.networking.k8s.io/v1 is the stable API for Gateway, GatewayClass and HTTPRoute; guides that use v1beta1 still work but predate it.

Console view

In the Google Cloud console

  1. Open Kubernetes Engine → Gateways, Services & Ingress and the Gateways tab.

  2. Open bookshelf-gateway to see its listeners, attached routes, load balancer address and the generated URL map.

  3. Open Network services → Load balancing and select the load balancer GKE created to see the backend services and network endpoint groups, health and logs. Resources created by GKE should be changed through Kubernetes, not here: controller reconciliation reverts manual edits.

  4. Open Security → Certificate Manager to see certificate status; a certificate stays in a provisioning state until DNS authorization succeeds.

With kubectl

kubectl get gateway,httproute -n bookshelf
kubectl describe httproute bookshelf-routes -n bookshelf

Reference: GatewayClass capabilities. Status conditions on the Gateway and routes (Accepted, Programmed, ResolvedRefs) are the first place to look when traffic does not flow.

Google policies: backends, health checks and the gateway

The standard API has no field for IAP, Cloud Armor, backend timeouts or custom health checks, so GKE adds three policy resources that attach to a target with targetRef:

apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
  name: catalog-api-backend
  namespace: bookshelf
spec:
  default:
    securityPolicy: bookshelf-policy      # Cloud Armor policy
    timeoutSec: 30
    logging:
      enabled: true
      sampleRate: 500000                  # 50 percent, in millionths
  targetRef:
    group: ""
    kind: Service
    name: catalog-api
---
apiVersion: networking.gke.io/v1
kind: HealthCheckPolicy
metadata:
  name: catalog-api-health
  namespace: bookshelf
spec:
  default:
    checkIntervalSec: 15
    timeoutSec: 10
    healthyThreshold: 2
    unhealthyThreshold: 2
    config:
      type: HTTP
      httpHealthCheck:
        portSpecification: USE_SERVING_PORT
        requestPath: /actuator/health/readiness
  targetRef:
    group: ""
    kind: Service
    name: catalog-api

Derived from Configure Gateway resources using Policies. Only one GCPBackendPolicy can attach to a target Service at a time, and a HealthCheckPolicy must live in the same namespace as its Service. Without a HealthCheckPolicy, GKE derives a health check from the Pod’s readiness probe when it can, and otherwise defaults to an HTTP check on /, which is a common cause of backends marked unhealthy. GCPGatewayPolicy sets Gateway-level options such as the SSL policy (minimum TLS version and cipher suites) and global access for internal Gateways. The Cloud Armor policy itself, including rate limiting and managed rule sets, is created as described on Load Balancing and Cloud Armor.

Canary releases with weighted backends

A weighted backendRefs list splits traffic between two Services. Run the new version as catalog-api-canary and move the weights in steps:

  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api/catalog
      backendRefs:
        - name: catalog-api
          port: 80
          weight: 90
        - name: catalog-api-canary
          port: 80
          weight: 10

Reference: Configure traffic management for Gateways. The same mechanism does header-based routing and mirroring. Rollout strategies in general (blue-green, canary, Argo Rollouts) are in Deployment Strategies; for Cloud Run, the equivalent is the revision traffic split on Cloud Run Scaling, Traffic and Jobs.

Custom request headers with client location

The load balancer can add headers that describe the client, which a service can use to serve region-specific content. GKE exposes this through the backend policy’s custom request headers (older guides configure it with a BackendConfig on an Ingress). A header template such as X-Client-Region:{client_region} is expanded by the load balancer using variables like {client_region}, {client_city} and {client_rtt_msec}. Check Custom request and response headers for the variable list and the current backend policy field, since this is one of the options that has moved between APIs.

Internal Gateways and multi-cluster Gateways

  • Internal. Use gke-l7-rilb for private APIs; it needs the proxy-only subnet, takes a regional address, and is reachable only from the VPC, connected networks (VPN, Interconnect) or, with GCPGatewayPolicy global access, other regions of the VPC.

  • Multi-cluster. Register clusters in a fleet and use the multi-cluster GatewayClasses (for example gke-l7-global-external-managed-mc) with ServiceExport and ServiceImport resources to route to the nearest healthy cluster and fail over between regions. This is the successor to Multi Cluster Ingress, the "GKE Hub" feature that older guides use; fleets themselves are covered on GKE Clusters. Reference: Deploying multi-cluster Gateways.

GKE Ingress, and the ingress-nginx retirement

GKE Ingress (the gce ingress class, configured with BackendConfig and FrontendConfig) is still supported and creates the same kind of Google load balancer, but new features land on the Gateway API. Google documents a migration path from Ingress to Gateway, and the upstream ingress2gateway tool converts many manifests. The community ingress-nginx controller was retired by the Kubernetes project in March 2026 (no further releases or security fixes); it was never GKE’s default, but clusters that installed it themselves should plan a move to the Gateway API, to GKE Ingress or to another maintained controller. Reference: Migrate from Ingress to Gateway and the Kubernetes blog post Ingress NGINX Retirement: What You Need to Know. Generic Ingress material is in Ingress.

Identity-Aware Proxy for GKE applications

Identity-Aware Proxy (IAP) puts a Google sign-in and an IAM check in front of an application that sits behind an Application Load Balancer, so an internal tool such as a Bookshelf admin console needs no login code of its own. On a Gateway, IAP is enabled on the GCPBackendPolicy for the Service behind the route. Older guides (the Ingress era) use a BackendConfig with an iap block instead; the current approach is the policy.

In the Google Cloud console

  1. Open Security → Identity-Aware Proxy. If prompted, configure the OAuth consent screen (choose Internal to restrict to your organization).

  2. Create an OAuth client of type Web application under APIs & Services → Credentials; add the redirect URI that IAP documents (https://iap.googleapis.com/v1/oauth/clientIds/CLIENT_ID:handleRedirect).

  3. Create a Kubernetes Secret holding the client secret, apply the GCPBackendPolicy below, and wait for the load balancer to update.

  4. Back in Identity-Aware Proxy, select the backend service and Add principal with the role IAP-secured Web App User.

With the gcloud CLI and kubectl

kubectl -n bookshelf create secret generic iap-oauth \
  --from-literal=key="$OAUTH_CLIENT_SECRET"   # value read from your secret store, never committed

cat <<'YAML' | kubectl apply -f -
apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
  name: admin-iap
  namespace: bookshelf
spec:
  default:
    iap:
      enabled: true
      clientID: my-oauth-client-id.apps.googleusercontent.com
      oauth2ClientSecret:
        name: iap-oauth
  targetRef:
    group: ""
    kind: Service
    name: admin-ui
YAML

gcloud iap web add-iam-policy-binding \
  --resource-type=backend-services --service=BACKEND_SERVICE_NAME \
  --member="group:bookshelf-admins@example.com" \
  --role="roles/iap.httpsResourceAccessor"

Derived from Enabling IAP for GKE and gcloud iap web add-iam-policy-binding. BACKEND_SERVICE_NAME is the generated Compute Engine backend service name shown by gcloud compute backend-services list. The client ID, group and domain are placeholders. The OAuth client secret is the one credential this flow needs; keep it in Secret Manager and create the Kubernetes Secret from it in your pipeline, or use the Secret Manager add-on below. Recent IAP configurations can use a Google-managed OAuth client instead for supported setups; check the IAP enablement page for the option that applies to your load balancer type.

Reading the identity in Spring Boot

IAP adds headers such as X-Goog-Authenticated-User-Email and X-Goog-Authenticated-User-Id, and a signed X-Goog-IAP-JWT-Assertion. The plain headers can be forged by anything that reaches the Pod directly, so trust only the signed JWT, verified against Google’s IAP keys, with the expected audience for the backend service. The verification code and the audience values are on Identity and Access; add a NetworkPolicy (Network Policies) so that only the load balancer path can reach the Pod.

Service-to-service traffic

Inside the cluster, services call each other by DNS name (orders-api.bookshelf.svc.cluster.local, or just orders-api within the namespace); see DNS and Service Discovery. GKE adds two things on top:

  • Cloud Service Mesh is Google’s managed Istio-based service mesh. It gives workloads mutual TLS, authorization policy, retries, traffic splitting and telemetry through a managed control plane, without changing application code. An ambient (sidecar-less) mode is in Preview; confirm its launch stage on the Cloud Service Mesh documentation before using it. Adopt a mesh when you need mTLS between services or fine-grained traffic policy, not by default; generic mesh background is in Cluster Networking, CNI and Service Mesh.

  • Network policy enforcement (Dataplane V2 on new clusters) restricts which Pods may talk to which, using standard NetworkPolicy objects.

Reference: Cloud Service Mesh overview and Network policies in GKE.

Autoscaling

Autoscaling on GKE has two layers: Pod autoscalers decide how many replicas (or how large) and the platform supplies the capacity. On Autopilot the platform layer is automatic; on Standard it is the cluster autoscaler or node auto-provisioning described on GKE Clusters. The generic controllers are explained in Autoscaling; this section is the GKE-specific wiring.

Horizontal Pod Autoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: catalog-api
  namespace: bookshelf
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: catalog-api
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300

Reference: Configure horizontal Pod autoscaling. CPU utilisation is measured against the Pod’s CPU request, which is another reason to set requests deliberately.

Custom and external metrics: the notifier

The notifier should scale on the Pub/Sub backlog, not CPU. Cloud Monitoring metrics reach the HPA through the Custom Metrics Stackdriver Adapter (or, on current GKE versions, the supported Google Cloud Managed Service for Prometheus adapter path), exposed as an External metric:

  metrics:
    - type: External
      external:
        metric:
          name: pubsub.googleapis.com|subscription|num_undelivered_messages
          selector:
            matchLabels:
              resource.labels.subscription_id: notifier-sub
        target:
          type: AverageValue
          averageValue: "20"

Reference: Optimize Pod autoscaling based on metrics. The adapter needs permission to read Cloud Monitoring (roles/monitoring.viewer), granted to its Kubernetes ServiceAccount with Workload Identity Federation for GKE. The Pub/Sub side is on Messaging and Integration. KEDA is an alternative that scales to zero and has a Pub/Sub scaler; it is a community add-on you install and operate yourself.

Vertical Pod Autoscaler and multidimensional scaling

The Vertical Pod Autoscaler (VPA) recommends or applies CPU and memory requests from observed usage. Use it in Off (recommendation-only) mode to right-size the requests that Autopilot bills for, and avoid running VPA and an HPA on the same CPU or memory metric. Multidimensional Pod autoscaling combines horizontal scaling on CPU with vertical scaling on memory in a single object, for workloads where both matter. Enable VPA on the cluster, then create the object:

gcloud container clusters update "$CLUSTER" --location="$REGION" --enable-vertical-pod-autoscaling
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: orders-api
  namespace: bookshelf
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-api
  updatePolicy:
    updateMode: "Off"

Derived from Configure vertical Pod autoscaling and Scale container resource requests and limits. Read recommendations with kubectl describe vpa orders-api -n bookshelf.

Pod credentials: Workload Identity Federation for GKE

Pods call Google Cloud APIs (Cloud SQL, Secret Manager, Cloud Storage, Pub/Sub) as a Google identity. The old pattern was to create a Google service account, download a JSON key and mount it as a Secret; that is a long-lived credential that leaks, and new organizations block key creation by default (see Identity and Access). The current approach is Workload Identity Federation for GKE (formerly "Workload Identity"). It is always on in Autopilot and enabled per cluster or node pool on Standard. A Pod’s Kubernetes ServiceAccount becomes an IAM principal, and Google Cloud client libraries pick the credentials up automatically through Application Default Credentials, with no annotation or code change.

sequenceDiagram participant Pod as Pod (catalog-api) participant Meta as GKE metadata server participant STS as Security Token Service participant API as Google Cloud API Pod->>Meta: ADC requests a token Meta->>STS: Exchange the Kubernetes ServiceAccount token STS-->>Meta: Federated access token Meta-->>Pod: Access token for the KSA principal Pod->>API: Call with token API->>API: IAM check on principal://...ns/bookshelf/sa/catalog-api API-->>Pod: Response

Granting roles directly to the Kubernetes ServiceAccount

The recommended way grants IAM roles straight to the principal that represents the ServiceAccount; no Google service account and no annotation are needed:

export NAMESPACE=bookshelf
export KSA=catalog-api
export PRINCIPAL="principal://iam.googleapis.com/projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$PROJECT_ID.svc.id.goog/subject/ns/$NAMESPACE/sa/$KSA"

gcloud projects add-iam-policy-binding "$PROJECT_ID" \
  --role=roles/cloudsql.client --member="$PRINCIPAL" --condition=None

gcloud secrets add-iam-policy-binding catalog-db-password \
  --role=roles/secretmanager.secretAccessor --member="$PRINCIPAL" --condition=None

Derived from Authenticate to Google Cloud APIs from GKE workloads and gcloud projects add-iam-policy-binding. The pool is PROJECT_ID.svc.id.goog; PROJECT_NUMBER (not the ID) is part of the principal. Binding the role on the specific secret, bucket or topic rather than the whole project keeps the grant narrow. To grant a whole namespace use the principalSet form with the namespace attribute. Binding to a Google service account (the iam.gke.io/gcp-service-account annotation and roles/iam.workloadIdentityUser) still works and is needed only for the few APIs that do not yet support direct principal access or where a single identity must be shared across clusters and other platforms.

Application code

The Spring Boot service needs no credential code. With Spring Cloud GCP 8.x (Spring Boot 4; 7.4.x for Spring Boot 3) the starters use Application Default Credentials, which resolve to the Pod’s principal on GKE and to your own login (gcloud auth application-default login) or an impersonated service account on a laptop:

<dependency>
  <groupId>com.google.cloud</groupId>
  <artifactId>spring-cloud-gcp-starter-secretmanager</artifactId>
</dependency>
# application.properties
spring.config.import=sm://
catalog.db.password=${sm://catalog-db-password}

Derived from the Spring Cloud GCP reference (check the exact starter names for your version) and Application Default Credentials. General Spring configuration patterns are on Configuration and Profiles. For Cloud SQL, connect through the Cloud SQL Auth Proxy sidecar or the connector library with IAM database authentication; see Cloud SQL.

Configuration and secrets

  • ConfigMaps hold non-sensitive settings (catalog-api-config above, for example the database host and feature flags). Mount them as environment variables or files; changing a ConfigMap does not restart Pods unless the Deployment is rolled (a Helm checksum annotation is a common trick). See ConfigMaps.

  • Kubernetes Secrets are only base64-encoded; GKE encrypts them at rest, and application-layer encryption with a Cloud KMS key is an option on the cluster. Keep their values out of Git. See Secrets.

  • Secret Manager add-on. A CSI driver add-on mounts Secret Manager secrets into Pods as files, authenticated by the Pod’s Workload Identity Federation principal, so the value never sits in a Kubernetes Secret. The add-on has been generally available since 2024-08-20; automatic rotation went GA on 2025-09-01 and syncing mounted secrets into Kubernetes Secrets went GA on 2026-04-16. Enable it with gcloud container clusters update "$CLUSTER" --location="$REGION" --enable-secret-manager, then describe the secrets in a SecretProviderClass. Reference: Use the Secret Manager add-on with GKE.

  • Parameter Manager stores versioned, non-secret configuration such as bookshelf-settings outside the cluster and can reference Secret Manager values. The backing services are on Secret Manager and Cloud KMS and Runtime Configuration.

Config Sync: manifests from Git

Config Sync is a GitOps controller that continuously applies manifests from a Git repository, OCI image or Helm chart to one cluster or a whole fleet and reverts drift. Since 2025-09-02 the former GKE Enterprise features — fleets, Teams, Config Sync, Policy Controller and Connect Gateway — are part of GKE at no extra charge, and clusters no longer have a tier.

gcloud services enable anthosconfigmanagement.googleapis.com gkehub.googleapis.com
gcloud container fleet config-management enable
gcloud container fleet config-management apply \
  --membership="$CLUSTER" --location="$REGION" \
  --config=config-management.yaml
# config-management.yaml
applySpecVersion: 1
spec:
  configSync:
    enabled: true
    sourceFormat: unstructured
    syncRepo: https://github.com/example/bookshelf-config
    syncBranch: main
    policyDir: k8s/overlays/prod
    secretType: none

Derived from Install Config Sync and gcloud container fleet config-management apply; the repository URL is a placeholder and the exact fields vary by version, so follow the installation page. Argo CD and Flux are alternatives you run yourself; the general GitOps pattern is in GitOps and CI/CD.

Reliability: rollouts, probes, disruption budgets and spreading

  • Rolling updates with maxUnavailable: 0 and a readiness probe keep capacity during a deploy. Probe design and graceful shutdown (terminationGracePeriodSeconds, a preStop sleep so the load balancer drains) are in Health Probes and Container Lifecycle. With container-native load balancing, the load balancer needs a few seconds to remove a terminating Pod’s endpoint, so give shutdown that headroom.

  • PodDisruptionBudgets limit voluntary disruption during node upgrades and maintenance. An upgrade of a GKE node pool honours them for a bounded time, so a budget that forbids any disruption stalls upgrades:

    apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: catalog-api
  namespace: bookshelf
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: catalog-api

  • Spread across zones with topologySpreadConstraints on topology.kubernetes.io/zone, so a zone outage leaves capacity (regional clusters have nodes in several zones; see GKE Clusters). Details are in Scheduling and Disruptions.

  • Requests are mandatory in practice: Autopilot needs them to schedule and bill, and Standard needs them for the scheduler and HPA to behave. See Resource Management.

Reference: GKE best practices for availability and Container-native load balancing.

CI/CD: GitHub Actions to Artifact Registry to Helm

A pipeline needs two permissions: push images, and deploy to the cluster. Both come from Workload Identity Federation for the pipeline (not a key): the GitHub repository is trusted by a workload identity pool as described on Identity and Access, and the resulting service account holds roles/artifactregistry.writer on the repository and roles/container.developer on the project.

name: deploy
on:
  push:
    branches: [main]
permissions:
  contents: read
  id-token: write
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: google-github-actions/auth@v3
        with:
          workload_identity_provider: projects/123456789012/locations/global/workloadIdentityPools/github/providers/bookshelf-repo
          service_account: deployer@my-project-id.iam.gserviceaccount.com
      - uses: google-github-actions/get-gke-credentials@v3
        with:
          cluster_name: bookshelf-gke
          location: us-central1
      - name: Build and push with Jib
        run: |
          gcloud auth configure-docker us-central1-docker.pkg.dev --quiet
          ./mvnw -B compile jib:build -Dimage.tag="${GITHUB_SHA}"
      - name: Deploy
        run: |
          helm upgrade --install catalog-api ./charts/catalog-api \
            --namespace bookshelf --set image.tag="${GITHUB_SHA}" --atomic --timeout 5m

Derived from the google-github-actions/auth, get-gke-credentials and Workload Identity Federation with deployment pipelines documentation. The project number, pool, provider and service account above are placeholders, and the action major versions should be checked for the latest. Workflow syntax and caching are on Docker CI/CD with GitHub Actions. For gated promotion between environments, deploy through Cloud Deploy instead of calling Helm from the workflow: the workflow builds the image and creates a release, and Cloud Deploy applies it to each target with approvals and rollback.

Networking, security and limits at a glance

Area What to know

Cluster networking

VPC-native (alias IP) clusters are required for container-native load balancing and Gateway. Pod and Service ranges are sized at creation; see GKE Clusters.

Egress

Private nodes have no public IPs; outbound internet needs Cloud NAT, and Google APIs are reached through Private Google Access. See VPC Networking on Google Cloud.

Workload identity

Direct principal bindings per Kubernetes ServiceAccount; least privilege per service.

Pod security

Autopilot enforces a hardened baseline; Standard clusters should enforce Pod Security Admission. Binary Authorization and Policy Controller can require signed images and enforce constraints.

Quotas

Limits apply to clusters per location, nodes per cluster, Pods per node, load balancer forwarding rules, backend services and Certificate Manager certificates; check IAM & Admin → Quotas and the GKE quotas page.

Gateway limits

Each Gateway and route attachment maps to load balancer components (URL maps, backend services, NEGs) that have their own quotas; many small Gateways consume them faster than one shared Gateway with many routes.

Reference: GKE quotas and limits and Security posture.

Pricing model

Prices are not repeated here; use the GKE pricing page and the pricing calculator, and check the date on them. The cost drivers for a microservice deployment are:

  • Cluster management fee. A per-cluster hourly fee applies, with a monthly free-tier credit per billing account that covers one zonal or Autopilot cluster; regional Standard clusters and extra clusters pay the fee.

  • Compute. Autopilot bills for the CPU, memory and storage that Pods request (per second, with Spot and committed use discounts available); Standard bills for the Compute Engine nodes whether or not Pods use them. Right-size requests with VPA recommendations.

  • Load balancing. Forwarding rules, proxy instances and processed data for each load balancer; one shared Gateway for many routes costs less than one per service.

  • Networking. Cloud NAT, cross-zone and egress data transfer; topology spread across zones increases cross-zone traffic, so measure it.

  • Add-ons and observability. Cloud Logging ingestion, Managed Service for Prometheus samples, Cloud Armor policy and request charges, Certificate Manager, Artifact Registry storage, and Cloud Service Mesh where it is billed.

Scale-to-zero is not native to Kubernetes Deployments, so idle services still cost their minimum replicas; Spot Pods for tolerant workers and committed use discounts for steady baselines are the largest levers. Cost tooling is on Cost Management and Billing.

What changed recently, and what older guides still show

  • The Container Registry host gcr.io was shut down on 2025-03-18; use Artifact Registry docker.pkg.dev repositories.

  • "Workload Identity" is now Workload Identity Federation for GKE, and direct principal:// bindings replace the annotation-plus-service-account pattern for most APIs.

  • On 2025-09-02 the GKE Enterprise features (fleets, Teams, Config Sync, Policy Controller, Connect Gateway) were folded into GKE; guides that mention Anthos licensing or a cluster tier are out of date.

  • The Gateway API is the recommended way to expose new services; GKE Ingress with BackendConfig is still supported but guides that start there predate Gateway policies (GCPBackendPolicy, HealthCheckPolicy, GCPGatewayPolicy). gke-l7-global-external-managed is recommended over gke-l7-gxlb.

  • IAP for Gateway-based backends is configured on GCPBackendPolicy, not on a BackendConfig.

  • Multi Cluster Ingress ("GKE Hub") guides map to fleets plus multi-cluster Gateways.

  • The Secret Manager add-on is GA (2024-08-20), with rotation GA (2025-09-01) and sync to Kubernetes Secrets GA (2026-04-16).

  • The Kubernetes project retired ingress-nginx in March 2026.

  • Skaffold, Jib and Cloud Code remain current; Cloud Deploy, the Skaffold-based delivery service, is newer than many older Skaffold guides.

  • The container startup agent (konlet) for Compute Engine VMs is being retired (deprecated 2025-07-21, VM/MIG creation blocked since 2026-07-31, support ends 2027-07-31); for container workloads the supported destinations are Cloud Run, GKE or Batch.

  • Kubernetes versions move quickly; the beta Ingress APIs extensions/v1beta1 and networking.k8s.io/v1beta1 are long gone, so use networking.k8s.io/v1 and gateway.networking.k8s.io/v1.

Cloud Run versus GKE for microservices

Question Cloud Run GKE

Operational burden

None below the container: no nodes, no cluster upgrades.

Autopilot removes node management; you still own manifests, add-ons, versions and policies.

Scaling

Per-request concurrency autoscaling, scale to zero, per-revision limits.

HPA/VPA and node scaling; minimum replicas keep running; full control over signals.

Networking model

One URL per service; Direct VPC egress; fewer moving parts.

Full Kubernetes networking, Gateway, internal load balancing, service mesh, network policy.

Workload fit

Stateless HTTP, gRPC, jobs and workers that fit its limits.

Anything Kubernetes runs: stateful sets, DaemonSets, GPUs, custom operators, long-lived connections.

Portability and ecosystem

Knative-compatible API; Cloud Run specific features.

Standard Kubernetes tooling (Helm, Argo CD, operators) on any cloud.

Cost shape

Pay for use; idle can be zero.

Pay for requested or provisioned capacity; shared across many services.

For Bookshelf, catalog-api, orders-api and notifier are all good Cloud Run fits (see Cloud Run); choose GKE when the platform team already runs Kubernetes, when you need mesh policy, stateful or GPU workloads, or one operational model across clouds. See Choosing a Compute Service for the full decision flow.

Troubleshooting

  • ImagePullBackOff or ErrImagePull: the image path or tag is wrong, or the node service account lacks roles/artifactregistry.reader on the repository. kubectl describe pod prints the registry error.

  • Pods Pending on Autopilot: requests are outside the allowed range or a compute class is unavailable in the zone; the Pod event explains it. A scale-up takes a minute or two.

  • Gateway stuck without an address, or Programmed false: the proxy-only subnet is missing (regional classes), the named static address does not exist, or the Gateway API is not enabled on a Standard cluster. Read the events from kubectl describe gateway.

  • Backends UNHEALTHY/502 from the load balancer: the health check path returns non-200, a firewall rule blocks the Google health-check ranges (130.211.0.0/22 and 35.191.0.0/16), or the container listens on a different port. Add a HealthCheckPolicy.

  • 403 or 401 at the IAP sign-in: the user lacks IAP-secured Web App User on that backend, the OAuth client is wrong, or the redirect URI is missing.

  • Pods get 403 Permission denied calling an API: the IAM binding is missing, uses the project ID instead of the project number in the principal, or the namespace or ServiceAccount name differs from the binding. IAM changes can take a couple of minutes to propagate.

  • Certificate stays in PROVISIONING: the DNS authorization CNAME record is missing or not propagated.

  • HPA shows <unknown> for a metric: no metrics server yet, missing CPU request, or the external metrics adapter lacks permission to read Cloud Monitoring.

  • A Helm upgrade is rolled back by --atomic: Pods never became ready; check kubectl rollout status and Pod logs.

Debugging commands and a systematic flow are in Debugging and Troubleshooting, and logs and metrics are on Monitoring and Observability. Reference: Troubleshooting GKE.

Clean up

Remove what this page created, in dependency order, in a throwaway project:

helm uninstall catalog-api -n bookshelf
kubectl delete -f gateway.yaml
kubectl delete -f k8s/
kubectl delete namespace bookshelf

gcloud certificate-manager maps entries delete bookshelf-entry --map=bookshelf-map
gcloud certificate-manager maps delete bookshelf-map
gcloud certificate-manager certificates delete bookshelf-cert
gcloud certificate-manager dns-authorizations delete bookshelf-auth
gcloud compute addresses delete bookshelf-ip --global

gcloud projects remove-iam-policy-binding "$PROJECT_ID" \
  --role=roles/cloudsql.client --member="$PRINCIPAL" --condition=None

Derived from the helm uninstall reference and gcloud certificate-manager. Delete the Gateway and let GKE remove the load balancer before deleting the reserved address, otherwise the address is still in use; orphaned load balancer resources (forwarding rules, backend services, NEGs) keep billing, so confirm none remain under Network services → Load balancing. Delete the cluster itself as described on GKE Clusters, and the images with a cleanup policy or gcloud artifacts docker images delete. If the OAuth client for IAP was created for the exercise, delete it under APIs & Services → Credentials. Deleting the whole throwaway project removes everything at once (see Resource Hierarchy and Projects).

Related pages

References