Messaging and Integration

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

See the section bibliography for the reference material consulted while preparing these pages.

This page covers the Google Cloud services that let one part of an application hand work or facts to another without waiting for it: publish/subscribe messaging (Pub/Sub), event routing (Eventarc), task queues (Cloud Tasks), cron (Cloud Scheduler), orchestration (Workflows) and managed Kafka (Managed Service for Apache Kafka), plus a brief note on Application Integration. Commands are current for the gcloud CLI 586 line, the Google Cloud Client Libraries released through libraries-bom, and Spring Cloud GCP 8.x (Spring Boot 4) or 7.4.x (Spring Boot 3); product status, launch stages and names were checked against the official documentation on 2026-10-01. Re-run gcloud version and the linked pages before relying on exact numbers. Examples use the placeholder project my-project-id, project number <project-number> and region us-central1.

In the running Bookshelf scenario (see the section introduction), orders-api accepts an order and must tell several other parties: the notifier that sends the confirmation email (see Email sending), an archive consumer and, later, a fulfilment workflow. Calling each of them synchronously would make order placement as slow and as fragile as the slowest dependency. Instead orders-api publishes one message to the Pub/Sub topic orders, and each interested party consumes it at its own pace. The Cloud Run side of the notifier is in Cloud Run functions and events; this page shows where the topic and subscriptions come from.

Protocol and pattern theory is not repeated here; it lives in messaging models and patterns, delivery guarantees and messaging protocols. For the Azure and AWS counterparts see Azure and AWS messaging and integration.

Choosing between the services

The first decision is what you are sending. A command tells one consumer to do something and expects it to be done eventually; an event states that something happened and may interest zero, one or many consumers; a stream is an ordered, replayable log that many consumers read at their own offsets.

Service Model Typical use Delivery and ordering

Pub/Sub

Topic with independent subscriptions; each subscription receives every message, and its consumers share the work

Fan-out of events, buffering, streaming ingestion into BigQuery or Cloud Storage, the orders topic

At least once by default, exactly once per subscription when enabled; ordered per ordering key when enabled

Eventarc

Event router: sources (Google services, audit logs, Pub/Sub, your own providers) to destinations (Cloud Run, GKE, Workflows)

Reacting to platform events such as an upload to Cloud Storage, delivered as CloudEvents

At least once; built on Pub/Sub for most sources

Cloud Tasks

Queue of explicit HTTP tasks, each addressed to one handler

Rate-limited calls to a downstream API, deferred work, retries with backoff, deduplicated tasks

At least once, dispatch rate and concurrency controlled by the queue

Cloud Scheduler

Cron-style trigger

Nightly jobs, periodic polling, kicking off a workflow

At least once per scheduled time

Workflows

Serverless state machine written in YAML or JSON

Multi-step flows with retries, branching, waiting and callbacks

Each execution runs its steps in order; steps calling services should be idempotent

Managed Service for Apache Kafka

Apache Kafka clusters operated by Google

Existing Kafka applications and ecosystem, replayable ordered logs

Kafka semantics, ordered per partition

Application Integration

Low-code integrations with connectors

Connecting SaaS and enterprise systems without writing glue code

Depends on the trigger and connector

A rule of thumb for new work on Google Cloud: start with Pub/Sub when several consumers need the same fact or when a producer must be decoupled from a slow consumer, use Eventarc when the trigger is something that happens inside the platform, use Cloud Tasks when you need to control the rate or timing of calls to one specific handler, use Cloud Scheduler for time-based triggers, use Workflows when the order of steps matters, and reach for Managed Service for Apache Kafka only when you need the Kafka protocol and ecosystem. Pub/Sub Lite is not an option for new work (see below).

The Bookshelf order flow combines them: orders-api publishes OrderPlaced to the orders topic, two subscriptions fan it out, and failed messages move to a dead-letter topic.

flowchart LR API["orders-api
publishes OrderPlaced"] --> T(("Pub/Sub topic
orders")) T -->|"push subscription
notifier-push"| N["notifier
Cloud Run"] T -->|"BigQuery subscription
orders-archive"| BQ["BigQuery table
orders_archive"] N --> M["Email provider
confirmation email"] N -. "after 5 failed deliveries" .-> D(("Dead-letter topic
orders-dlq")) D --> DS["orders-dlq-pull
inspect and replay"]

Cloud Pub/Sub

Pub/Sub is a global, fully managed messaging service. A publisher sends messages to a topic; each subscription attached to the topic gets its own copy of every message published after the subscription was created. Within one subscription, messages are shared among the subscribers that use it, which gives you both fan-out (several subscriptions) and competing consumers (several clients on one subscription). A message holds a payload of up to 10 MB and optional string attributes; Pub/Sub adds a message ID and a publish time.

Topics and subscription types

Subscription type How messages are delivered Use it for

Pull

The client calls the API (streaming pull in the client libraries) and acknowledges each message

Workers that control their own pace, GKE deployments, Cloud Run worker pools, batch consumers

Push

Pub/Sub POSTs each message to an HTTPS endpoint; a 2xx response acknowledges it

Cloud Run services and functions, webhooks, scale-to-zero consumers

Push with authentication

As push, with an OIDC token for a service account in the Authorization header

Keeping the Cloud Run service private (roles/run.invoker for that service account)

BigQuery

Pub/Sub writes each message into a BigQuery table, optionally using the topic schema

Analytics and archive without a consumer to run

Cloud Storage

Pub/Sub batches messages into files in a bucket (text or Avro)

Cheap archive and data-lake landing

Export to other products

Dataflow templates and connectors read a pull subscription

Transformations and joins, see below

Practical rules for the main settings:

  • The acknowledgement deadline (default 10 seconds, up to 600) is how long Pub/Sub waits before redelivering an unacknowledged message. Client libraries extend it automatically while a handler runs.

  • Message retention on a subscription (default 7 days, up to 31) controls how long unacknowledged messages are kept; a topic can also retain messages so that new subscriptions or a seek can replay history.

  • Delivery is at least once unless you enable exactly-once delivery, so handlers must be idempotent. The notifier should record the order ID it already mailed.

  • A subscription that nobody pulls from accumulates backlog until retention expires; alert on subscription/num_undelivered_messages and on the oldest unacknowledged message age.

Ordering keys

Pub/Sub does not order messages by default. When you publish with an ordering key and the subscription has message ordering enabled, messages with the same key are delivered in the order they were published, per region. Use the order ID as the key so all events of one order stay in sequence, and keep the number of distinct keys high enough to parallelize. Ordering is a property of the subscription (--enable-message-ordering); the publisher must also enable it and use a regional endpoint so that all messages for a key reach the same location. If publishing a message with a key fails, the client library blocks later messages with that key until you resume publishing for the key.

Exactly-once delivery

With exactly-once delivery enabled on a pull subscription, Pub/Sub does not redeliver a message after a successful acknowledgement, and the acknowledgement response tells the client whether it succeeded. It applies per subscription and per region, and is available for pull and StreamingPull subscriptions only; push, BigQuery and Cloud Storage subscriptions do not offer it. It limits redelivery from Pub/Sub, not duplicate publishes by your producer or duplicate side effects inside your handler, so idempotent processing stays the safe design.

Dead-letter topics and retry policies

A dead-letter topic receives messages that a subscription could not get acknowledged after a maximum number of delivery attempts (5 to 100, default 5). Messages arrive with attributes naming the original subscription and the delivery attempt. To use one you must also give the Pub/Sub service agent the right roles: Publisher on the dead-letter topic and Subscriber on the source subscription. The retry policy chooses what happens between attempts: the default redelivers immediately, while exponential backoff (a minimum and maximum backoff, from 0 to 600 seconds) spaces attempts out, which protects a struggling downstream service and is the setting to prefer for push to Cloud Run.

Schemas

A Pub/Sub schema (Avro or Protocol Buffers) attached to a topic makes Pub/Sub reject messages that do not match, with JSON or binary encoding. Schemas also let a BigQuery subscription write columns instead of a raw payload. They are versioned by revisions, so a producer and consumer can evolve independently inside compatible changes. For schema evolution rules in general see Kafka serialization and schemas.

Message attributes, publish time and event time

Every message has a publish time assigned by Pub/Sub when it accepts the message. That is not when the business event occurred: a retry, a batch, or a late mobile client makes the two differ. Put the real event time in an attribute (for example eventTime in RFC 3339 format) and tell downstream processors to use it. Apache Beam on Dataflow, for instance, can take the event timestamp from a named attribute instead of the publish time, which keeps windowed aggregations correct when messages arrive late. Use other attributes for routing and filtering: a subscription can carry a filter such as attributes.eventType = "OrderPlaced", so it only receives matching messages, and filtered-out messages are acknowledged automatically and not billed as delivered to it. The filter is fixed when the subscription is created.

Pub/Sub Lite

Pub/Sub Lite was a zonal, lower-cost variant. It is deprecated: new customers could no longer enroll after 2024-09-24 and Google has announced that it will be turned down on 2027-01-31. Do not use it for new work; the official migration guidance points to standard Pub/Sub or Managed Service for Apache Kafka. Older articles that recommend it for cost reasons are outdated.

In the Google Cloud console

  1. Pick the project in the project picker, or open Cloud Shell, and enable the API if prompted (APIs & Services, Enable APIs and services, search for Cloud Pub/Sub API).

  2. Open Pub/Sub, then Topics, then Create topic. Enter the Topic ID orders, keep Add a default subscription cleared, and under Retention optionally enable Retain messages (7 days). Choose Create.

  3. Create the dead-letter topic the same way with the ID orders-dlq, and a pull subscription orders-dlq-pull on it.

  4. Open Subscriptions, then Create subscription. Set Subscription ID notifier-push, select the topic orders, and set Delivery type to Push.

  5. Enter the Endpoint URL of the Cloud Run service, tick Enable authentication, choose the service account pubsub-invoker, and set the Audience to the service URL.

  6. Under Retry policy choose Retry after exponential backoff delay, under Dead lettering choose Enable dead lettering, select orders-dlq and set Maximum delivery attempts to 5. When the console offers to grant the Pub/Sub service account the needed roles, accept.

  7. Optionally enable Message ordering and Exactly once delivery (pull subscriptions only), and set a Subscription filter. The filter cannot be changed after the subscription is created.

  8. To test, open the topic, choose the Messages tab, Publish message, add an attribute and send it, then use the Messages tab of a pull subscription to pull it.

With the gcloud CLI

PROJECT_ID=$(gcloud config get-value project)
PROJECT_NUMBER=$(gcloud projects describe "$PROJECT_ID" --format='value(projectNumber)')
REGION=us-central1

gcloud services enable pubsub.googleapis.com

# Topics: the main topic and its dead-letter topic
gcloud pubsub topics create orders --message-retention-duration=7d
gcloud pubsub topics create orders-dlq
gcloud pubsub subscriptions create orders-dlq-pull --topic=orders-dlq

# Service account that Pub/Sub uses to call the private Cloud Run service
gcloud iam service-accounts create pubsub-invoker \
  --display-name="Pub/Sub push invoker"
gcloud run services add-iam-policy-binding notifier --region="$REGION" \
  --member="serviceAccount:pubsub-invoker@$PROJECT_ID.iam.gserviceaccount.com" \
  --role=roles/run.invoker

# Push subscription with auth, exponential backoff, a filter and a dead-letter topic
SERVICE_URL=$(gcloud run services describe notifier --region="$REGION" \
  --format='value(status.url)')
gcloud pubsub subscriptions create notifier-push --topic=orders \
  --push-endpoint="$SERVICE_URL/events" \
  --push-auth-service-account="pubsub-invoker@$PROJECT_ID.iam.gserviceaccount.com" \
  --push-auth-token-audience="$SERVICE_URL" \
  --ack-deadline=60 \
  --min-retry-delay=10s --max-retry-delay=300s \
  --message-filter='attributes.eventType = "OrderPlaced"' \
  --dead-letter-topic=orders-dlq --max-delivery-attempts=5

# The Pub/Sub service agent must be allowed to forward dead letters
PUBSUB_SA="service-$PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com"
gcloud pubsub topics add-iam-policy-binding orders-dlq \
  --member="serviceAccount:$PUBSUB_SA" --role=roles/pubsub.publisher
gcloud pubsub subscriptions add-iam-policy-binding notifier-push \
  --member="serviceAccount:$PUBSUB_SA" --role=roles/pubsub.subscriber

# A pull subscription with ordering and exactly-once delivery
gcloud pubsub subscriptions create orders-worker --topic=orders \
  --enable-message-ordering --enable-exactly-once-delivery

# Publish with an attribute and an ordering key, then pull and acknowledge
gcloud pubsub topics publish orders \
  --message='{"orderId":"o-1001","status":"placed"}' \
  --attribute=eventType=OrderPlaced,eventTime=2026-10-01T09:30:00Z \
  --ordering-key=o-1001
gcloud pubsub subscriptions pull orders-worker --auto-ack --limit=5

References: gcloud pubsub topics create, gcloud pubsub subscriptions create, Handle message failures and Push subscriptions.

BigQuery, Cloud Storage and schema-backed subscriptions use the same command with different flags. The BigQuery table must exist, and the Pub/Sub service agent needs BigQuery Data Editor on it.

# Avro schema, topic that enforces it, and a BigQuery subscription that writes columns
gcloud pubsub schemas create order-schema --type=avro \
  --definition-file=order.avsc
gcloud pubsub topics create orders-typed \
  --schema=order-schema --message-encoding=json
gcloud pubsub subscriptions create orders-archive --topic=orders-typed \
  --bigquery-table="$PROJECT_ID:bookshelf.orders_archive" \
  --use-topic-schema --write-metadata

# A Cloud Storage subscription writing hourly Avro files
gcloud pubsub subscriptions create orders-files --topic=orders \
  --cloud-storage-bucket="bookshelf-uploads-$PROJECT_ID" \
  --cloud-storage-file-prefix=orders/ --cloud-storage-output-format=avro \
  --cloud-storage-max-duration=1h

References: Create BigQuery subscriptions, Create Cloud Storage subscriptions and Schemas.

IAM

Grant roles to the workload’s service account, not to users. roles/pubsub.publisher allows publishing to a topic, roles/pubsub.subscriber allows pulling and acknowledging on a subscription, roles/pubsub.viewer reads configuration, and roles/pubsub.editor or roles/pubsub.admin manage resources. Bind roles on the specific topic or subscription rather than the project. For the Bookshelf, orders-api gets Publisher on orders, and the notifier needs no Pub/Sub role at all with a push subscription because Pub/Sub calls it with the pubsub-invoker identity. See Identity and access for service accounts and Application Default Credentials.

Java code

Add the Google Cloud Libraries BOM and the Pub/Sub client. Credentials come from Application Default Credentials: the attached service account on Cloud Run, or gcloud auth application-default login with impersonation locally; never download a service account key.

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>libraries-bom</artifactId>
      <version>${libraries-bom.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>
<dependencies>
  <dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-pubsub</artifactId>
  </dependency>
</dependencies>

A publisher with an ordering key, batching and an event-time attribute:

package com.example.bookshelf.orders;

import com.google.api.core.ApiFuture;
import com.google.cloud.pubsub.v1.Publisher;
import com.google.protobuf.ByteString;
import com.google.pubsub.v1.PubsubMessage;
import com.google.pubsub.v1.TopicName;
import java.time.Instant;
import java.util.concurrent.TimeUnit;

public class OrderPublisher implements AutoCloseable {

    private final Publisher publisher;

    public OrderPublisher(String projectId, String region) throws Exception {
        TopicName topic = TopicName.of(projectId, "orders");
        // A regional endpoint is required for ordered delivery.
        this.publisher = Publisher.newBuilder(topic)
                .setEndpoint(region + "-pubsub.googleapis.com:443")
                .setEnableMessageOrdering(true)
                .build();
    }

    public String publish(String orderId, String json) throws Exception {
        PubsubMessage message = PubsubMessage.newBuilder()
                .setData(ByteString.copyFromUtf8(json))
                .putAttributes("eventType", "OrderPlaced")
                .putAttributes("eventTime", Instant.now().toString())
                .setOrderingKey(orderId)
                .build();
        ApiFuture<String> messageId = publisher.publish(message);
        return messageId.get(10, TimeUnit.SECONDS);
    }

    @Override
    public void close() throws Exception {
        publisher.shutdown();
        publisher.awaitTermination(30, TimeUnit.SECONDS);
    }
}

A streaming-pull subscriber for the pull subscription orders-worker. It acknowledges only after the work succeeds, so a failure leads to redelivery and, eventually, the dead-letter topic:

package com.example.bookshelf.notifier;

import com.google.cloud.pubsub.v1.AckReplyConsumer;
import com.google.cloud.pubsub.v1.MessageReceiver;
import com.google.cloud.pubsub.v1.Subscriber;
import com.google.pubsub.v1.ProjectSubscriptionName;
import com.google.pubsub.v1.PubsubMessage;
import java.util.concurrent.TimeUnit;

public class OrderSubscriber {

    public static void main(String[] args) throws Exception {
        String projectId = System.getenv("GOOGLE_CLOUD_PROJECT");
        ProjectSubscriptionName subscription =
                ProjectSubscriptionName.of(projectId, "orders-worker");

        MessageReceiver receiver = (PubsubMessage message, AckReplyConsumer consumer) -> {
            try {
                String orderId = message.getOrderingKey();
                String json = message.getData().toStringUtf8();
                handle(orderId, json);
                consumer.ack();
            } catch (Exception e) {
                consumer.nack();
            }
        };

        Subscriber subscriber = Subscriber.newBuilder(subscription, receiver).build();
        subscriber.startAsync().awaitRunning();
        subscriber.awaitTerminated(1, TimeUnit.HOURS);
    }

    private static void handle(String orderId, String json) {
        // Idempotent: skip when the order was already processed.
    }
}

References: Publish messages, Pull subscriptions and Java client library reference.

Spring Cloud GCP

Spring Cloud GCP wraps the client in PubSubTemplate and Spring Integration channel adapters, with Spring Boot auto-configuration reading spring.cloud.gcp.*. Use the 8.x line with Spring Boot 4 and the 7.4.x line with Spring Boot 3 and import the matching spring-cloud-gcp-dependencies BOM.

<dependency>
  <groupId>com.google.cloud</groupId>
  <artifactId>spring-cloud-gcp-starter-pubsub</artifactId>
</dependency>
spring:
  cloud:
    gcp:
      project-id: ${GOOGLE_CLOUD_PROJECT}
      pubsub:
        subscriber:
          executor-threads: 4
          max-ack-extension-period: 600
package com.example.bookshelf.orders;

import com.google.cloud.spring.pubsub.core.PubSubTemplate;
import com.google.cloud.spring.pubsub.support.GcpPubSubHeaders;
import java.util.Map;
import org.springframework.stereotype.Service;

@Service
public class OrderEvents {

    private final PubSubTemplate pubSubTemplate;

    public OrderEvents(PubSubTemplate pubSubTemplate) {
        this.pubSubTemplate = pubSubTemplate;
    }

    public void orderPlaced(String orderId, String json) {
        pubSubTemplate.publish("orders", json, Map.of(
                "eventType", "OrderPlaced",
                GcpPubSubHeaders.ORDERING_KEY, orderId));
    }
}

For a consumer, pubSubTemplate.subscribe("orders-worker", message → { …​; message.ack(); }) registers a streaming pull receiver. To keep application code independent of the broker, Spring Cloud GCP also ships a Spring Cloud Stream binder (spring-cloud-gcp-pubsub-stream-binder): declare a Consumer<String> bean, bind it with spring.cloud.stream.bindings.notify-in-0.destination=orders and a group that becomes the subscription name, and the binder creates the subscription for you. The programming model is explained in Spring Cloud Stream, the general Spring side in Spring Boot messaging overview and testing messaging, and the Quarkus equivalent in Quarkus messaging (Quarkus has a Google Cloud Pub/Sub connector for SmallRye Reactive Messaging). Configuration and profiles are in configuration and profiles.

References: Spring Cloud GCP and the Spring Cloud GCP project documentation.

The Pub/Sub emulator

The Pub/Sub emulator is a local server that implements the API, so tests need no cloud project. It ships as a gcloud component and as a container image.

gcloud components install pubsub-emulator
gcloud beta emulators pubsub start --project=my-project-id --host-port=localhost:8085

# in another shell, point clients at it; no credentials are needed
$(gcloud beta emulators pubsub env-init)
echo "$PUBSUB_EMULATOR_HOST"

The Java client and Spring Cloud GCP switch to the emulator when PUBSUB_EMULATOR_HOST is set (for Spring set spring.cloud.gcp.pubsub.emulator-host=localhost:8085). For automated tests prefer Testcontainers with the emulator image, as described in testing messaging. The emulator does not enforce everything the service does, so keep a few integration tests against a real throwaway topic.

Reference: Test applications locally with the emulator.

Eventarc

Eventarc routes events from Google Cloud services, audit logs, Pub/Sub topics and your own applications to destinations, delivering them as CloudEvents. Standard Eventarc (the edition most guides describe) creates triggers that filter on event attributes and deliver to Cloud Run, GKE or Workflows. Eventarc Advanced adds a message bus, a pipeline with transformations, and enrollments that match messages on the bus, so you can route events across projects and from custom sources. Both are billed per event processed, with details on the pricing page; check the launch stage of each feature before using it, because some Advanced capabilities were still in Preview at the time of writing.

Use Eventarc when the trigger is something the platform already emits, for example google.cloud.storage.object.v1.finalized for an upload, a Firestore document write, or an audit-logged administrative action. When you control the producer, publishing to your own Pub/Sub topic and subscribing is simpler and gives you retention, replay and dead-lettering; the comparison is in Cloud Run functions and events.

flowchart LR S["Event source
Cloud Storage upload"] --> E["Eventarc trigger
filter by type and bucket"] E -->|"CloudEvent over HTTP"| R["Cloud Run service
thumbnailer"] E -. "uses" .-> P(("Pub/Sub topic
managed transport")) R --> O["bookshelf-uploads
thumbnails written back"]

In the Google Cloud console

  1. Enable the Eventarc API, then open Eventarc, Triggers, Create trigger.

  2. Name it uploads-to-thumbnailer, choose a region, and under Event provider select Cloud Storage with the event type google.cloud.storage.object.v1.finalized, filtering on the bucket bookshelf-uploads-my-project-id.

  3. As Destination choose Cloud Run, select the service and path, and for Service account pick a dedicated account that has roles/run.invoker and roles/eventarc.eventReceiver.

  4. Choose Create. For Cloud Storage triggers, the Cloud Storage service agent also needs roles/pubsub.publisher.

With the gcloud CLI

gcloud services enable eventarc.googleapis.com run.googleapis.com

gcloud iam service-accounts create eventarc-trigger \
  --display-name="Eventarc trigger identity"
for ROLE in roles/eventarc.eventReceiver roles/run.invoker; do
  gcloud projects add-iam-policy-binding "$PROJECT_ID" \
    --member="serviceAccount:eventarc-trigger@$PROJECT_ID.iam.gserviceaccount.com" \
    --role="$ROLE"
done

gcloud eventarc triggers create uploads-to-thumbnailer \
  --location="$REGION" \
  --destination-run-service=thumbnailer \
  --destination-run-region="$REGION" \
  --destination-run-path=/events \
  --event-filters="type=google.cloud.storage.object.v1.finalized" \
  --event-filters="bucket=bookshelf-uploads-$PROJECT_ID" \
  --service-account="eventarc-trigger@$PROJECT_ID.iam.gserviceaccount.com"

# A trigger over your own Pub/Sub topic
gcloud eventarc triggers create orders-to-notifier \
  --location="$REGION" \
  --destination-run-service=notifier \
  --destination-run-region="$REGION" \
  --event-filters="type=google.cloud.pubsub.topic.v1.messagePublished" \
  --transport-topic=orders \
  --service-account="eventarc-trigger@$PROJECT_ID.iam.gserviceaccount.com"

gcloud eventarc triggers list --location="$REGION"

References: gcloud eventarc triggers create, Eventarc documentation and Eventarc Advanced.

Cloud Tasks

Cloud Tasks is a managed queue of tasks, each an HTTP request (or App Engine request) that the service dispatches to a handler you name. Unlike Pub/Sub, where the producer does not know who consumes, a task names its target and the queue controls how it is dispatched: a maximum dispatch rate, a maximum number of concurrent dispatches, and a retry configuration with exponential backoff. Tasks can be scheduled for a future time (up to 30 days ahead) and a task given an explicit name is deduplicated for a period after creation.

Use it to protect a downstream API with a rate limit, to defer work out of a request path (a PDF render after checkout), or to retry a flaky call with a backoff you choose. A handler returns a 2xx status to complete a task; any other response triggers a retry per the queue configuration. Handlers must be idempotent, since delivery is at least once.

In the Google Cloud console

  1. Enable the Cloud Tasks API, then open Cloud Tasks, Queues, Create queue (or use Cloud Shell).

  2. Set the name invoice-render, a region, a Max dispatches per second of 5, Max concurrent dispatches of 10 and Max attempts of 8 with a minimum backoff of 10 seconds.

  3. Open the queue and choose Create task, then pick HTTP, enter the handler URL, the method, a body, and a service account for an OIDC token whose audience is the handler URL.

With the gcloud CLI

gcloud services enable cloudtasks.googleapis.com

gcloud tasks queues create invoice-render --location="$REGION" \
  --max-dispatches-per-second=5 --max-concurrent-dispatches=10 \
  --max-attempts=8 --min-backoff=10s --max-backoff=300s

HANDLER_URL=$(gcloud run services describe orders-api --region="$REGION" \
  --format='value(status.url)')
gcloud tasks create-http-task \
  --queue=invoice-render --location="$REGION" \
  --url="$HANDLER_URL/internal/render-invoice" \
  --method=POST --body-content='{"orderId":"o-1001"}' \
  --header=Content-Type:application/json \
  --oidc-service-account-email="tasks-invoker@$PROJECT_ID.iam.gserviceaccount.com" \
  --oidc-token-audience="$HANDLER_URL" \
  --schedule-time=2026-10-02T08:00:00Z

gcloud tasks list --queue=invoice-render --location="$REGION"

References: gcloud tasks queues create, gcloud tasks create-http-task and Cloud Tasks documentation.

The service account that signs the token needs roles/run.invoker on the handler, and the caller that creates tasks needs roles/cloudtasks.enqueuer and permission to act as that service account (roles/iam.serviceAccountUser). From Java use com.google.cloud:google-cloud-tasks with CloudTasksClient.createTask, building a Task with an HttpRequest and OidcToken.

Cloud Scheduler

Cloud Scheduler runs jobs on a cron schedule (unix-cron syntax) in a time zone you choose, and each run sends an HTTP request, publishes a Pub/Sub message, or calls an App Engine endpoint. It is the trigger for time-based work: the Bookshelf nightly-report Cloud Run job, a periodic cache refresh, or starting a workflow. Jobs have a retry policy and can be paused. Execution is at least once, so handlers must tolerate a repeat. A Cloud Run job is started by calling the Cloud Run Admin API with an OAuth token for a service account that has roles/run.invoker.

In the Google Cloud console

  1. Enable the Cloud Scheduler API, then open Cloud Scheduler, Create job.

  2. Enter the name nightly-report-trigger, region, frequency 0 2 * * * and time zone Europe/Madrid.

  3. For Target type select HTTP, paste the run URL of the job, choose POST, add an OAuth token with the service account scheduler-invoker, and configure retries. Choose Create, then Force run to test.

With the gcloud CLI

gcloud services enable cloudscheduler.googleapis.com

# Start the Cloud Run job nightly-report at 02:00 every day
gcloud scheduler jobs create http nightly-report-trigger \
  --location="$REGION" --schedule="0 2 * * *" --time-zone="Europe/Madrid" \
  --uri="https://run.googleapis.com/v2/projects/$PROJECT_ID/locations/$REGION/jobs/nightly-report:run" \
  --http-method=POST \
  --oauth-service-account-email="scheduler-invoker@$PROJECT_ID.iam.gserviceaccount.com" \
  --max-retry-attempts=3

# Publish a Pub/Sub message every hour
gcloud scheduler jobs create pubsub hourly-sync --location="$REGION" \
  --schedule="0 * * * *" --topic=orders --message-body='{"type":"sync"}'

gcloud scheduler jobs run nightly-report-trigger --location="$REGION"

References: gcloud scheduler jobs create http, gcloud scheduler jobs create pubsub and Cloud Scheduler documentation. Scheduling inside the application is discussed in Quarkus scheduling and mail; an external scheduler such as this one is preferable when the service scales to zero or runs several instances.

Workflows

Workflows runs a workflow: a series of steps written in YAML or JSON that call HTTP endpoints, Google Cloud APIs through connectors, and other workflows. It handles retries with backoff, switch branching, loops, parallel branches, sub-workflows, waiting for a callback, and long waits, while keeping state between steps. Use it instead of chaining services through topics when the order of steps and the handling of failure matter. An execution is started through the API, from Cloud Scheduler, from Eventarc, or from the console.

A Bookshelf fulfilment workflow that charges a payment, reserves stock and then publishes a completion event:

main:
  params: [order]
  steps:
    - charge:
        try:
          call: http.post
          args:
            url: https://payments.example.com/charge
            auth:
              type: OIDC
            body:
              orderId: ${order.orderId}
              amount: ${order.total}
          result: charged
        retry:
          predicate: ${http.default_retry_predicate}
          max_retries: 5
          backoff:
            initial_delay: 2
            max_delay: 60
            multiplier: 2
    - reserve:
        call: http.post
        args:
          url: https://orders-api-REPLACE_ME.run.app/internal/reserve
          auth:
            type: OIDC
          body:
            orderId: ${order.orderId}
        result: reserved
    - announce:
        call: googleapis.pubsub.v1.projects.topics.publish
        args:
          topic: projects/my-project-id/topics/orders
          body:
            messages:
              - data: ${base64.encode(text.encode("{\"type\":\"Fulfilled\"}"))}
    - done:
        return: ${reserved.body}

In the Google Cloud console

  1. Enable the Workflows API, then open Workflows, Create.

  2. Name it fulfil-order, pick a region, choose a Service account (create workflow-runner with the roles it needs, for example roles/pubsub.publisher and roles/run.invoker), then Next.

  3. Paste the YAML into the editor (the console shows a graph of the steps) and choose Deploy.

  4. Choose Execute and supply the input JSON \{"orderId":"o-1001","total":42\} to test.

With the gcloud CLI

gcloud services enable workflows.googleapis.com workflowexecutions.googleapis.com

gcloud workflows deploy fulfil-order --location="$REGION" \
  --source=fulfil-order.yaml \
  --service-account="workflow-runner@$PROJECT_ID.iam.gserviceaccount.com"

gcloud workflows run fulfil-order --location="$REGION" \
  --data='{"orderId":"o-1001","total":42}'

gcloud workflows executions list fulfil-order --location="$REGION" --limit=5

References: gcloud workflows deploy, Workflows syntax reference and Workflows documentation.

Managed Service for Apache Kafka

Managed Service for Apache Kafka runs Apache Kafka clusters for you inside a VPC, with Google operating brokers, upgrades and storage. Use it when you need the Kafka protocol, Kafka Connect, Kafka Streams or the wider ecosystem, or when you migrate an existing Kafka application. Clients connect over private IP from a subnet that the cluster is attached to. They authenticate with SASL: either OAuth bearer tokens issued from a Google identity (the GcpLoginCallbackHandler library) or SASL/PLAIN with a service account credential. Access is granted with IAM roles such as roles/managedkafka.client. Pricing depends mainly on the cluster’s vCPU and memory, storage and cross-zone network traffic; check the launch stage and the supported Kafka version on the product page, which changes faster than this guide.

Compared with Pub/Sub, Kafka gives you a replayable ordered log per partition, consumer groups with offsets you control, compaction and a mature client ecosystem, at the price of capacity planning and partition design. Pub/Sub scales without partitions and integrates natively with Google services. The Kafka side is covered in depth in Kafka architecture, producers, consumers, security and Spring Boot Kafka; this page only shows the Google-specific parts.

In the Google Cloud console

  1. Enable the Managed Service for Apache Kafka API, then open Managed Service for Apache Kafka, Clusters, Create.

  2. Name the cluster bookshelf-kafka, choose the region, set the vCPU and memory (the smallest supported size is enough for a lab), and pick the VPC and subnet the clients live in.

  3. Choose Create; creation takes some time. Then open the cluster, Topics, Create topic, name order-events, with 3 partitions and a replication factor of 3.

  4. Grant the application’s service account roles/managedkafka.client on the project.

With the gcloud CLI

gcloud services enable managedkafka.googleapis.com

gcloud managed-kafka clusters create bookshelf-kafka \
  --location="$REGION" --cpu=3 --memory=3GiB \
  --subnets="projects/$PROJECT_ID/regions/$REGION/subnetworks/default" \
  --async

gcloud managed-kafka topics create order-events \
  --cluster=bookshelf-kafka --location="$REGION" \
  --partitions=3 --replication-factor=3

gcloud projects add-iam-policy-binding "$PROJECT_ID" \
  --member="serviceAccount:orders-api@$PROJECT_ID.iam.gserviceaccount.com" \
  --role=roles/managedkafka.client

A Spring Boot client uses the standard Kafka properties with the Google login handler; the bootstrap address and the token handler come from the cluster page.

spring:
  kafka:
    bootstrap-servers: ${KAFKA_BOOTSTRAP}
    properties:
      security.protocol: SASL_SSL
      sasl.mechanism: OAUTHBEARER
      sasl.login.callback.handler.class: com.google.cloud.hosted.kafka.auth.GcpLoginCallbackHandler
      sasl.jaas.config: org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;

References: Managed Service for Apache Kafka quickstart, gcloud managed-kafka and client authentication. Operations topics are in Kafka operations and monitoring, and the comparison with RabbitMQ in Kafka vs RabbitMQ.

Application Integration

Application Integration is a low-code integration platform: you design an integration in a visual editor with triggers (API, Pub/Sub, scheduler), tasks (data mapping, branching, calling REST) and connectors to SaaS and enterprise systems such as Salesforce, SAP or databases, backed by managed authentication. It suits integration teams who prefer configuration to code and need many prebuilt connectors; for a developer-owned flow between your own services, Pub/Sub with Workflows is usually lighter. Console path: Application Integration, Create integration. Pricing is per execution and connector node, on the product’s pricing page.

Reference: Application Integration overview.

Pricing model

All the services are pay-for-use, with no hard-coded prices here; use the dated pricing pages linked under References.

  • Pub/Sub: throughput-based billing on the volume of data published and delivered (with a free monthly allowance), plus storage for retained messages, and extra charges for subscription types that write to other products. Many small messages cost more than their bytes suggest because of per-request minimums.

  • Eventarc: per event processed, with a free tier on the Standard edition.

  • Cloud Tasks and Cloud Scheduler: per million task operations and per job per month, each with a small free allowance.

  • Workflows: per step executed, with separate rates for internal and external steps.

  • Managed Service for Apache Kafka: per vCPU-hour and memory, plus storage and inter-zone traffic.

Quotas and limits

Quotas change, so each limit below should be confirmed on the linked page. A Pub/Sub message payload is limited to 10 MB and a message may have up to 100 attributes. A topic can have up to 10,000 subscriptions in a project. Acknowledgement deadlines range from 10 to 600 seconds. A Cloud Tasks task body is limited to 1 MB and a task can be scheduled up to 30 days ahead. Workflows limit the size of an execution’s variables and the number of steps per execution. Request quota increases from IAM & Admin, Quotas & System Limits.

Troubleshooting

  • Messages redelivered repeatedly. The handler is slower than the acknowledgement deadline or fails. Raise the deadline, or let the client library extend it, and make the handler idempotent.

  • Push endpoint returns 403. The subscription’s service account lacks roles/run.invoker, or the token audience does not match the service URL.

  • Dead letters never arrive. The Pub/Sub service agent lacks Publisher on the dead-letter topic or Subscriber on the subscription.

  • Out-of-order delivery despite an ordering key. The subscription was created without message ordering, or publishers used the global endpoint instead of a regional one.

  • An Eventarc trigger never fires. The service account is missing roles/eventarc.eventReceiver, the Cloud Storage service agent cannot publish to Pub/Sub, or the trigger region differs from the bucket location.

  • Cloud Tasks retries a completed task. The handler returned a non-2xx status or timed out. Return 2xx quickly and do long work asynchronously.

  • Scheduler job fails with 401 or 403. The job uses an OAuth token for a Google API URL, or an OIDC token for your own service; match the token type to the target.

Where older guides are outdated

  • Older articles recommend Pub/Sub Lite for cheap streaming; it is deprecated and scheduled for turn-down on 2027-01-31.

  • Pub/Sub push to a service remains valid, but Eventarc triggers are now the general path for platform events, as covered in Cloud Run functions and events.

  • Guides that publish with gcloud pubsub topics publish in a loop for load tests and rely on global endpoints for ordering predate regional-endpoint ordering requirements.

  • Tutorials that use downloaded service account keys to authenticate publishers should use attached service accounts and Application Default Credentials.

  • Older Cloud Functions guides use the functions.googleapis.com event names; the current model is Cloud Run functions with Eventarc triggers.

  • Dataflow SQL and some legacy Dataflow templates have changed; check the current Dataflow documentation before copying a streaming-SQL example.

Clean up

Delete resources in dependency order so nothing keeps triggering something that is already gone.

gcloud scheduler jobs delete nightly-report-trigger --location="$REGION" --quiet
gcloud scheduler jobs delete hourly-sync --location="$REGION" --quiet
gcloud eventarc triggers delete uploads-to-thumbnailer --location="$REGION" --quiet
gcloud eventarc triggers delete orders-to-notifier --location="$REGION" --quiet
gcloud workflows delete fulfil-order --location="$REGION" --quiet
gcloud tasks queues delete invoice-render --location="$REGION" --quiet
gcloud pubsub subscriptions delete notifier-push orders-worker orders-archive \
  orders-files orders-dlq-pull --quiet
gcloud pubsub topics delete orders orders-typed orders-dlq --quiet
gcloud pubsub schemas delete order-schema --quiet
gcloud managed-kafka topics delete order-events \
  --cluster=bookshelf-kafka --location="$REGION" --quiet
gcloud managed-kafka clusters delete bookshelf-kafka --location="$REGION" --quiet

Cloud Tasks queue names cannot be reused for about a week after deletion. A Managed Kafka cluster keeps billing until it is deleted, so delete it first in a lab. If you created everything in a throwaway project, deleting the project removes all of it (see Resource hierarchy and projects).

References

Related pages