AI, Data and Analytics Overview

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

See the section bibliography for the reference material consulted while preparing these pages.

This page is a hands-on overview of the Google Cloud services for storing, moving, analysing and learning from data: BigQuery for warehousing and SQL-based machine learning, Dataflow, Dataproc and Cloud Data Fusion for processing, Pub/Sub for streaming ingestion, Vertex AI for custom machine learning, Gemini for generative AI, and Looker for dashboards. It is deliberately an overview: each product gets a short explanation, a console and a command-line first step, and links to the official documentation for everything deeper. Commands are current for the gcloud CLI 586 line and the bq tool that ships with it; product names, editions, launch stages and prices of these services change faster than most of the platform, so everything was checked against the official documentation on 2026-10-01 and should be re-checked before you rely on it. Examples use the placeholder project my-project-id, project number <project-number> and region us-central1.

In the running Bookshelf scenario (see the section introduction) the operational data lives in Cloud SQL and Firestore, and orders-api publishes an OrderPlaced message to the Pub/Sub topic orders (see Messaging and integration). The business now wants answers that a transactional database is poor at: revenue per day, the books most often bought together, a daily deduplicated order history, and a model that predicts which orders will be returned. This page builds that analytical side. Transactional database choices are in Managed databases overview.

flowchart LR API["orders-api"] --> T(("Pub/Sub topic
orders")) T -->|"BigQuery subscription"| RAW["BigQuery
bookshelf.orders_raw"] T -->|"Dataflow streaming job"| DF["Dataflow
enrich and window"] DF --> FACT["BigQuery
bookshelf.orders
partitioned and clustered"] GCS["Cloud Storage
bookshelf-uploads"] -->|"batch load / Spark"| FACT RAW -->|"MERGE"| FACT FACT --> ML["BigQuery ML
return-risk model"] FACT --> BI["Looker Studio
dashboards"] FACT --> VAI["Vertex AI
custom training, endpoints"] FACT --> GEM["Gemini
summaries and SQL help"]

Choosing a service

Most data questions on Google Cloud begin with the same decision: do you want to write SQL against a managed warehouse, run your own pipeline code, or run an existing Hadoop or Spark job?

Service What it is Reach for it when

BigQuery

Serverless, columnar data warehouse with separated storage and compute, built-in streaming ingest, SQL machine learning and vector search

You can express the work as SQL, the data is large, or you want no cluster to operate

Dataflow

Managed runner for Apache Beam batch and streaming pipelines, with prebuilt templates

You need per-record transformation, windowing, joins on streams, or exactly-once streaming into a sink

Dataproc

Managed Hadoop and Spark clusters, and Serverless for Apache Spark batches and sessions

You have existing Spark, Hive or Hadoop code, or need the open-source ecosystem

Cloud Data Fusion

Visual, code-free pipeline builder based on the open-source CDAP, executing on Dataproc

Teams want drag-and-drop ETL with many ready connectors

Pub/Sub

Global messaging service; subscriptions can write directly to BigQuery and Cloud Storage

Events must flow into analytics continuously

Vertex AI

Platform for custom machine learning: notebooks, training, model registry, endpoints, batch prediction (now branded Gemini Enterprise Agent Platform)

You train and serve your own models, or build agents on Google models

Gemini

Google’s family of generative models, reachable from the platform, the Gemini API and the console

You need text, code, image or multimodal generation and embeddings

Looker and Looker Studio

Business intelligence: a governed semantic layer (Looker) and a free dashboard tool (Looker Studio)

Business users need dashboards over BigQuery

A rule of thumb: try BigQuery first; add Pub/Sub plus a BigQuery subscription for simple streaming; move to Dataflow when the transformation cannot be SQL; use Dataproc when you inherit Spark; and use Vertex AI when BigQuery ML does not offer the model you need.

BigQuery

BigQuery stores tables in a columnar format and runs SQL on a large pool of workers that you never see. Storage and compute are billed separately, and the same data can be queried from SQL, from the client libraries, from Spark, and from notebooks. Resources are organised as project, dataset and table (or view, routine, model); the dataset holds the access control and, importantly, its location (a region such as us-central1 or a multi-region such as US), which is fixed when the dataset is created. Queries cannot join tables across locations, so choose the dataset location to match your data and the services that write to it. For the Bookshelf the dataset is bookshelf in us-central1.

Editions, slots and pricing model

Compute is billed in one of two ways. On-demand pricing charges for the bytes a query processes, which is why scanning fewer bytes (by selecting only needed columns and by partitioning and clustering, below) directly lowers cost. Capacity pricing, sold as BigQuery editions (Standard, Enterprise and Enterprise Plus), charges for slots, units of compute that you reserve or let autoscale, regardless of bytes. Editions also gate features: for instance BigQuery vector indexes require the Enterprise or Enterprise Plus edition, and the more advanced governance and long-term-commitment options sit in the higher editions. Storage is billed per logical or physical byte, with a lower rate for tables or partitions not modified for 90 days; time travel and fail-safe storage (see below) can add to physical storage cost. Loading data from Cloud Storage with a standard load job is free of query charges, while streaming through the Storage Write API is billed per byte ingested. A monthly amount of query processing and storage is free of charge; check the dated pricing page before planning. Set a project-level or user-level maximum bytes billed and custom quotas on query usage per day to avoid a runaway query.

Console tour and first query

In the Google Cloud console

  1. Open the console, select the project in the project picker, and open BigQuery from the navigation menu (or press the search bar and type BigQuery). The API is enabled by default in new projects; otherwise enable it from APIs & Services.

  2. In the Explorer pane, click the three-dot menu on the project, choose Create dataset, enter bookshelf, pick the location us-central1, and click Create dataset. The location cannot be changed afterwards.

  3. Click the dataset, choose Create table, set Create table from to Upload or Google Cloud Storage, pick a CSV such as orders.csv, choose the file format, name the table orders_raw and tick Auto detect for the schema (or paste a schema as JSON). Click Create table.

  4. Open the SQL query editor tab. Before running, look at the validator message at the top right: it shows how many bytes the query will process, which is a dry run. Run the query and read the result and the Job information tab for bytes billed and slot time.

  5. Repeating an identical query within 24 hours is normally answered from the query cache at no charge. Open Query settings to disable the cache when you are benchmarking.

With the gcloud CLI

BigQuery is driven by the bq command-line tool, installed with the gcloud CLI; gcloud itself has only a few gcloud bigquery commands (for example for migration and reservations).

gcloud services enable bigquery.googleapis.com bigquerystorage.googleapis.com
PROJECT_ID=$(gcloud config get-value project)
REGION=us-central1

# Create a dataset in a fixed location
bq --location="$REGION" mk --dataset \
  --description "Bookshelf analytics" "$PROJECT_ID:bookshelf"

# Load a CSV from Cloud Storage with schema auto-detection
bq load --source_format=CSV --autodetect --skip_leading_rows=1 \
  bookshelf.orders_raw "gs://bookshelf-uploads-$PROJECT_ID/exports/orders.csv"

# Dry run first: report bytes processed without running or billing
bq query --use_legacy_sql=false --dry_run \
  'SELECT order_id, total FROM bookshelf.orders_raw WHERE status = "placed"'

# Run it, capping the bytes this query may bill
bq query --use_legacy_sql=false --maximum_bytes_billed=1000000000 \
  'SELECT status, COUNT(*) AS n FROM bookshelf.orders_raw GROUP BY status'

# Inspect a table schema and size
bq show --format=prettyjson bookshelf.orders_raw

References: Create datasets, Load CSV data from Cloud Storage, Estimate and control query costs, bq command-line reference and Introduction to BigQuery editions.

Older guides still use gsutil for staging files; the current tool for Cloud Storage is gcloud storage (see Cloud Storage).

Partitioning and clustering

A partitioned table is divided into segments by a date or timestamp column, by an integer range, or by ingestion time. When a query filters on the partition column, BigQuery reads only the matching partitions (partition pruning) and bills only those bytes. Clustering sorts the data within each partition by up to four columns, so filters and aggregations on those columns read fewer blocks. Partitioning gives a predictable, up-front cost reduction and the ability to expire old partitions; clustering gives a best-effort reduction and works well with high-cardinality columns. Use both: the Bookshelf orders table is partitioned by order date and clustered by customer_id and status. Ingestion-time partitioning uses a pseudo column _PARTITIONTIME and is convenient when records have no usable event date. You can require a partition filter so that a forgotten WHERE clause cannot scan the whole table, and set a partition expiration to age data out automatically.

bq query --use_legacy_sql=false '
CREATE TABLE bookshelf.orders (
  order_id STRING NOT NULL,
  customer_id STRING,
  status STRING,
  total NUMERIC,
  order_ts TIMESTAMP
)
PARTITION BY DATE(order_ts)
CLUSTER BY customer_id, status
OPTIONS (
  require_partition_filter = TRUE,
  partition_expiration_days = 1095,
  description = "Bookshelf orders, partitioned by day"
)'

# Measure the effect: compare bytes for a filter on the partition column vs not
bq query --use_legacy_sql=false --dry_run \
  'SELECT SUM(total) FROM bookshelf.orders
   WHERE DATE(order_ts) BETWEEN "2026-09-01" AND "2026-09-07" AND status = "placed"'

Reference: Introduction to partitioned tables, Introduction to clustered tables and Create partitioned tables.

In the console the same options appear under Partition and cluster settings when you create a table. A dry run is the tool for judging a design: run the same query against the unpartitioned and the partitioned copy and compare the bytes reported.

Useful SQL patterns

GoogleSQL has a few constructs that appear constantly in warehouse work.

Pivoting turns rows into columns with PIVOT:

SELECT *
FROM (SELECT DATE_TRUNC(DATE(order_ts), MONTH) AS month, status, total
      FROM bookshelf.orders
      WHERE DATE(order_ts) >= "2026-01-01")
PIVOT (SUM(total) FOR status IN ("placed", "shipped", "returned"))
ORDER BY month;

Top one per group is cheapest with ARRAY_AGG and LIMIT 1, which avoids a self join:

SELECT customer_id,
       ARRAY_AGG(STRUCT(order_id, total, order_ts)
                 ORDER BY order_ts DESC LIMIT 1)[OFFSET(0)] AS latest
FROM bookshelf.orders
WHERE DATE(order_ts) >= "2026-01-01"
GROUP BY customer_id;

Deduplication matters because streaming and retried loads deliver at-least-once. Keep the newest row per key with QUALIFY and a window function, and fold new data into the warehouse table with an idempotent MERGE:

MERGE bookshelf.orders AS t
USING (
  SELECT * EXCEPT (rn) FROM (
    SELECT *, ROW_NUMBER() OVER (PARTITION BY order_id ORDER BY order_ts DESC) AS rn
    FROM bookshelf.orders_raw
  ) WHERE rn = 1
) AS s
ON t.order_id = s.order_id AND DATE(t.order_ts) = DATE(s.order_ts)
WHEN MATCHED THEN
  UPDATE SET status = s.status, total = s.total
WHEN NOT MATCHED THEN
  INSERT (order_id, customer_id, status, total, order_ts)
  VALUES (s.order_id, s.customer_id, s.status, s.total, s.order_ts);

References: PIVOT operator, MERGE statement and QUALIFY clause.

Putting the partition column in the MERGE join condition lets BigQuery prune partitions of the target table, which keeps the cost of a nightly merge proportional to the new data instead of to the whole table.

Time travel and recovery

BigQuery keeps the previous states of a table for a time travel window (7 days by default, configurable between 2 and 7 days per dataset). Within that window you can query a table as it was, restore a dropped or overwritten table, or undelete a deleted table. After the window closes, data stays for a further 7-day fail-safe period that only Google support can use. Deleted datasets can be restored for the same window too.

In the Google Cloud console

  1. In the editor, query the past state with FOR SYSTEM_TIME AS OF, for instance an hour ago.

  2. To recover a deleted table, run a table copy from a snapshot decorator in SQL, or use the bq cp command below. The console does not offer a one-click undelete for tables.

  3. To change the window, open the dataset, choose Edit details, expand Advanced options and set the Time travel window.

With the gcloud CLI

# Query the table as it was 1 hour ago
bq query --use_legacy_sql=false \
  'SELECT COUNT(*) FROM bookshelf.orders
   FOR SYSTEM_TIME AS OF TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR)'

# Restore a deleted table: copy it from a point in time (milliseconds since epoch) to a new name
NOW_MS=$(($(date +%s) * 1000))
bq cp "bookshelf.orders@$((NOW_MS - 600000))" bookshelf.orders_restored

# Lengthen or shorten the time travel window for the dataset (hours)
bq update --max_time_travel_hours=168 "$PROJECT_ID:bookshelf"

# Keep a long-lived, cheap copy of a table with a snapshot
bq cp --snapshot --no_clobber \
  bookshelf.orders bookshelf.orders_snapshot_20261001

References: Data retention with time travel and fail-safe, Introduction to table snapshots and Introduction to table clones.

Time travel is a safety net for mistakes, not a backup strategy. Table snapshots, scheduled exports to Cloud Storage and BigQuery’s cross-region dataset replication cover retention and disaster recovery.

Loading and streaming data

There are four ways in, and the cost and latency differ.

Method Latency and cost Use it for

Batch load job (files in Cloud Storage)

Minutes; the load itself is free of query charges

Daily files, backfills, exports from other systems

Storage Write API

Seconds; billed per byte ingested, with a free monthly allowance, supports exactly-once through stream offsets

Application and pipeline streaming; the replacement for the older insertAll streaming API

Pub/Sub BigQuery subscription

Seconds; no code to run

Every message written to a table, optionally with the topic schema

Dataflow template or pipeline

Seconds to minutes; you pay for the workers

Transformation, enrichment and windowing before the write

Mixing sources is common and cost-effective. Streaming inserts put new rows within seconds in a write-optimised buffer, whereas batch loads cost less per byte; many pipelines stream recent data for dashboards and then run a nightly batch load or MERGE to produce the compacted history. A Pub/Sub BigQuery subscription is configured in Messaging and integration. From Java, write with the BigQuery Storage Write API client in the Google Cloud Client Libraries (the google-cloud-bigquerystorage artifact managed by libraries-bom) using Application Default Credentials and an attached service account, never a downloaded key.

References: Storage Write API overview, Introduction to loading data and Pub/Sub BigQuery subscriptions.

BigQuery ML

BigQuery ML lets you create, evaluate and run models using only SQL, with the data staying in the warehouse. It supports linear and logistic regression, boosted trees, k-means clustering, matrix factorisation, time-series forecasting, deep neural networks and imported TensorFlow or ONNX models, and it can call remote models such as Gemini or models deployed on Vertex AI. The workflow is CREATE MODEL, ML.EVALUATE, ML.PREDICT.

CREATE OR REPLACE MODEL bookshelf.return_risk
OPTIONS (model_type = "LOGISTIC_REG", input_label_cols = ["returned"]) AS
SELECT total, EXTRACT(HOUR FROM order_ts) AS hour, customer_segment, returned
FROM bookshelf.orders_features
WHERE DATE(order_ts) < "2026-09-01";

SELECT * FROM ML.EVALUATE(MODEL bookshelf.return_risk, (
  SELECT total, EXTRACT(HOUR FROM order_ts) AS hour, customer_segment, returned
  FROM bookshelf.orders_features
  WHERE DATE(order_ts) >= "2026-09-01"));

SELECT order_id, predicted_returned_probs
FROM ML.PREDICT(MODEL bookshelf.return_risk, (
  SELECT order_id, total, EXTRACT(HOUR FROM order_ts) AS hour, customer_segment
  FROM bookshelf.orders_features
  WHERE DATE(order_ts) = CURRENT_DATE()));

References: Introduction to BigQuery ML, CREATE MODEL and ML.PREDICT.

Training on-demand is billed by bytes processed, with some model types (such as deep neural networks and boosted trees at scale) requiring a capacity edition; check the model-type table. To train outside SQL, read BigQuery data into pandas or tf.data through the BigQuery Storage API, which streams columns in parallel and is far faster than exporting CSV.

Vector search in BigQuery

BigQuery can store embeddings in an ARRAY<FLOAT64> column, generate them with ML.GENERATE_EMBEDDING over a remote model, and find nearest neighbours with the VECTOR_SEARCH function, optionally accelerated by a vector index (CREATE VECTOR INDEX, which needs the Enterprise or Enterprise Plus edition). It suits semantic search over data already in the warehouse, such as finding books similar to a description, alongside the other vector options compared in Managed databases overview and in Comparing vector databases. The general ideas of embeddings and retrieval are in Embeddings and RAG fundamentals.

CREATE VECTOR INDEX book_idx ON bookshelf.books(embedding)
OPTIONS (index_type = "IVF", distance_type = "COSINE");

SELECT base.title, distance
FROM VECTOR_SEARCH(TABLE bookshelf.books, "embedding",
  (SELECT embedding FROM bookshelf.queries WHERE id = "q1"),
  top_k => 5, distance_type => "COSINE");

References: Introduction to vector search, Manage vector indexes and Generate text embeddings.

Access control and governance

Grant access at the dataset level (and finer with table, column and row-level security): roles/bigquery.dataViewer reads, roles/bigquery.dataEditor writes tables, roles/bigquery.jobUser lets a principal run queries in a project (and is where query cost is attributed), and roles/bigquery.admin manages everything. The Bookshelf pipeline’s service account gets dataEditor on the bookshelf dataset and jobUser on the project. Use authorized views to share a subset of a dataset without sharing its tables, policy tags for column-level security of personal data, and row access policies to filter rows per user. Audit logs record who ran which job (see Governance, security and compliance), and the billing export to BigQuery described in Cost management and billing is the common way to analyse spend with SQL.

gcloud projects add-iam-policy-binding "$PROJECT_ID" \
  --member="serviceAccount:bookshelf-pipeline@$PROJECT_ID.iam.gserviceaccount.com" \
  --role=roles/bigquery.jobUser

bq add-iam-policy-binding \
  --member="serviceAccount:bookshelf-pipeline@$PROJECT_ID.iam.gserviceaccount.com" \
  --role=roles/bigquery.dataEditor "$PROJECT_ID:bookshelf"

References: BigQuery IAM roles and permissions, Authorized views and Column-level security.

Dataflow

Dataflow is Google’s fully managed runner for Apache Beam pipelines. You describe a pipeline as transforms over PCollections with the Beam SDK (Java, Python or Go), and Dataflow provisions workers, autoscales them, shuffles data and restarts failed work. The same code handles batch and streaming; streaming adds windows (fixed, sliding, session) and triggers that decide when partial results are emitted, driven by event time rather than by processing time. For Pub/Sub sources, Beam can read the event timestamp from a message attribute instead of the publish time, which keeps windows correct when messages arrive late (see the discussion in Messaging and integration).

Three styles of use cover most needs:

  • Provided templates are ready-made pipelines you start with parameters only, for example Pub/Sub to BigQuery, Pub/Sub to Cloud Storage, Cloud Storage text to BigQuery, and BigQuery to Bigtable. No code is required.

  • Flex Templates package your own Beam pipeline as a container image in Artifact Registry plus a JSON spec in Cloud Storage, so that others can launch it with parameters.

  • Your own pipeline launched directly from a development machine or CI.

A few Beam habits keep pipelines maintainable. Prefer Beam schemas (typed rows with named fields, from Java POJOs, Avro, Protocol Buffers or BigQuery) over hand-written coders; schemas let transforms such as Select, Join and GroupBy work by field name and map directly to BigQuery columns. Keep side effects idempotent, because retries can repeat work. Use a dead-letter output for records that fail parsing instead of failing the pipeline.

In the Google Cloud console

  1. Open Dataflow from the navigation menu, enable the Dataflow API if asked, and choose Jobs.

  2. Click Create job from template, enter a job name such as orders-to-bigquery, select the regional endpoint us-central1, and pick the template Pub/Sub Subscription to BigQuery.

  3. Enter the input subscription path, the output table my-project-id:bookshelf.orders_raw, and a temporary location in Cloud Storage. Under Optional parameters choose the worker service account (not the default Compute Engine one) and a maximum worker count.

  4. Click Run job. Open the job graph to watch throughput, system latency and data freshness, and use the Logs panel for errors.

With the gcloud CLI

gcloud services enable dataflow.googleapis.com compute.googleapis.com
PROJECT_ID=$(gcloud config get-value project)
REGION=us-central1
BUCKET="bookshelf-uploads-$PROJECT_ID"

# Worker identity with only what the job needs
gcloud iam service-accounts create bookshelf-dataflow \
  --display-name="Bookshelf Dataflow worker"
SA="bookshelf-dataflow@$PROJECT_ID.iam.gserviceaccount.com"
for ROLE in roles/dataflow.worker roles/pubsub.subscriber roles/bigquery.dataEditor; do
  gcloud projects add-iam-policy-binding "$PROJECT_ID" \
    --member="serviceAccount:$SA" --role="$ROLE"
done
gcloud storage buckets add-iam-policy-binding "gs://$BUCKET" \
  --member="serviceAccount:$SA" --role=roles/storage.objectAdmin

# Streaming job from a Google-provided template
gcloud dataflow jobs run orders-to-bigquery \
  --gcs-location="gs://dataflow-templates-$REGION/latest/PubSub_Subscription_to_BigQuery" \
  --region="$REGION" --service-account-email="$SA" \
  --staging-location="gs://$BUCKET/tmp" --max-workers=3 \
  --parameters=inputSubscription="projects/$PROJECT_ID/subscriptions/orders-dataflow",outputTableSpec="$PROJECT_ID:bookshelf.orders_raw"

# Watch and stop it
gcloud dataflow jobs list --region="$REGION" --status=active
gcloud dataflow jobs drain JOB_ID --region="$REGION"

References: Dataflow templates, gcloud dataflow jobs run, Pub/Sub Subscription to BigQuery template and Use Flex Templates.

The templates' exact parameter names change between versions; copy them from the current template page rather than from an older guide. drain finishes in-flight data before stopping a streaming job, whereas cancel stops it immediately. Replace JOB_ID with the identifier shown by jobs list.

A first Beam pipeline

A minimal batch pipeline in Python reads CSV lines from Cloud Storage, parses them and writes rows to BigQuery. Run it locally with DirectRunner while developing and on Dataflow with --runner=DataflowRunner. Install the SDK with pip install "apache-beam[gcp]".

import argparse
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions

def parse(line):
    order_id, customer_id, status, total, order_ts = line.split(",")
    return {"order_id": order_id, "customer_id": customer_id, "status": status,
            "total": float(total), "order_ts": order_ts}

def run(argv=None):
    parser = argparse.ArgumentParser()
    parser.add_argument("--input", required=True)
    parser.add_argument("--table", required=True)
    known, beam_args = parser.parse_known_args(argv)
    with beam.Pipeline(options=PipelineOptions(beam_args, save_main_session=True)) as p:
        (p
         | "Read" >> beam.io.ReadFromText(known.input, skip_header_lines=1)
         | "Parse" >> beam.Map(parse)
         | "Write" >> beam.io.WriteToBigQuery(
               known.table,
               schema="order_id:STRING,customer_id:STRING,status:STRING,total:NUMERIC,order_ts:TIMESTAMP",
               write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND,
               create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED))

if __name__ == "__main__":
    run()
python orders_pipeline.py \
  --input="gs://$BUCKET/exports/orders.csv" --table="$PROJECT_ID:bookshelf.orders_raw" \
  --runner=DataflowRunner --project="$PROJECT_ID" --region="$REGION" \
  --temp_location="gs://$BUCKET/tmp" --service_account_email="$SA"

References: Create a Dataflow pipeline, Apache Beam programming guide and Set Dataflow pipeline options.

A common pattern is to start a template when a file lands in a bucket: an Eventarc trigger on the Cloud Storage finalized event calls a small Cloud Run function that launches the template with the object path as a parameter (see Cloud Run functions and events). Older guides also show Dataflow SQL, a streaming SQL prototype that ran from the BigQuery console; its status and the console entry points have changed, so use Beam SQL in a Beam notebook or plain BigQuery SQL with a Pub/Sub BigQuery subscription instead, and check the current Dataflow documentation before copying a Dataflow SQL example.

Pricing, quotas and security

Dataflow bills for the vCPU, memory and storage of workers (plus the Shuffle or Streaming Engine service) per second, so cost follows worker count and job duration. Streaming jobs run until stopped, which makes forgotten jobs a classic bill. Quotas include the number of concurrent jobs per region, Compute Engine CPUs and in-use IP addresses. Run workers without public IP addresses in a subnet with Private Google Access (see VPC networking), grant the worker service account only the roles above, and use customer-managed encryption keys when policy requires them.

Dataproc and Spark

Dataproc runs open-source data tools. It has three modes: clusters on Compute Engine (you choose machine types, can use Spot VMs for secondary workers and autoscaling policies), clusters on GKE, and Serverless for Apache Spark, where you submit a batch or open an interactive session and Google provisions and scales the executors without a cluster to size. The Cloud Storage connector lets Spark and Hadoop read gs:// paths as if they were HDFS, and the spark-bigquery connector reads and writes BigQuery tables through the Storage API (it is preinstalled on recent image versions; otherwise add it with an initialization action or the --jars flag). Dataproc is the practical choice when you already have Spark or Hive jobs, need libraries outside Beam, or run notebooks on Spark.

In the Google Cloud console

  1. Open Dataproc from the navigation menu and enable the API when prompted.

  2. For Spark without a cluster, choose Serverless, Batches, Create, set the batch ID, region us-central1, the batch type (PySpark), the main Python file in Cloud Storage and a service account. Click Submit.

  3. For a cluster, choose Clusters, Create cluster, Cluster on Compute Engine, set the name and region, pick a single-node or standard layout and optional components, and click Create.

With the gcloud CLI

gcloud services enable dataproc.googleapis.com

# Serverless batch: no cluster to manage
gcloud dataproc batches submit pyspark "gs://$BUCKET/jobs/daily_orders.py" \
  --region="$REGION" --service-account="$SA" \
  --deps-bucket="$BUCKET" \
  -- --input="gs://$BUCKET/exports/orders.csv" --output="$PROJECT_ID.bookshelf.daily_orders"

# Classic cluster with the BigQuery connector initialization action, then a job
gcloud dataproc clusters create bookshelf-spark --region="$REGION" \
  --single-node --image-version=2.2-debian12 \
  --service-account="$SA" --scopes=cloud-platform \
  --max-idle=30m
gcloud dataproc jobs submit pyspark "gs://$BUCKET/jobs/daily_orders.py" \
  --cluster=bookshelf-spark --region="$REGION" \
  --jars=gs://spark-lib/bigquery/spark-bigquery-latest_2.12.jar

References: Serverless for Apache Spark overview, gcloud dataproc batches submit pyspark, BigQuery connector for Spark and Initialization actions.

--max-idle deletes an idle cluster automatically, the most effective cost control for classic clusters. Dataproc bills a per-vCPU management fee on top of the Compute Engine resources, while serverless batches bill by data compute units and are metered per second. Older guides use initialization actions to install connectors on every cluster; on current image versions prefer the bundled connectors or Dataproc’s component options and re-check the supported image list.

Cloud Data Fusion

Cloud Data Fusion is a managed, visual data-integration service built on the open-source CDAP. In the Studio you drag sources (Cloud Storage, BigQuery, databases through JDBC, Pub/Sub), transforms (Wrangler for interactive data preparation, joins, aggregators) and sinks onto a canvas and connect them. When you deploy and run a pipeline, Data Fusion creates an ephemeral Dataproc cluster in your project (or uses an existing one), runs the generated Spark job and deletes the cluster. It fits teams that prefer a GUI and the large library of plugins over writing Beam or Spark code, and it is the least cost-efficient of the processing options for small jobs because an instance bills by hour in addition to the Dataproc cluster. Pick the edition (Developer, Basic or Enterprise) by features and availability needs.

In the Google Cloud console

  1. Open Data Fusion from the navigation menu and enable the API. Choose Instances, Create an instance.

  2. Enter a name such as bookshelf-fusion, region us-central1 and the edition, then Create. Provisioning takes several minutes.

  3. Click View instance to open the CDAP web UI, then Studio. Add a GCS source, a Wrangler transform and a BigQuery sink, set the properties, and click Deploy. Click Run and follow logs in the pipeline summary.

With the gcloud CLI

gcloud services enable datafusion.googleapis.com

gcloud beta data-fusion instances create bookshelf-fusion \
  --location="$REGION" --edition=developer

gcloud beta data-fusion instances describe bookshelf-fusion \
  --location="$REGION" --format="value(apiEndpoint,state)"

References: Data Fusion overview, Create a Data Fusion instance and gcloud beta data-fusion instances create.

Pipelines can be exported as JSON and deployed with the instance’s REST API, which makes them part of a CI workflow. The command group is still under beta; check the reference if it has graduated.

Streaming with Pub/Sub

Pub/Sub is the entry point of most Google Cloud streaming architectures. The ways to move its data into analytics are the BigQuery subscription (no code, per-message rows), the Cloud Storage subscription (files for a data lake), Dataflow (transformations, windows, joins), and a Spark Structured Streaming job on Dataproc. The topic design, schemas, publish time versus event time, ordering and dead-letter handling are covered in Messaging and integration; the theory of event streams is in messaging models and patterns. For Bookshelf, give each consumer its own subscription on orders:

gcloud services enable pubsub.googleapis.com
gcloud pubsub subscriptions create orders-dataflow --topic=orders \
  --ack-deadline=60 --message-retention-duration=2d
gcloud pubsub subscriptions create orders-archive --topic=orders \
  --bigquery-table="$PROJECT_ID:bookshelf.orders_raw" \
  --write-metadata --drop-unknown-fields

Reference: gcloud pubsub subscriptions create and Stream from Pub/Sub to BigQuery with Dataflow.

Choose the BigQuery subscription when each message maps to one row, because it needs no workers and costs only the Pub/Sub throughput; choose Dataflow when you must enrich, aggregate or route messages. Apache Kafka users can run Managed Service for Apache Kafka and feed BigQuery with Kafka Connect or Dataflow; the Kafka concepts are in Messaging.

Vertex AI and the Gemini Enterprise Agent Platform

Vertex AI is Google Cloud’s platform for machine learning. At Cloud Next on 2026-04-22 it was rebranded Gemini Enterprise Agent Platform, and the documentation, console and some product names are moving to the new name while APIs, client libraries, IAM roles and CLI commands still say aiplatform or Vertex AI. Because this is the most volatile naming in the section, treat the table below as a map from names you will meet in older guides to the current ones and re-check the documentation before relying on it.

Older name Current name or home Notes

Vertex AI

Gemini Enterprise Agent Platform

API aiplatform.googleapis.com and the gcloud ai commands keep the old names

Vertex AI Workbench (managed and user-managed notebooks)

Workbench instances, and notebooks in the platform console

The older managed and user-managed notebook types were replaced by the single instances type; migrate existing notebooks

Vertex AI Search

Agent Search

Discovery Engine APIs and IAM names are unchanged

Vertex AI Vector Search 2.0

Agent Retrieval

Preview at the time of writing

Vertex AI custom training, endpoints, batch prediction, Model Registry

Same capabilities under the new platform name

Serverless training, dedicated endpoints and batch inference from Cloud Storage or BigQuery

Gemini on Vertex AI

Gemini models on the platform, also through the Gemini API

Model versions and availability change often

The Bookshelf uses the platform for the return-risk model once it outgrows BigQuery ML, and for a "recommend similar books" feature backed by Gemini embeddings.

Workbench notebooks

Workbench instances are JupyterLab environments on a Compute Engine VM, with preinstalled frameworks (PyTorch, TensorFlow, scikit-learn), optional GPUs, a BigQuery integration, scheduled notebook runs and idle shutdown. They are the place for exploration and prototyping before training moves to jobs. A custom container image, built with Cloud Build and stored in Artifact Registry (see Container registry and builds), pins the libraries so a notebook runs the same way for everyone.

In the Google Cloud console

  1. Open Vertex AI (or Gemini Enterprise Agent Platform) in the navigation menu, enable the API, and choose Workbench, then Instances, then Create new.

  2. Enter the name bookshelf-notebook, region us-central1 and zone, and choose a machine type. Under Advanced options select the service account, enable idle shutdown, and optionally add a GPU. Click Create.

  3. When the instance is active click Open JupyterLab. A first cell can read BigQuery data into pandas with the BigQuery client library.

With the gcloud CLI

gcloud services enable aiplatform.googleapis.com notebooks.googleapis.com

gcloud workbench instances create bookshelf-notebook \
  --location=us-central1-a \
  --machine-type=e2-standard-4 \
  --service-account-email="$SA" \
  --metadata=idle-timeout-seconds=3600

gcloud workbench instances list --location=us-central1-a
gcloud workbench instances stop bookshelf-notebook --location=us-central1-a

References: Workbench instances, gcloud workbench instances create and Create an instance with a custom container.

Notebook VMs bill like any Compute Engine VM while running, so stop them when idle. GPUs are subject to regional quotas that usually need a request.

Training, endpoints and batch prediction

Training a model in a notebook ties the work to one VM. Serverless custom training instead runs your code, as a Python package or a container, on resources that exist only for the duration of the job. The training service passes well-known locations to the code through environment variables; for instance AIP_MODEL_DIR is the Cloud Storage directory where the script must save the trained model. The saved model is registered in the Model Registry, then either deployed to an endpoint for low-latency online prediction or run against a dataset for batch prediction, reading from Cloud Storage or BigQuery and writing results back. Endpoints autoscale between a minimum and maximum replica count; dedicated public endpoints and private endpoints over Private Service Connect are the recommended types, and a model that stays deployed bills for its replicas even without traffic.

# Train with a prebuilt container and a local script
gcloud ai custom-jobs create --region="$REGION" \
  --display-name=return-risk-training \
  --worker-pool-spec=machine-type=n1-standard-4,replica-count=1,executor-image-uri=us-docker.pkg.dev/vertex-ai/training/sklearn-cpu.1-6:latest,local-package-path=./trainer,python-module=trainer.task \
  --service-account="$SA"

# Upload a trained model, create an endpoint and deploy it
gcloud ai models upload --region="$REGION" --display-name=return-risk \
  --artifact-uri="gs://$BUCKET/models/return-risk/" \
  --container-image-uri=us-docker.pkg.dev/vertex-ai/prediction/sklearn-cpu.1-6:latest
gcloud ai endpoints create --region="$REGION" --display-name=return-risk-endpoint
gcloud ai endpoints deploy-model ENDPOINT_ID --region="$REGION" \
  --model=MODEL_ID --display-name=return-risk-v1 \
  --machine-type=n1-standard-2 --min-replica-count=1 --max-replica-count=2

# Online and batch predictions
gcloud ai endpoints predict ENDPOINT_ID --region="$REGION" --json-request=request.json
gcloud ai batch-prediction-jobs create --region="$REGION" \
  --display-name=return-risk-batch --model=MODEL_ID \
  --input-uri="bq://$PROJECT_ID.bookshelf.orders_today" \
  --output-uri="bq://$PROJECT_ID.bookshelf.predictions"

References: Create a custom training job, Training code requirements and AIP_* variables, Get online predictions and Get batch predictions.

Replace ENDPOINT_ID and MODEL_ID with the numeric IDs printed by the previous commands. Prebuilt container image tags rise with each framework release, so copy the current URIs from the prebuilt-containers page. In the console the same actions are under Training, Model Registry, Online prediction and Batch predictions.

Gemini and generative AI

Gemini is Google’s family of multimodal generative models. They can be reached three ways: from the platform (Vertex AI API, with enterprise features such as IAM, VPC Service Controls, data residency and customer-managed keys), from the Gemini API in Google AI Studio (API-key based, aimed at prototyping), and as assistants inside Google products, such as Gemini Code Assist in the IDE and Gemini Cloud Assist in the console (Preview at the time of writing). For production work on Google Cloud use the platform path so that calls authenticate with a service account and Application Default Credentials, with no API key to manage. The platform also offers embeddings models, grounding with Google Search or your own data, context caching, function calling and an agent framework, and it hosts third-party and open models as well.

Calling Gemini from the console and the CLI

In the Google Cloud console

  1. Open the platform in the navigation menu and choose Studio (the prompt gallery and testing area).

  2. Choose a Gemini model, type a prompt such as "Summarise these order notes in one sentence", adjust the temperature and token limit, and click Submit.

  3. Use Get code to copy the same request as Python, Java or curl.

With the gcloud CLI

gcloud services enable aiplatform.googleapis.com
TOKEN=$(gcloud auth print-access-token)
MODEL=gemini-2.5-flash   # check the model list for current versions

curl -s -X POST \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  "https://$REGION-aiplatform.googleapis.com/v1/projects/$PROJECT_ID/locations/$REGION/publishers/google/models/$MODEL:generateContent" \
  -d '{"contents":[{"role":"user","parts":[{"text":"Summarise: customer asks to change the delivery address"}]}]}'

References: Google models, Generate content API reference and Quickstart.

Model identifiers are retired on published schedules, so read them from configuration (see Runtime configuration) instead of hard-coding them, and track the model-versions page for retirement dates.

From Java and Spring

Use the Google Gen AI SDK for Java (com.google.genai:google-genai), which targets the platform when configured with a project and location, so Application Default Credentials apply:

package com.example.bookshelf.ai;

import com.google.genai.Client;
import com.google.genai.types.GenerateContentResponse;

public class OrderNoteSummarizer {

    private final Client client = Client.builder()
            .vertexAI(true)
            .project(System.getenv("GOOGLE_CLOUD_PROJECT"))
            .location("us-central1")
            .build();

    public String summarize(String notes) {
        GenerateContentResponse response = client.models.generateContent(
                System.getenv().getOrDefault("GEMINI_MODEL", "gemini-2.5-flash"),
                "Summarise in one sentence: " + notes, null);
        return response.text();
    }
}

Reference: Generative AI SDKs and Application Default Credentials.

Spring applications can use Spring AI’s Google GenAI or Vertex AI integration instead of calling the SDK directly; see Integrating with Spring AI for chat clients, embeddings and vector stores, and Prompt augmentation and generation for the retrieval-augmented pattern. Give the Cloud Run service account roles/aiplatform.user to call models.

Retrieval, search and agents

For grounded answers over your own content, the platform provides Agent Search (formerly Vertex AI Search) for managed search and retrieval over websites, documents and structured data, Agent Retrieval (formerly Vector Search 2.0, in Preview) for vector retrieval at scale, and an agent framework with tool calling. You can also build retrieval-augmented generation yourself with embeddings stored in BigQuery, AlloyDB (ScaNN or pgvector), Cloud SQL (pgvector) or Firestore (see AlloyDB and PostgreSQL pgvector). Use managed search when you want results quickly; build your own when you need control of chunking, ranking and filters (see Document ingestion and chunking).

Pricing for generative models is per input and output token (or character, per modality), with different rates per model and discounts for batch use and context caching; there is no per-hour charge for on-demand Gemini, but provisioned throughput is a commitment. Every model has regional availability, per-minute quotas and a launch stage (Preview models carry no SLA). Prompts and responses are not used to train models for platform customers by default, and data residency and VPC Service Controls apply; verify the current data-governance page before processing personal data.

References: Generative AI pricing, Quotas and limits and Data governance.

Looker and Looker Studio

Looker Studio is a free, browser-based tool for building reports and dashboards, with connectors to BigQuery, Cloud SQL, Google Sheets and many other sources; it suits ad-hoc dashboards and sharing with colleagues who have Google accounts. Looker is the enterprise platform: its LookML modelling layer defines metrics once, so every dashboard and API consumer uses the same definitions, with governance, embedding and a developer workflow in Git. Both read BigQuery directly, so partitioned and clustered tables and BI Engine (an in-memory acceleration layer for BigQuery) keep dashboards fast and cheap. In the console, open Looker Studio from the Explore data menu of a BigQuery table, or go to the Looker Studio site and add a BigQuery data source. A viewer’s query is billed to the project of the data source’s credentials, so use a dedicated service account owner for shared dashboards.

References: Looker documentation, Looker Studio documentation and Introduction to BI Engine.

Cost, quotas and the shape of a bill

Data and AI services share a few cost traits worth planning for.

  • BigQuery on-demand cost follows bytes scanned; partitioning, clustering, selecting columns and dry runs control it. Capacity pricing follows slots and can be capped with a reservation’s maximum.

  • Streaming and storage costs accumulate silently: table storage, snapshots, long time travel windows and unexpired partitions all bill.

  • Dataflow streaming jobs, Dataproc clusters, Data Fusion instances, Workbench notebooks and deployed Vertex AI endpoints bill by the hour while they exist, whether or not they process data.

  • Gemini cost follows tokens; keep prompts short, cache context and use smaller models for simple tasks.

  • Egress between regions and out of Google Cloud is billed; keep datasets, buckets and jobs in one location.

  • Set budgets and alerts (see Cost management and billing), label resources by team, and export billing data to BigQuery.

References: BigQuery pricing, Dataflow pricing, Dataproc pricing and Vertex AI pricing. Pricing pages change; use the dated page and the pricing calculator.

What changed since older guides

  • The Google Cloud documentation host is docs.cloud.google.com; the older cloud.google.com/…​ URLs redirect.

  • Vertex AI was rebranded Gemini Enterprise Agent Platform on 2026-04-22; Vertex AI Search became Agent Search and Vector Search 2.0 became Agent Retrieval (Preview). APIs and IAM names keep the Vertex AI and Discovery Engine names.

  • Workbench’s managed and user-managed notebook types were replaced by Workbench instances.

  • Dataflow SQL and some legacy Dataflow templates have changed; prefer current templates, Beam SQL or BigQuery SQL.

  • The BigQuery Storage Write API supersedes the older insertAll streaming API for new work, and BigQuery editions (Standard, Enterprise, Enterprise Plus) replaced the older flat-rate commitments.

  • gsutil is superseded by gcloud storage for staging files; downloaded service-account keys are superseded by attached service accounts and Workload Identity Federation (see Identity and access).

  • Dataproc connectors are bundled in current image versions, so initialization actions for them are rarely needed.

Clean up

Delete billable resources first, in dependency order. Data services keep charging while idle.

# Streaming jobs and Spark
gcloud dataflow jobs drain JOB_ID --region="$REGION"
gcloud dataproc clusters delete bookshelf-spark --region="$REGION" --quiet
gcloud beta data-fusion instances delete bookshelf-fusion --location="$REGION" --quiet

# Vertex AI: undeploy before deleting the endpoint and model
gcloud ai endpoints undeploy-model ENDPOINT_ID --region="$REGION" --deployed-model-id=DEPLOYED_MODEL_ID
gcloud ai endpoints delete ENDPOINT_ID --region="$REGION" --quiet
gcloud ai models delete MODEL_ID --region="$REGION" --quiet
gcloud workbench instances delete bookshelf-notebook --location=us-central1-a --quiet

# Subscriptions created on this page
gcloud pubsub subscriptions delete orders-dataflow orders-archive --quiet

# BigQuery tables and the dataset (-r removes contained tables)
bq rm -r -f -d "$PROJECT_ID:bookshelf"

Deleted BigQuery tables can be restored for the time travel window, and storage still bills during it, so a deleted dataset is not instantly free. Delete the Dataflow worker service account (gcloud iam service-accounts delete) and the Cloud Storage objects under tmp and models, which otherwise keep accruing storage cost. If you created everything in a throwaway project, deleting the project removes all of it (see Resource hierarchy and projects).

References: gcloud dataflow jobs drain, gcloud dataproc clusters delete and Delete datasets.

References

Related pages