Spanner, Firestore and Bigtable

A Spanner instance is an allocation of compute and replication placement; it contains one or more databases. Three choices define an instance:

Small PU counts make Spanner usable for development and small production systems, and a 90-day free trial instance lets you try it without a charge (one per project, limited to 10 GB of storage, with a Standard-like feature set; see the free trial page for the current terms). Two dialects exist per database, chosen at creation and fixed afterwards: GoogleSQL and the PostgreSQL interface, which lets PostgreSQL drivers, syntax and many tools work against Spanner. They are the same engine; the dialect only changes how you write schema and queries.

You pay for allocated compute whether it is busy or not, so size to the workload. There are two ways:

Watch the high-priority CPU utilisation, the storage utilisation and the lock wait time in Cloud Monitoring (see Monitoring and Observability); sustained CPU above the recommended target is the signal to add capacity, and a low CPU with high latency usually points at schema or hotspot problems, not capacity.

Derived from Create and manage instances, Managed autoscaler and the gcloud spanner instances create reference.

Spanner stores every table sorted by its primary key and divides the key space into splits that move between servers to balance load. That has two consequences for design.

Avoid hotspots. A key that always increases (a timestamp, an auto-incremented number) sends every new row to the last split, and one server becomes the bottleneck whatever the instance size. Use a UUID v4 or another high-cardinality random value, a bit-reversed sequence (Spanner’s bit_reversed_positive sequences), or lead with a well-distributed column such as the customer and put the timestamp second. Never use a monotonically increasing value as the first key part of a write-heavy table. Indexes follow the same rule: an index on a timestamp column has the same hotspot as a timestamp key.

Interleave parent and child tables. Spanner can store the rows of a child table physically inside their parent row. Interleaving means that reading a customer with its orders touches one split, and that writes to both in one transaction are cheap. There are two forms, and older guides only show the first:

Use interleaving for tightly owned data (an order and its lines); use a plain foreign key for loosely related data. A very large parent with millions of interleaved children also forms a large key range, so interleave only what is read together. Secondary indexes can be interleaved in a table for the same locality reasons. The schema below is the Bookshelf order model in GoogleSQL.

Create the database with the DDL in one call (the statements are separate elements of the --ddl list, or a file passed with --ddl-file):

Derived from Schema and data model, Schema design best practices and Data definition language. In the PostgreSQL dialect the same ideas apply with INTERLEAVE IN PARENT in the CREATE TABLE statement and PostgreSQL types.

Spanner reads are strong by default and can be stale on request (a timestamp bound or exact staleness), which lets a replica answer locally and is the cheapest way to serve read-heavy pages that tolerate a few seconds of lag. Writes are grouped in read-write transactions (locks, retried automatically by the client libraries when aborted) or sent as mutations, a batch of inserts and updates applied atomically, which is more efficient than SQL for bulk changes. Partitioned DML runs large updates and deletes outside a single transaction, and change streams publish row changes to Dataflow or Pub/Sub for downstream consumers. Use SELECT …​ FROM Orders@{FORCE_INDEX=OrdersByStatus} only when the optimiser picks the wrong index; the query plan in the console shows what it picked.

Enterprise and higher editions add features that remove the need for a separate store for some workloads: Spanner Graph (property graphs queried with GQL, on the same tables), full-text search (search indexes with tokenisation and ranking) and vector search (approximate and exact nearest-neighbour search over embeddings stored in a column). For Bookshelf, a vector column on the catalog would give "books similar to this one"; the comparison with other vector stores is in Comparing Vector Databases. These features are newer than most guides and some are still Preview or limited to certain editions; check the release notes for the current launch stage of each.

Spanner uses IAM at the project, instance and database level, and optional fine-grained access control with database roles inside a database. Common predefined roles:

Grant the application’s service account the narrowest role on the database rather than on the project:

Derived from Spanner IAM and Fine-grained access control. Data is encrypted at rest with Google-managed keys by default, with customer-managed keys (CMEK) available per database (see Secret Manager and Cloud KMS). Reach Spanner from private networks over its normal Google API endpoints; private connectivity options are described in VPC Networking on Google Cloud.

Applications authenticate with Application Default Credentials, using the service account attached to Cloud Run or the Workload Identity Federation for GKE identity; locally use gcloud auth application-default login or service account impersonation, and never a downloaded key. The Java client library gives the fullest control:

Derived from the Spanner client libraries and Java reference. Add com.google.cloud:google-cloud-spanner with the libraries-bom so versions stay aligned. Three other routes reuse existing code:

For local development and tests, run the emulator: gcloud emulators spanner start (or the container image) and set SPANNER_EMULATOR_HOST=localhost:9010; the client libraries then skip authentication. The emulator does not reproduce performance or every feature, so keep a pre-production test on a real small instance.

Spanner is billed mainly on compute capacity (processing units or nodes, per hour, by edition and configuration), storage (the data and backups), network egress and, for some features, usage-based charges. Cost drivers to watch: an oversized instance left running, multi-region configurations (more replicas and more compute), long backup retention, and change streams or analytics reading heavily. Use the autoscaler or reduce PUs outside business hours for development instances, and delete throwaway instances. There is a free trial instance; there is no permanent free tier. Prices change, so use the pricing page and the pricing calculator rather than numbers in a guide. Limits (databases per instance, tables per database, mutations per commit, row and key size, sessions) are listed in Quotas and limits; hitting a limit is a design signal, not just a quota request.

Spanner offers point-in-time recovery (versions retained for 1 hour by default, up to 7 days with version_retention_period), backups (on demand or scheduled, retained up to the documented maximum) and, for Enterprise Plus, additional recovery options. Set the retention you need explicitly:

Derived from Point-in-time recovery, Backups and gcloud spanner backups create.

Firestore has two editions, and a project can hold several databases (--database), including one named (default):

The mode (Native or Datastore) is chosen when the database is created and cannot be changed afterwards; a project’s default database in Datastore mode cannot also use Native mode, which is why multiple databases are useful. Locations are regional (for example us-central1) or multi-region (nam5, eur3); they cannot be changed after creation. Older guides say a project may have only one Firestore database and name the product "Cloud Firestore" next to "Cloud Datastore"; both statements are out of date.

Do not confuse Firestore with Filestore, which is a managed NFS file share (see Block and File Storage on Google Cloud).

A document is a set of typed fields (strings, numbers, booleans, timestamps, maps, arrays, references, geo points, vectors); a collection is a container of documents; a document can contain subcollections. Documents are limited to roughly 1 MiB and field and nesting depths are limited too, so large blobs belong in Cloud Storage (Cloud Storage) with a reference stored in the document. Reads address a document by path (orders/o-1001) or query a collection; there are no joins, so data is denormalised: store what a screen needs together, for example the order lines inside the order document, and keep copies in sync from your code or with a function.

Indexing is automatic and central to cost and speed:

Derived from Create and manage databases, Editions and the gcloud firestore databases create reference. Delete protection prevents accidental deletion until you turn it off with gcloud firestore databases update --no-delete-protection.

Derived from Manage data retention with TTL, Schedule and manage backups and Point-in-time recovery. Replace BACKUP_ID with an identifier from gcloud firestore backups list.

Firestore has two separate access paths and it is important not to mix them up.

A rule set is a file (firestore.rules) that matches document paths and states conditions. The classic pattern is "a user owns their own documents", using the authenticated user ID request.auth.uid:

Deploy the rules with the Firebase CLI (firebase deploy --only firestore:rules) after firebase init firestore, or in the console under Firestore → Rules. The Rules Playground tests a request without deploying, and the emulator runs the rules in unit tests. Treat rules as code: review them, test them with the @firebase/rules-unit-testing package, and never leave allow read, write: if true in a project with real data. Rules are not filters: a query must be written so that every possible result satisfies the rule, otherwise it fails as a whole. Security rules apply to Native mode only; Datastore mode and MongoDB compatibility use IAM. Derived from Get started with Firestore security rules, rules conditions and the Firestore IAM reference.

Native-mode Firestore supports vector fields and k-nearest-neighbour queries over them with a vector index, so a small semantic search ("books like this one") can live next to the data. It has limits on dimensions and on the number of results; for large or specialised workloads see Comparing Vector Databases. See the vector search guide.

Server code uses the Firestore client library with Application Default Credentials. The database ID is set on the options builder, because a project can hold several databases:

Derived from the Firestore server client libraries guide. The composite index created earlier is what the latestFor query needs. With Spring, the Spring Cloud GCP Firestore starter (spring-cloud-gcp-starter-firestore) provides reactive repositories and FirestoreReactiveOperations, configured with spring.cloud.gcp.firestore.database-id; see the Spring Cloud GCP Firestore reference and, for the MongoDB-compatible alternative that works with Spring Data MongoDB, the next page in this section.

For development and tests use the Firestore emulator (gcloud emulators firestore start, then FIRESTORE_EMULATOR_HOST=localhost:8080); it also runs security rules through the Firebase Emulator Suite.

Firestore bills per document read, write and delete, storage (documents, indexes and, if enabled, PITR and backups), network egress and some feature usage. Cost drivers: queries that read many documents to show a few, unneeded indexes on every field, real-time listeners that fan out large result sets, and long PITR windows. There is a free quota per day for the (default) database (a documented number of reads, writes and deletes and some storage), which makes it cheap for prototypes; check the pricing page for the current figures. Limits to design around are the sustained write rate to a single document (about once per second), index entries per document, document size and the maximum transaction duration; see quotas and limits and pricing.

Bigtable stores rows in sorted order by row key and splits the table into tablets that move between nodes. A good key is one you can scan by prefix and that spreads load across the whole key space. Avoid keys that start with a timestamp or a sequential number, which send all writes to one node (the same hotspot problem as in Spanner), and avoid very long keys. A common time-series pattern is entity#reversed-timestamp, for example the ISBN followed by a descending timestamp, so the newest price for a book is the first row of its prefix. Group columns that are read together in the same family and keep the number of families small; use garbage-collection policies (maximum age or maximum number of versions) to bound storage.

Derived from Create an instance, Autoscaling and the cbt tool. The gcloud components install command is only available when the CLI was installed from the Google package rather than a system package manager.

Bigtable supports GoogleSQL for Bigtable (SQL queries on tables, with the same functions as BigQuery where they apply), alongside the native ReadRows and MutateRows API. Clients are the Cloud Bigtable client libraries (Java: BigtableDataClient), the HBase client (bigtable-hbase) for applications written against Apache HBase, Dataflow, Dataproc and BigQuery external tables. There is no Spring Data module for Bigtable, so use the client library from a Spring bean. IAM roles include roles/bigtable.admin, roles/bigtable.user (read and write data), roles/bigtable.reader and roles/bigtable.viewer; give the application roles/bigtable.user on the instance. The emulator is started with gcloud emulators bigtable start.

Derived from the Bigtable client libraries. In a real application create one long-lived client and reuse it; opening one per call as above only keeps the sample short.

Choose Bigtable for time series, IoT and telemetry, monitoring data, ad and recommendation features, and any workload with hundreds of thousands of operations per second addressed by a well-designed key, with large data sizes (terabytes and up). Do not choose it for small datasets (a single node is a fixed monthly cost regardless of use), relational queries, multi-row transactions or ad hoc filtering on columns. Billing is on node hours (per cluster), storage (SSD or HDD, plus backups) and network egress; the cheapest way to cut cost is autoscaling and removing clusters you do not need. See pricing and quotas and limits.