Cloud Storage

A default bucket has a flat namespace: folders are only prefixes. Renaming a "folder" therefore means copying and deleting every object under it, which is slow and not atomic. A bucket created with hierarchical namespace has real folder resources, atomic folder renames and higher read/write rates for file-oriented workloads such as analytics and AI training. It requires uniform bucket-level access, can only be chosen when the bucket is created, and is required by Rapid Buckets. Bookshelf’s two buckets stay flat because their access is by whole object name.

Separate from that, managed folders are IAM-bearing prefixes on a flat bucket; they are covered with access control in Cloud Storage Security and Static Sites.

The location is chosen at creation and cannot be changed afterwards (to move data you copy it to a new bucket). There are four kinds.

Derived from the Cloud Storage locations and availability and durability pages.

Derived from Cloud Storage quotas and limits; limits change, so check the page for the current numbers.

gcloud storage is the current command group for Cloud Storage. Its sub-groups are buckets, objects, folders, managed-folders and operations, plus the shortcut verbs cp, mv, rm, rsync, ls, cat and du. It replaces gsutil, which is covered in gsutil is legacy.

Derived from gcloud storage buckets create, gcloud storage buckets update and Create buckets.

Other creation variants use the same command with different flags.

Derived from Cloud Storage locations, Use turbo replication and Create and manage buckets with hierarchical namespace.

A zonal Rapid Bucket is created with a zone placement and the RAPID default class (flags are listed under gcloud storage buckets create --help); see Rapid Bucket for the current recipe, since the feature is new. Organizations created on or after 2024-05-03 enforce uniform bucket-level access and public access prevention by default through secure-by-default organization policies; older guides that create a bucket and then run gsutil acl ch -u AllUsers:R fail there, and the right pattern is in the security page.

Derived from gcloud storage reference, Upload objects and Copy, rename, and move objects.

rsync with the delete flag makes the destination match the source, so run it with --dry-run first. Wildcards such as gs://$BUCKET/covers/** work and are expanded by the tool, not the shell, so quote them. gcloud storage returns a non-zero exit code when any object fails, which makes it safe in scripts.

Every object has a generation that changes on each overwrite. --if-generation-match=0 means "create only if absent", which gives optimistic concurrency without locks; a violated precondition returns 412. See Request preconditions.

gcloud storage splits work across processes and threads on its own, so the -m option and .boto tuning that older gsutil guides recommend are unnecessary. For a single large file it uses parallel composite uploads: the file is uploaded as several temporary component objects in parallel and then composed into the final object, which is automatically enabled above a threshold (150 MiB by default). Smaller files use a resumable upload, which sends data in chunks and can continue after a network failure from the last confirmed byte, using a session URI that stays valid for a week.

Derived from Parallel composite uploads, Resumable uploads and gcloud topic configurations.

A composite object has a CRC32C checksum but no MD5 hash, so a client that insists on MD5 verification cannot validate it. The temporary component objects are deleted after the compose, and if an upload is interrupted they remain visible until you delete them or a lifecycle rule removes them. Parallel composite uploads need permission to create and delete objects in the bucket, so a role with only create rights is not enough.

Older guides use gsutil cp, gsutil -m rsync and gsutil lifecycle set. gsutil is now a legacy, minimally maintained tool: it does not understand newer features such as soft delete restores and managed folders, and after March 2027 it will no longer be bundled with the gcloud CLI (it will remain installable from PyPI). The current approach is gcloud storage for everything in this page. Most commands map directly (gsutil cp to gcloud storage cp, gsutil ls -l to gcloud storage ls --long, gsutil lifecycle set to gcloud storage buckets update --lifecycle-file), but output formats and some flags differ, so re-test scripts rather than aliasing the old name. The bq and kubectl companions are unaffected.

See Transition from gsutil to gcloud storage for the command-by-command table and the support notice.

For bulk movement between systems use Storage Transfer Service (STS) instead of looping cp. It is a managed, parallel service that copies from Amazon S3, Azure Blob Storage, other Cloud Storage buckets, public HTTP(S) URL lists or on-premises file systems (through agents you install), can run once or on a schedule, preserves metadata, verifies checksums, retries and can delete the source after a successful copy. Event-driven transfers start a copy when new objects arrive in an S3 bucket (through SQS) or a Cloud Storage bucket (through Pub/Sub). Check the Storage Transfer Service pricing page for what is charged beyond the source’s egress and operations.

In the console open Data transfer → Storage Transfer Service, choose Create transfer job, then the source, the destination and when to run it.

Derived from Storage Transfer Service overview, gcloud transfer jobs create and event-driven transfers.

For on-premises sources create an agent pool (gcloud transfer agent-pools create) and run the agent in a container on a host that can reach the data; the agent only makes outbound HTTPS connections. For tens or hundreds of terabytes over a slow link, Transfer Appliance is a rack-mountable device Google ships to you, which you fill and send back for upload. Small, occasional copies are better done with gcloud storage cp. To move data in the other direction, gcloud storage cp and rsync also accept s3:// URLs when AWS credentials are configured, which suits one-off checks but not bulk work.

Autoclass manages the class for you. When it is enabled on a bucket, every new object starts in Standard; objects that have not been read or written move to Nearline after 30 days, Coldline after 90 days and (optionally) Archive after 365 days, and an object that is read moves back to Standard. Autoclass charges no retrieval fees or early deletion fees for the moves it makes, and instead bills a small management fee per object, so it pays off for large buckets whose access pattern is unknown or changes, and not for buckets where you know exactly when data goes cold (a lifecycle rule is cheaper and more predictable there). Objects smaller than 128 KiB are not moved. You can enable it on an existing bucket, but a bucket with Autoclass cannot also have lifecycle rules that set storage classes.

Derived from Autoclass and Using Autoclass.

Soft delete is on by default for new buckets since March 2024: a deleted or overwritten object (and a deleted bucket’s contents) is kept for the retention duration, 7 days by default and configurable between 7 and 90 days (0 turns it off), and can be restored. Soft-deleted bytes are billed as storage in their original class until they expire, so a bucket holding mostly temporary data (build caches, scratch files) can see its bill rise unexpectedly; turn soft delete off or shorten it there. The setting is per bucket, with a project default that can be set through organization policy.

Derived from Soft delete, Use soft delete and gcloud storage restore.

A restored object becomes a new live object in the same bucket and, as the page above explains, in the Standard class whatever it used before. In the console, use the bucket’s Objects tab and its Soft deleted filter, select the objects and choose Restore. Soft delete is not a backup against someone with delete rights who also disables it; combine it with the retention controls in the security page for regulated data.

Object versioning keeps every overwritten or deleted object as a noncurrent version identified by its generation number. Reading a name returns the live version; reading name#<generation> returns a specific one. It protects against bad overwrites by an application, which soft delete also does, but it keeps versions indefinitely (until lifecycle rules remove them) and exposes them for listing and restore without a time limit. Enable it for data you cannot regenerate, and always pair it with a lifecycle rule such as numNewerVersions so noncurrent versions do not accumulate.

Derived from Object versioning and Use object versioning.

Bucket retention policies with Bucket Lock, Object Retention Lock and holds block deletion even by owners and are irreversible once locked; they are in Cloud Storage Security and Static Sites. A locked policy also stops you deleting the bucket during the retention period.

Derived from Cloud Storage FUSE overview, Install Cloud Storage FUSE and caching.

The VM’s attached service account needs a role such as roles/storage.objectUser on the bucket (to read and write; use roles/storage.objectViewer for read-only), and --implicit-dirs makes prefixes with no placeholder object visible as directories.

The GKE CSI driver mounts a bucket into a Pod from a csi volume and works with Workload Identity Federation for GKE; it is enabled with --addons=GcsFuseCsiDriver on Standard clusters and is on by default in Autopilot. Cloud Run mounts a bucket with two flags.

On GKE, annotate the Pod with gke-gcsfuse/volumes: "true" and declare a csi volume with driver gcsfuse.csi.storage.gke.io and bucketName in its volumeAttributes.

Derived from Cloud Run — Cloud Storage volume mounts and GKE — Cloud Storage FUSE CSI driver.

Import the BOM once and add the artifact without a version.

Derived from Cloud Storage client libraries — Java and the Java API reference. Signed URLs for letting a browser upload directly to the bucket are in the security page.

Spring Cloud GCP Storage exposes buckets as Spring Resource objects with the gs:// protocol, so existing resource loading and WritableResource code works unchanged. Use Spring Cloud GCP 8.x with Spring Boot 4 and 7.4.x with Spring Boot 3.

Derived from the Spring Cloud GCP — Cloud Storage reference. Testing against fake-gcs-server, signed URLs and the multi-cloud comparison are in File Storage & Object Stores.

pip install google-cloud-storage, then storage.Client().bucket(name).blob("covers/x.jpg").upload_from_filename("cover.jpg", if_generation_match=0) uses ADC the same way; see the Python client documentation.