S3 Object Storage

S3 is an object store, not a file system. You create a bucket (a named container in one Region) and put objects into it. An object is data plus metadata, addressed by its key. The key covers/9780134685991.jpg is one flat string; the slash is only a convention. S3 has no real folders: the console shows covers/ as a folder because it groups keys by prefix and delimiter, and an empty "folder" is just a zero-byte object whose key ends in a slash.

Since December 2020 S3 is strongly consistent for all operations in every Region: after a successful PUT, DELETE or overwrite, the next GET or LIST sees the change. There is no eventual-consistency window to code around, which older guides still describe. Concurrent writers to the same key are still last-writer-wins unless you use versioning or conditional writes (below).

Object metadata comes in three kinds: system metadata such as Content-Type, Content-Length, ETag and storage class; user-defined metadata (x-amz-meta- headers, up to 2 KB); and *tags (up to 10 key-value pairs per object) that drive lifecycle filters, cost allocation and access control. Metadata is immutable once written; changing it means copying the object onto itself.

Every object has a storage class that sets its price per GB, its availability and its minimum duration. You pick the class at upload time (--storage-class) or move objects later with a lifecycle rule or a copy.

Standard-IA, One Zone-IA and Glacier Instant Retrieval bill a minimum of 128 KB per object, so they are a poor fit for small objects such as thumbnails. Glacier Flexible Retrieval and Deep Archive add about 40 KB of metadata per object (32 KB at the archive rate and 8 KB at the Standard rate). REDUCED_REDUNDANCY still exists in the API but AWS recommends against it; Standard is cheaper. Availability design targets and the full comparison are in the storage class overview.

A lifecycle configuration is a list of rules, each with a filter (prefix, tags, object size) and actions: transition to a cheaper class after N days, expire (delete) after N days, expire noncurrent versions, remove expired delete markers, and abort incomplete multipart uploads. S3 evaluates them once a day, so an action happens within a day or so of the due date. Transitions run only "downhill" (Standard to Standard-IA after at least 30 days, and so on), and by default objects smaller than 128 KB are not transitioned to the IA and Glacier Instant classes.

The rule set below is what Bookshelf applies to the uploads bucket: temporary uploads expire after a week, uploads move to Standard-IA after 30 days and Glacier Instant Retrieval after 90, old versions are removed after 90 days, and failed multipart uploads are cleaned up after 7 days.

Derived from S3 User Guide — Managing the lifecycle of objects and aws s3api put-bucket-lifecycle-configuration.

In the console, open the bucket, choose Management → Lifecycle rules → Create lifecycle rule, scope it by prefix or tag, tick the actions (Transition current versions, Expire current versions, Permanently delete noncurrent versions, Delete expired object delete markers or incomplete multipart uploads) and enter the days; the console warns about costs when a rule moves many small objects. A lifecycle transition is a billed request, and objects deleted or transitioned before a class’s minimum duration are still charged for the full minimum. Do not stack a Standard-IA transition on objects that will be deleted within 30 days.

Where the choice is not obvious, S3 Storage Class Analysis observes access patterns for a prefix and suggests when to transition from Standard to Standard-IA, and Intelligent-Tiering removes the guesswork altogether at the cost of the monitoring fee.

With versioning on, every PUT creates a new version with its own version ID, and a DELETE without a version ID only adds a delete marker; the previous versions stay and are billed. That makes accidental overwrites and deletions recoverable and is a prerequisite for replication and Object Lock. Versioning can be suspended but never turned fully off. A bucket is in one of three states: unversioned, enabled or suspended.

Derived from S3 User Guide — Retaining multiple versions of objects with S3 Versioning.

MFA delete additionally requires a one-time code from the bucket owner’s root-account MFA device to change the versioning state or permanently delete a version. It can only be enabled by the root user with the CLI or API, which is why most teams rely on IAM policies, Object Lock and lifecycle rules (see the security page) instead.

Replication copies new objects automatically and asynchronously to another bucket. Cross-Region Replication (CRR) targets a bucket in another Region (disaster recovery, lower latency for remote users, compliance); Same-Region Replication (SRR) targets a bucket in the same Region (log aggregation, separating production and test data, a copy in a different account). Both buckets need versioning enabled, S3 needs an IAM role it can assume to read the source and write the destination, and only objects created after the rule exists are replicated by default. Existing objects are copied with S3 Batch Replication, which can also retry objects whose replication failed.

S3 Replication Time Control (RTC) adds a service-level objective: 99.99 percent of objects replicated within 15 minutes, with metrics and event notifications for the exceptions. It is billed per GB replicated on top of normal replication charges. Delete markers are replicated only if you enable DeleteMarkerReplication, and deleting a specific version is never replicated, so a malicious delete in the source does not propagate.

Derived from S3 User Guide — Replicating objects within and across Regions and aws s3api put-bucket-replication.

The unquoted JSON heredoc delimiter above lets the shell expand the two variables. The role’s trust policy lets s3.amazonaws.com assume it, and its permissions policy needs the following; replace the bucket names with yours.

Derived from S3 User Guide — Setting up permissions for replication. If either bucket uses SSE-KMS the role also needs kms:Decrypt on the source key and kms:Encrypt on the destination key, covered in the security page.

In the console: Management → Replication rules → Create replication rule, choose the scope, the destination bucket (this account or another), let the console create the IAM role, and tick Replication Time Control and Replication metrics if you need them. The console offers to run a one-time Batch Replication job for existing objects.

A bucket can emit events (s3:ObjectCreated:*, s3:ObjectRemoved:*, s3:Replication:*, lifecycle and restore events) to a Lambda function, an SQS queue, an SNS topic or Amazon EventBridge. Bookshelf’s thumbnailer Lambda function runs on s3:ObjectCreated:* for the covers/ prefix (see AWS Lambda and Lambda: Event-Driven and Serverless Apps for the function and its execution role). Notifications are delivered at least once, so the handler must be idempotent.

Derived from S3 User Guide — Amazon S3 Event Notifications and aws s3api put-bucket-notification-configuration. Do not write results back to the same prefix a notification watches, or the function triggers itself in a loop. EventBridge delivery (aws s3api put-bucket-notification-configuration with an an empty EventBridgeConfiguration object) adds filtering, multiple targets and replay at slightly higher latency.

A presigned URL is an ordinary S3 URL with a signature, expiry and the signer’s permissions baked into the query string. Whoever holds it can perform exactly one operation on exactly one object until it expires, without AWS credentials. Bookshelf uses them so that a browser uploads a cover directly to S3 and a reader downloads a PDF without the catalog-api proxying the bytes: the API authenticates the user, then returns a short-lived URL.

The URL’s power equals the signer’s permissions at the time of use, and it stops working when the signing credentials expire even if the URL’s own expiry is later. A URL signed with a role session’s temporary credentials is therefore valid for at most the remaining session lifetime (at most 12 hours for an assumed role, 6 hours for an instance profile, and the Lambda or task credential window in practice). To get the full 7 days that Signature Version 4 allows, sign with an IAM user’s long-term access keys, which is the opposite of best practice, so keep URLs to minutes.

The recommended pattern is the same as everywhere on AWS: no keys in code. The catalog-api runs with a task role (see ECS and Fargate) and the SDK finds its credentials through the default credential provider chain. The role needs only what the code uses; this policy grants the application read, write and list access to the uploads bucket and nothing else.

Derived from S3 User Guide — How Amazon S3 works with IAM and the Service Authorization Reference for Amazon S3. Bucket-level actions go on the bucket ARN and object-level actions on the /* ARN; mixing them up is the classic cause of AccessDenied. If the bucket uses SSE-KMS the role also needs kms:GenerateDataKey and kms:Decrypt on the key (see the security page). IAM policy evaluation itself is in Identity and Access.

S3 scales request rate per prefix: at least 3,500 PUT, COPY, POST or DELETE and 5,500 GET or HEAD requests per second for each prefix in a bucket, with no limit on the number of prefixes. Throughput ramps up gradually, and during the ramp you may see 503 Slow Down responses that the SDKs retry with backoff. Older guidance to randomize key prefixes with hashes no longer applies; spread hot traffic over several prefixes only when one prefix saturates.

Derived from S3 User Guide — Best practices design patterns: optimizing Amazon S3 performance and Mountpoint for Amazon S3.

S3 is a regional service reached over public endpoints (s3.<region>.amazonaws.com and <bucket>.s3.<region>.amazonaws.com) with TLS, and it supports IPv6 on dual-stack endpoints. Workloads in private subnets do not need an internet path: a gateway VPC endpoint for S3 adds a route-table entry that sends S3 traffic over the AWS network at no hourly or per-GB charge, and an endpoint policy can restrict which buckets and actions the VPC may use. An interface endpoint (AWS PrivateLink) additionally gives S3 private IP addresses reachable from on-premises networks over VPN or Direct Connect. A bucket policy can then require traffic to come through a specific endpoint with the aws:SourceVpce condition key (see the security page). Virtual networks and routing are covered in AWS Global Infrastructure for the geography and in the networking pages of this section.

Derived from aws ec2 create-vpc-endpoint and Gateway endpoints for Amazon S3. In the console use VPC → Endpoints → Create endpoint, filter for the service com.amazonaws.us-east-1.s3 of type Gateway, and select the route tables of your private subnets.

Derived from S3 User Guide — Assessing your storage activity and usage with S3 Storage Lens and aws s3api put-bucket-inventory-configuration. The destination bucket needs a bucket policy that lets S3 write the report.

Prices change and differ by Region, so this page lists the drivers rather than numbers; use the S3 pricing page and the AWS Pricing Calculator (pricing checked 2026-10-01).

The practical levers are an AbortIncompleteMultipartUpload rule on every bucket, noncurrent-version expiration on every versioned bucket, Intelligent-Tiering or lifecycle transitions for aging data, a gateway endpoint to keep traffic off the NAT gateway, and CloudFront in front of public content.

Derived from the S3 User Guide — Troubleshooting and the S3 API error code list.

Bookshelf buckets are versioned, so aws s3 rb --force is not enough: it deletes current objects (adding delete markers) but leaves every version and delete marker behind, and delete-bucket then fails with BucketNotEmpty. Delete in dependency order: replication and notification configuration first (so nothing writes while you clean), then versions and delete markers, then in-progress multipart uploads, then the bucket.

Derived from S3 User Guide — Deleting a bucket and aws s3api delete-objects.

In the console, select the bucket and choose Empty, confirm by typing permanently delete, then choose Delete and type the bucket name; Empty removes all versions and delete markers. If you used a lifecycle rule to expire everything, the bucket empties itself within a day or two. Also delete the replication IAM role, the Lambda permission statement, any gateway endpoint created only for this page, the Inventory destination bucket, and any S3 Tables or Vectors buckets you tried. Objects under Object Lock retention or legal hold cannot be deleted until the hold ends.