S3 object storage explained: buckets, keys and endpoints

Object storage is a big flat key-value store you talk to over HTTP. You put a whole file under a name in a bucket, and later get the whole file back by that name. There are no directories, no appends and no editing in place - change one byte and you upload the object again. S3 is Amazon's API for doing that, and it became the standard: "S3-compatible" storage is any service that answers the same HTTP requests, so the AWS CLI, rclone, restic, Cyberduck and every S3 library work against it by changing one setting, the endpoint. That model is exactly right for backups, uploaded media, static assets and archives, and wrong for databases and anything a program edits constantly. This post explains the model properly, so the tools in the rest of this series make sense.

Buckets, objects and keys#

Three nouns cover the whole data model.

• A bucket is a named container. Every object lives in exactly one bucket. Credentials and settings usually apply per bucket.
• An object is the data - any bytes, from zero to very large - plus metadata about it.
• A key is the object's name within the bucket. backups/2026-10-08/world.tar.gz is one key; the slashes are just characters in it.

On Amazon S3, bucket names are 3 to 63 characters of lowercase letters, digits, hyphens and dots, starting and ending with a letter or digit, and not shaped like an IP address. S3-compatible servers often accept more, but sticking to lowercase letters, digits and hyphens keeps a bucket portable and avoids certificate trouble with dots later. Keys can be up to 1,024 bytes of UTF-8 and may contain almost anything, though spaces, +, # and non-ASCII characters invite URL-encoding bugs in less careful tools.

Folders are an illusion

The namespace inside a bucket is flat. What tools show as folders is a listing trick: ask for keys with the prefix backups/ and the delimiter /, and the server returns the objects directly under that prefix plus a list of the next-level "common prefixes" - backups/2026-10-07/, backups/2026-10-08/ - which clients draw as folders.

bucket: renode-backups  backups/2026-10-07/world.tar.gz  backups/2026-10-07/db.sql.gz  backups/2026-10-08/world.tar.gz  media/avatars/u123.png

Consequences that follow directly:

• An "empty folder" does not exist unless some tool created a zero-byte object ending in / to fake one.
• Renaming a folder means copying and deleting every object under its prefix. On a million objects, that is two million requests.
• Listing is paginated - S3's ListObjectsV2 returns at most 1,000 keys per request - so counting a big bucket takes many round trips.

What an object really is#

An object is immutable bytes plus metadata. The metadata worth knowing:

Metadata is set when the object is written. Changing it means copying the object onto itself with new metadata - another full write as far as the server is concerned.

Large objects are uploaded in parts. A multipart upload starts, sends parts (on Amazon, each 5 MiB to 5 GiB except the last, up to 10,000 parts), and completes; the server then assembles them into one object. Parts upload in parallel and a failed part is retried alone, which is why every serious tool switches to multipart above a threshold. The ETag of a multipart object is not the MD5 of the file - it ends in - and the number of parts - so tools that verify uploads by ETag need to know which kind they made. On Amazon a single PUT is limited to 5 GiB and an object to 5 TiB; other implementations set their own limits.

An unfinished multipart upload leaves its parts stored until it is completed or aborted. Tools normally clean up after themselves, but a crashed upload can leave parts behind that use space without appearing as an object. aws s3api list-multipart-uploads --bucket NAME shows them.

The API is just HTTP#

Every operation is an HTTP request to the endpoint. That is why S3 is so easy to support and why it works through ordinary proxies and firewalls.

Ranged GETs are what make object storage usable for video streaming and for backup tools like restic, which read small pieces of large pack files. Server-side copy is what makes "rename" possible without downloading.

Credentials and signatures

You get two strings: an access key ID, which identifies you and is not secret, and a secret access key, which is. The secret is never sent over the network. Instead, the client signs each request with Signature Version 4: it builds a canonical description of the request (method, path, query, selected headers, a hash of the body), and computes an HMAC chain from the secret, the date, the region and the service name. The server recomputes the signature with its copy of the secret and rejects the request if they differ.

Two practical results:

• The region is part of the signature. Client and server must agree on it. Amazon has real regions; an S3-compatible endpoint may accept any name. Pick one - us-east-1 is the conventional placeholder - and use the same one in every tool.
• Time matters. The signature includes a timestamp, and Amazon rejects requests more than 15 minutes away from its clock with RequestTimeTooSkewed. A server with a broken clock produces baffling signature errors; check date before anything else.

Signing also enables presigned URLs: a link with the signature in the query string, valid for a limited time, that lets someone without credentials upload or download one specific object. S3 presigned URLs for uploads covers them.

Endpoints, addressing and HTTPS#

The endpoint is the base URL of the service. Requests name the bucket in one of two ways:

• Path-style: https://s3.example.com/my-bucket/photos/cat.jpg - the bucket is the first path segment.
• Virtual-hosted style: https://my-bucket.s3.example.com/photos/cat.jpg - the bucket is a subdomain.

Amazon prefers virtual-hosted style; many S3-compatible servers, especially single-tenant ones on one host name, use path-style because it needs no wildcard DNS or wildcard certificate. Most tools default to virtual-hosted and need one setting to switch. Path-style vs virtual-hosted S3 goes into the reasons.

Use HTTPS for anything that crosses the internet. A SigV4 signature protects the secret key and stops tampering with the signed parts of a request, but over plain HTTP the object data, the key names and your access key ID are readable by anyone on the path. On RE:NODE the endpoint answers plain HTTP on its port, and each plan carries a proxy slot: point a hostname's A record at it and the certificate is issued and renewed automatically, so clients connect over HTTPS to your own name.

What "S3-compatible" does and does not promise#

The core of the S3 API - buckets, PUT, GET, HEAD, DELETE, listing, multipart uploads, copy, presigned URLs - is implemented consistently by nearly every compatible server, and it is all that backup and sync tools need. Amazon's S3 also has a long tail of features that sit on top: versioning, lifecycle rules that expire or move objects, object lock for write-once retention, bucket policies and IAM, storage classes such as Glacier, event notifications, replication and several flavours of server-side encryption. Compatible servers implement varying subsets of that tail, or none of it.

So "S3-compatible" means "your S3 tools will connect and move data", not "every feature in the AWS documentation exists". Before you design around a feature, check that the service you use offers it.

The RE:NODE storage line is described in terms of the core: an S3-compatible endpoint, an access key and secret key generated per server, a first bucket created for you, path-style addressing, any region name, and it works with the AWS CLI, rclone, Cyberduck, restic and S3 drivers in applications. Versioning, lifecycle rules, object lock and bucket policies are not part of that description, so do not plan on them. Retention for backups belongs in the backup tool - restic's forget policy, for example - which has the side benefit of working the same against any S3 endpoint you later move to.

Then there is consistency. Since December 2020 Amazon S3 guarantees strong read-after-write consistency: once a PUT returns, every GET and listing sees it. Most single-node compatible servers behave that way naturally, but it is a property of the implementation, not of the API, and tools written before 2020 are built to tolerate less.

What object storage is good for, and what it is not#

Good fits:

• Backups and archives. Written once, read rarely, by tools designed for it. Restic backups to S3 and rclone with S3 storage cover the two main tools.
• User uploads and media. Avatars, attachments, images. The application stores a key in its database and serves the file through a URL; the app server's disk stays small and the uploads survive a redeploy. S3 from Node and Python shows the code, and WordPress media offload to S3 the CMS version.
• Static assets and build artefacts. Versioned file names, written once.
• Logs and exports. Append-only data rotated into new objects.

Poor fits:

• Databases. A database writes small pieces of large files constantly. Object storage can only replace whole objects. Store database dumps in it, not databases.
• Files edited in place. A spreadsheet saved every minute is a full upload every minute.
• Millions of tiny files. Every object costs a request and some metadata. Pack small files into archives, which is exactly what backup tools do.
• A drop-in filesystem. Mounting a bucket with rclone mount or similar works for browsing and occasional copies, but applications that expect POSIX semantics - locks, renames, partial writes - behave badly on it.

Durability: one copy is still one copy#

Amazon advertises eleven nines of durability for S3 Standard because it stores every object redundantly across several facilities. That figure is a property of Amazon's storage design, not of the S3 API, and an S3-compatible service is exactly as durable as the disks and copies behind it.

RE:NODE's storage line keeps one copy of your data, on NVMe, in one location in Germany. It is not replicated. That makes it a good second location for backups - off the machine they protect, and in a format every backup tool speaks - and a poor only location. For anything you cannot afford to lose, follow the 3-2-1 rule: three copies, on two kinds of storage, one somewhere else entirely. S3 storage sizing and the 3-2-1 rule works through what that looks like, and backups that actually restore explains why the restore test matters more than the copy.

FAQ#

Is S3 the same thing as Amazon S3?

S3 began as Amazon's Simple Storage Service, and the name now also refers to its API, which many other providers and open-source servers implement. When people say "S3 storage" outside AWS they usually mean S3-compatible: the same requests, a different endpoint.

Can I edit part of a file in object storage?

No. Objects are replaced whole. You can read part of an object with a ranged GET, but writing means uploading the complete new version. Tools that appear to edit in place are downloading, changing and re-uploading.

What is the difference between object storage and block storage?

Block storage is a disk: the operating system puts a filesystem on it and reads and writes any byte. Object storage is a service: you send whole files over HTTP and address them by key. Databases and operating systems need block storage; backups and media are happier in object storage.

Why do I need a region if the server does not care?

Because the request signature includes it. The client uses the region to compute the signing key, and the server must use the same one to verify. On an endpoint that accepts any region name, choose one and configure every client with it.

Do I need HTTPS if the requests are signed?

Yes, for anything outside one private network. The signature protects your secret key and the integrity of signed parts, but not confidentiality: without TLS, the contents of your files and their names travel in plain text.