Object storage (S3-compatible)

Another way to store files

We're used to keeping files in a file system, from the lesson on FHS: a directory tree, permissions, a path. That works well for a server, but it scales badly to terabytes and millions of files reachable from different machines. That's what object storage is for.

The reference here is Amazon S3, and its API became the de facto standard. Yandex Object Storage, Selectel, MinIO, and many others are "S3-compatible": you talk to them with the same requests and the same tools.

Bucket and object

Object storage has no directory tree. There are two concepts:

• A bucket is a named container for objects, the top level of the storage.
• An object is a file's data plus metadata, stored in a bucket under its own key. A key looks like a path (backups/2026-06/db.dump), but it's just an identifier string, not nested folders.

bucket: company-backups
  key: db/2026-06-18.dump   → object (data + metadata)
  key: static/logo.png      → object

AWS states directly that a bucket has a flat structure, not a hierarchy like a file system. The "folders" the web console shows are assembled from a shared start of keys, that is, from a prefix. For example, db/ is the prefix of key db/2026-06-18.dump.

How you talk to it

Access to an object happens over HTTP through an API, not by mounting a disk. Putting an object means sending PUT, getting it means GET, deleting it means DELETE. Usually a CLI like aws s3 or mc, or an SDK in application code, makes these requests, but under the hood it's still HTTP.

An object's address is made of the storage address, the bucket name, and the key:

PUT    https://<storage>/company-backups/db/2026-06-18.dump   put
GET    https://<storage>/company-backups/db/2026-06-18.dump   get
DELETE https://<storage>/company-backups/db/2026-06-18.dump   delete
GET    https://<storage>/company-backups?list-type=2&prefix=db/   list keys with prefix db/

The list of objects comes back as XML. If you add delimiter=/ to the request, the storage groups keys by the next / and returns the groups in CommonPrefixes blocks. That's how the console draws "folders".

In a real cloud, every such request is signed with an access key and a secret key, and the CLI or SDK computes that signature for you. That's why you don't usually hit AWS with bare curl. This lesson's labs use a training storage without signatures, so the requests stay fully visible.

How it differs from a file system

These differences are exactly why people choose object storage:

• Scale. The volume and number of objects are nearly unlimited, and you don't have to think about disk size. You pay for what's stored and for requests.
• HTTP access from any machine. An object is reached by address, not from a specific server, which suits distributed systems well. Access is closed by default at the same time: new buckets and objects in S3 aren't public, only a key with the right permissions can read them.
• An object changes only as a whole. Appending a line at the end or editing a chunk in the middle, like in a file, usually isn't possible. You download the object, change it, and upload it again under the same key, and the new version replaces the old one. There's an exception (appending in the S3 Express One Zone storage class), but that's a separate feature of one storage class, so don't count on it. That's why object storage suits whole files (a backup, an image, an archive), not databases that need in-place writes.

Where this shows up at work

Object storage covers a few tasks you already know:

• backups, from the lesson on backup strategies: an off-site copy in a bucket in another region satisfies the 3-2-1 requirement;
• CI artifacts and Terraform state from earlier lessons are often kept in exactly this kind of S3-compatible bucket;
• app static files and uploads (images, user files): these live in a bucket rather than on the server's disk, so the app stays stateless, as in the lesson on 12-factor apps.

Access to a bucket is granted on the principle of least privilege: a key with permissions for just the bucket you need (or even just one prefix), not the whole storage.

Common mistakes

• "I deleted the folder, but the files are still there." There's no folder, there's a prefix. A DELETE request on logs/ removes only the object with the key exactly logs/, if one even exists. Objects logs/app.log and logs/web.log are different keys, and you delete them one at a time. A CLI loops through this for you: aws s3 rm has a --recursive flag.
• A prefix without a slash. Prefix logs catches both logs/app.log and logs-archive.zip. If you mean a "folder", write the prefix with a trailing slash: logs/.
• Trying to append to an object. A repeated PUT under the same key doesn't append, it replaces the object. To add a line, download the whole object first.
• Uploaded and didn't check. A 200 response code on PUT says the write went through. A backup counts as working once you've downloaded it and compared it with the original.
• A key with permissions for everything. A leaked "whole storage" key hands over all buckets at once. A narrow key for one bucket limits the damage.
• A bucket name "like everyone else's". Bucket names in S3 are shared across all accounts within an AWS partition, so backups is most likely taken. Add your project or company name to the name.

In short

• An object's key is a string that looks like a path. There are no real folders: a "folder" means a shared key prefix.
• An object changes only as a whole: you can't append a line or edit a chunk in the middle, only download, change, and upload again under the same key.
• There's no "delete folder" operation: cleaning up by prefix means deleting each object with that prefix in a separate request.
• Scale and access from any machine are nearly unlimited, but new buckets and objects are closed by default.
• Access is granted with a narrow key for one bucket or prefix, not the whole storage.

What you'll need for the tasks

• an object's address: http://127.0.0.1:PORT/<bucket>/<key>;
• create a bucket: curl -X PUT http://127.0.0.1:PORT/<bucket>;
• put a file as an object: curl -X PUT --data-binary @<file> <object address>;
• get an object: curl -o <file> <object address>;
• delete an object: curl -X DELETE <object address>;
• list keys: curl '<bucket address>?list-type=2', filtered with &prefix=..., grouped with &delimiter=/;
• response code: curl -s -o /dev/null -w '%{http_code}\n' ...;
• compare two files: cmp, pull keys out of XML: grep -o '<Key>[^<]*</Key>'.

The next lesson covers cloud models: IaaS, PaaS, and SaaS.