Using the AWS CLI with S3-compatible storage

The AWS CLI works with any S3-compatible storage once it knows three things Amazon would otherwise assume: the endpoint URL, that the bucket goes in the path rather than the host name, and a region to sign with. Put all three in a named profile and every aws s3 command works unchanged - aws s3 ls --profile renode lists your buckets, aws s3 sync ./backups s3://my-bucket/backups --profile renode mirrors a folder. Recent CLI versions add one more requirement for many non-Amazon servers: a setting that stops the CLI sending checksum headers the server does not understand. This post sets all of that up and then covers the commands you will actually use, with the flags that matter.

Install the CLI and check the version#

Use AWS CLI version 2. It is a self-contained installer for Windows, macOS and Linux from Amazon's documentation, and it does not depend on your system Python. Linux distribution packages are sometimes years old or still version 1, so check:

$ aws --versionaws-cli/2.27.50 Python/3.13.4 Linux/6.8.0 exe/x86_64.ubuntu.24

Anything from 2.13 onward supports endpoint_url in the configuration file, which is what makes a profile for a custom endpoint possible. Before that you had to pass --endpoint-url on every command. Version 1 also works with the flag, but it is the older line; there is no reason to start a new setup on it.

Credentials, endpoint and path-style in one profile#

The CLI reads two files in ~/.aws/ (%USERPROFILE%\.aws\ on Windows): credentials for keys and config for everything else. Create a named profile so your S3-compatible storage never mixes with an Amazon account you might also have.

$ aws configure --profile renodeAWS Access Key ID [None]: RNAKEXAMPLE123AWS Secret Access Key [None]: ****************************************Default region name [None]: us-east-1Default output format [None]: json

That writes the keys and region. Add the endpoint and addressing style by editing ~/.aws/config:

[profile renode]region = us-east-1output = jsonendpoint_url = https://s3.example.comrequest_checksum_calculation = when_requiredresponse_checksum_validation = when_requireds3 =  addressing_style = path

[renode]aws_access_key_id = RNAKEXAMPLE123aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

What each line is for:

• endpoint_url - the base URL of the storage. Use the HTTPS hostname if you have one; a plain http://host:port endpoint also works, but sends your data unencrypted.
• region - any name the server accepts. It is part of every request signature, so it must be consistent. us-east-1 is the conventional placeholder and avoids the CLI sending a location constraint when creating buckets.
• addressing_style = path - puts the bucket in the URL path (https://s3.example.com/bucket/key) instead of a subdomain (https://bucket.s3.example.com/key), which needs wildcard DNS and a wildcard certificate the endpoint may not have. The nested s3 = block with indented settings is the CLI's syntax for service-specific options.
• request_checksum_calculation and response_checksum_validation - explained in the next section.

Note the asymmetry: in config the section is [profile renode], in credentials it is [renode]. Getting that wrong is the commonest reason for The config profile (renode) could not be found.

On RE:NODE the panel shows the access key and secret key generated for the storage server, and a first bucket already exists. The endpoint is the plan's address and port over plain HTTP, or your own hostname over HTTPS once you point an A record at the plan's proxy slot, which issues and renews the certificate.

The checksum change in newer CLI versions#

In January 2025, AWS CLI 2.23.0 and the AWS SDKs released at the same time changed a default: uploads now calculate a CRC checksum and send it in new headers on every request where the operation supports one, and downloads validate checksums when the server returns them. Amazon S3 understands those headers. Many S3-compatible servers did not, at least at first, and the result was failed uploads with errors that do not mention checksums at all - SignatureDoesNotMatch, MissingContentLength, XAmzContentSHA256Mismatch or a generic InvalidArgument.

The two settings in the profile above restore the old behaviour - checksums only where an operation requires them:

$ aws configure set request_checksum_calculation when_required --profile renode$ aws configure set response_checksum_validation when_required --profile renode

Or as environment variables, which affect every SDK-based tool in the same shell:

export AWS_REQUEST_CHECKSUM_CALCULATION=when_requiredexport AWS_RESPONSE_CHECKSUM_VALIDATION=when_required

If your server handles the new checksums correctly, you lose nothing important by setting these; integrity is still protected by TLS and by the Content-MD5 or payload hash where required. If uploads fail with odd signature errors on a CLI that worked last year, this is the first thing to check. The same settings exist in boto3 and the other SDKs, covered in S3 from Node and Python.

Environment variables instead of files#

For containers, CI jobs and one-off scripts, environment variables are often simpler than config files. The CLI reads:

Path-style addressing has no environment variable of its own in the CLI, so keep a minimal config file with the s3 = addressing_style = path block even when the keys come from the environment. AWS_CONFIG_FILE points the CLI at a config file somewhere other than ~/.aws/config, which is handy in containers.

For a one-off command against a different endpoint, --endpoint-url https://s3.example.com on the command line overrides everything else. Environment variables and secrets covers keeping the secret key out of shell history and repositories.

Everyday commands#

The high-level aws s3 commands handle multipart uploads, retries and recursion for you.

With the profile from above, every command takes --profile renode, or you export AWS_PROFILE=renode once per shell.

$ export AWS_PROFILE=renode$ aws s3 ls2026-10-08 09:12:44 my-first-bucket$ aws s3 cp ./world-2026-10-08.tar.gz s3://my-first-bucket/minecraft/upload: ./world-2026-10-08.tar.gz to s3://my-first-bucket/minecraft/world-2026-10-08.tar.gz

A trailing slash on the destination means "into this prefix, keep the file name". Without it, the key is exactly what you typed.

Checking that an upload is really there

A command that printed upload: and exited with status 0 did succeed, but scripts should verify rather than trust. The cheapest check is size: aws s3api head-object returns ContentLength, and comparing it with the local file's size catches truncated sources - the classic case being a dump that failed halfway and produced a small but valid-looking file. For a single-part upload the ETag is normally the MD5 of the content, so md5sum on the local file should match it; for a multipart upload the ETag ends in - and a part count and will never match an MD5, so compare sizes or keep a .sha256 file alongside the object instead.

$ stat -c %s world-2026-10-08.tar.gz734003200$ aws s3api head-object --bucket my-first-bucket \    --key minecraft/world-2026-10-08.tar.gz --query ContentLength734003200

The real test is downloading it again somewhere else and opening it - testing a restore before you need it covers how often and how.

cp reads from standard input when the source is -, which lets you stream a backup without writing a temporary file:

$ mysqldump --single-transaction app | gzip \    | aws s3 cp - s3://my-first-bucket/db/app-$(date +%F).sql.gz

For streams larger than about 50 GB add --expected-size with the approximate byte count, so the CLI can choose part sizes that fit within the 10,000-part limit. Database dumps to S3 on a schedule builds a full job around this.

presign creates a URL with the signature in the query string. The default lifetime is 3,600 seconds and the maximum is seven days (604,800 seconds). Anyone with the URL can download the object until it expires, so treat it like a password with a time limit. S3 presigned URLs for uploads covers the upload side, which the CLI cannot generate.

sync: what it compares and how to filter#

aws s3 sync copies whatever is new or changed from source to destination - local to bucket, bucket to local, or bucket to bucket.

$ aws s3 sync ./backups s3://my-first-bucket/backups --dryrun$ aws s3 sync ./backups s3://my-first-bucket/backups

It decides a file has changed when the size differs or the local modification time is newer than the object's Last-Modified. It does not compare contents. Two flags change that:

• --size-only - compare size only. Useful when timestamps are unreliable, for example after files were copied without preserving times. Misses changes that keep the size.
• --exact-timestamps - when downloading, treat same-size files with any timestamp difference as changed.

By default sync never deletes. --delete removes objects from the destination that no longer exist in the source - which turns a mistake on the source into a mistake on the destination. A sync with --delete is a mirror, not a backup. Always run it once with --dryrun and read the output.

Filters are --exclude and --include patterns, applied in order, with later ones taking precedence:

# Only .tar.gz files$ aws s3 sync ./backups s3://my-first-bucket/backups \    --exclude "*" --include "*.tar.gz"# Everything except logs and temp files$ aws s3 sync ./site s3://my-first-bucket/site \    --exclude "*.log" --exclude "tmp/*"

Sync works in the other direction too, which is how you restore: aws s3 sync s3://my-first-bucket/backups ./restore downloads whatever is missing or different locally. Point it at an empty directory rather than the live one, check what arrived, and only then move it into place. Bucket-to-bucket sync on the same endpoint (aws s3 sync s3://bucket-a s3://bucket-b) uses server-side copies, so nothing passes through your machine.

Patterns are matched against the path relative to the source directory. --exclude "*" --include "*.tar.gz" works; the reverse order excludes everything, because the later --exclude "*" wins.

Speed, multipart and bandwidth#

The CLI uploads files above 8 MB in 8 MB parts, ten requests at a time. For large files over a fast link, larger parts and more concurrency help; on a slow or shared link, a bandwidth cap keeps the upload from starving everything else.

[profile renode]s3 =  addressing_style = path  max_concurrent_requests = 10  multipart_threshold = 64MB  multipart_chunksize = 64MB  max_bandwidth = 20MB/s

max_bandwidth takes bytes per second with a suffix, so 20MB/s is 160 Mbit/s. Raising max_concurrent_requests past 20-30 rarely helps against a single small endpoint; the server and the network between you become the limit. Very large parts mean a failed part takes longer to retry; 16-64 MB is a sensible range.

Low-level commands with s3api#

aws s3api maps one-to-one onto API operations. It is verbose but exposes everything:

# Metadata of one object: size, ETag, content type, user metadata$ aws s3api head-object --bucket my-first-bucket --key db/app-2026-10-08.sql.gz# Keys and sizes under a prefix, filtered with JMESPath$ aws s3api list-objects-v2 --bucket my-first-bucket --prefix db/ \    --query 'Contents[].[Key,Size]' --output text# Find and abort abandoned multipart uploads$ aws s3api list-multipart-uploads --bucket my-first-bucket$ aws s3api abort-multipart-upload --bucket my-first-bucket \    --key big.tar --upload-id 'EXAMPLEuploadID'

Set the content type explicitly when uploading files a browser will fetch - aws s3 cp logo.svg s3://bucket/ --content-type image/svg+xml - because a guessed binary/octet-stream makes browsers download instead of display.

Troubleshooting#

`Could not connect to the endpoint URL`. Wrong host, port or scheme in endpoint_url, or a firewall. Try curl -I against the endpoint; any HTTP response, even 403, proves the network path.

`InvalidAccessKeyId`. The access key is wrong, or the CLI is using a different profile than you think. aws configure list --profile renode shows which credentials and region it resolved, and where from.

`SignatureDoesNotMatch`. The secret key is wrong, the region differs from what the server expects, the system clock is off by minutes, or - on CLI 2.23 and later - the checksum headers above. Check the clock with date, then the checksum settings.

`RequestTimeTooSkewed`. The machine's clock is too far from the server's. Fix time synchronisation (timedatectl on systemd Linux) rather than anything in the CLI.

`AccessDenied` on one bucket but not another. The keys belong to a different server or account than the bucket, or the bucket name is misspelled - some servers answer a bucket you cannot see with access denied rather than not found.

`NoSuchBucket` or DNS errors for `bucket.s3.example.com`. The CLI is using virtual-hosted addressing. Add addressing_style = path.

`SSL: CERTIFICATE_VERIFY_FAILED`. You are using an HTTPS endpoint with a certificate the CLI does not trust, or an IP address on a certificate issued for a name. Use the hostname on the certificate. --no-verify-ssl makes the error disappear by removing the protection; do not leave it in scripts.

For anything else, --debug prints every request and response header, which shows exactly what was signed and what the server said.

FAQ#

Do I need an AWS account to use the AWS CLI?

No. The CLI is just a client. Point it at any S3-compatible endpoint with that endpoint's keys and it never contacts Amazon.

Can I use the same profile for Amazon S3 and another provider?

Use separate profiles. Each profile carries its own endpoint, keys and addressing style, and --profile or AWS_PROFILE picks one. Mixing them in one profile is how backups end up in the wrong place.

Why is sync re-uploading files that have not changed?

Usually modification times: something touched the files, or they were copied without preserving timestamps, so the local copies look newer. --size-only avoids it when you trust sizes. Another cause is a destination prefix that differs from the last run by a trailing slash or a typo.

Is sync with --delete a backup?

No. It makes the destination match the source, including deletions and corruption. For backups with history use a tool that keeps snapshots, such as restic - see restic backups to S3 - or rclone with a backup directory for changed and deleted files.

Can I copy between buckets on two different providers?

Not in one command. aws s3 cp and sync use one endpoint per invocation, so copying from one provider to another means downloading to a local directory with one profile and uploading with the other. rclone can do it in one step by streaming between two remotes.

How do I schedule uploads with cron?

Cron runs with a minimal environment, so give the job everything explicitly: the full path to aws, AWS_PROFILE, and HOME (so the CLI finds ~/.aws). Redirect output to a log file and check it, because a cron job that fails silently is a backup that does not exist.