rclone with S3 storage: sync, copy, mount and crypt

rclone is the tool to reach for when you want to move files between your machine, a server and S3 storage with more control than the AWS CLI gives you. Configure one remote of type s3 with provider Other, your endpoint, access key, secret key, any region and force_path_style = true, and then rclone copy, rclone sync, rclone mount and an encrypted crypt layer all work against it. The commands are simple; the one that needs care is sync, because it deletes. This post sets up the remote, explains how rclone decides what to transfer, and shows the flags that keep a scheduled job fast, polite to your bandwidth and safe from its own mistakes.

Configure the remote#

Install rclone from rclone.org (a single binary; distribution packages are often far behind) and check rclone version. Then create a remote. The interactive way is rclone config, choosing s3, then provider Other for a generic S3-compatible service. The direct way is one command:

$ rclone config create renode s3 \    provider=Other \    access_key_id=RNAKEXAMPLE123 \    secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \    endpoint=https://s3.example.com \    region=us-east-1 \    force_path_style=true \    no_check_bucket=true

Which writes this to the config file - rclone config file prints where that is:

[renode]type = s3provider = Otheraccess_key_id = RNAKEXAMPLE123secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEYendpoint = https://s3.example.comregion = us-east-1force_path_style = trueno_check_bucket = true

The settings that matter:

• provider = Other - tells rclone not to assume any provider's quirks.
• endpoint - the base URL. A hostname with HTTPS is the right choice for anything crossing the internet; http://host:port works but sends data in plain text.
• region - any name the server accepts; it is part of the request signature. Keep it consistent across tools.
• force_path_style = true - the bucket goes in the URL path rather than a subdomain. It is rclone's default, but stating it protects you from a provider preset that turns it off.
• no_check_bucket = true - stops rclone trying to create the bucket before each upload. Useful when the bucket already exists, and necessary when your keys may not create buckets.

On RE:NODE the panel shows the access key and secret key generated for the storage server, and a first bucket is created for you. The endpoint is the plan's address and port over plain HTTP, or your own hostname over HTTPS once its A record points at the plan's proxy slot, which takes care of the certificate.

The config file holds the secret key. Either protect the file (chmod 600), or encrypt the whole config with rclone config and the "Set configuration password" option, supplying the password at run time in RCLONE_CONFIG_PASS. For containers and CI, a remote can live entirely in environment variables - RCLONE_CONFIG_RENODE_TYPE=s3, RCLONE_CONFIG_RENODE_ENDPOINT=... and so on, one per setting - with no file at all.

First commands and the path syntax#

Paths are remote:bucket/prefix. A local path is just a path.

$ rclone lsd renode:                       # buckets$ rclone lsf renode:my-bucket/backups/     # one level, names only$ rclone ls renode:my-bucket               # every object with size$ rclone size renode:my-bucket             # object count and total bytes$ rclone ncdu renode:my-bucket             # interactive usage browser

rclone ncdu is the quickest way to find out what is filling a bucket - it walks the whole tree and lets you browse it by size, like the Unix tool it is named after.

copy, sync, move: what each one does#

copy is safe by construction: the worst it can do is upload too much. sync is a mirror: if a file disappears or gets corrupted in the source, the next run reproduces that at the destination. That makes a plain sync a replica, not a backup.

# Always look first$ rclone sync /srv/data renode:my-bucket/data --dry-run# Then do it, with a ceiling on how much it may delete$ rclone sync /srv/data renode:my-bucket/data --max-delete 50

--dry-run lists what would happen. --interactive (-i) asks before each destructive action. --max-delete 50 aborts the run if it would delete more than 50 files, which is the guard that catches "the source disk was not mounted and looked empty".

sync with a safety net

--backup-dir turns sync into something much closer to a backup: anything that would be overwritten or deleted is moved into another directory instead. Combined with a dated name, each run keeps the previous versions of whatever it changed:

$ rclone sync /srv/data renode:my-bucket/current \    --backup-dir renode:my-bucket/archive/$(date +%F) \    --max-delete 100 --log-file /var/log/rclone-data.log --log-level INFO

The backup directory must be on the same remote as the destination and must not overlap it. On S3, "moving" a file into it is a server-side copy plus delete, so nothing is downloaded. You prune old archive folders yourself - with rclone purge renode:my-bucket/archive/2026-07-01, or by deleting folders older than your retention with a small script, or with rclone delete --min-age 90d renode:my-bucket/archive.

How rclone decides what to transfer#

By default rclone compares size and modification time. S3 has no native modification time other than when the object was written, so rclone stores the source file's time in the object's metadata (X-Amz-Meta-Mtime) on upload and reads it back to compare. Reading that metadata costs one HEAD request per object, which is slow on large trees.

The flags that change the comparison:

For uploads to S3, --checksum is usually the best choice: it is fast, because the ETag in the listing carries the MD5 for normal uploads, and it is precise. For files uploaded in multiple parts the ETag is not an MD5; rclone stores the real MD5 in metadata (X-Amz-Meta-Md5chksum) when it uploads them, so it can still compare - but objects uploaded by other tools may lack it.

--fast-list lists the whole bucket in fewer requests, at the cost of memory to hold the listing. On buckets up to a few million objects it is usually faster; on a machine with little memory and a huge bucket, leave it off.

rclone check src dst compares without transferring, and --one-way checks only that everything in the source exists at the destination. Run it after the first big upload and periodically after that.

Filters#

rclone's filter rules decide which files are included:

# Only .tar.gz and .sql.gz files$ rclone copy /srv/backups renode:my-bucket/backups \    --include "*.tar.gz" --include "*.sql.gz"# Everything except caches and logs, using one ordered filter list$ rclone sync /srv/site renode:my-bucket/site --filter-from filters.txt

- cache/**- *.log- node_modules/**+ **

Do not mix --include and --exclude on one command line; rclone's documentation warns that the result is hard to predict. Use --filter or --filter-from, where each line begins with + or - and the first matching rule wins. --max-age 7d limits a run to recently changed files, and --min-size and --max-size filter by size. Test any filter with rclone ls --filter-from filters.txt /srv/site before trusting it with sync.

Speed and bandwidth limits#

rclone transfers 4 files at a time (--transfers 4) with 8 checkers comparing files in parallel (--checkers 8). Files above the upload cutoff - 200 MiB by default (--s3-upload-cutoff) - go up in multipart chunks of 5 MiB (--s3-chunk-size), four chunks at a time per file (--s3-upload-concurrency).

Memory for multipart uploads is roughly chunk size times upload concurrency times transfers - 64 MiB x 8 x 4 is 2 GiB, which is too much for a small server. Size these to the machine.

--bwlimit is in bytes per second, not bits: --bwlimit 10M is 10 MiB/s, about 84 Mbit/s. It accepts a timetable, so a backup can crawl during the day and run flat out at night:

$ rclone sync /srv/data renode:my-bucket/current \    --bwlimit "08:00,2M 19:00,8M 23:00,off"

Separate upload and download limits are written --bwlimit 4M:off (upload 4 MiB/s, download unlimited). For a game server or web server sharing its uplink with the backup job, a daytime limit is the difference between a backup nobody notices and one that causes lag.

Mounting a bucket as a drive#

rclone mount presents a remote as a filesystem. On Linux it needs FUSE (fuse3); on Windows it needs WinFsp and mounts to a drive letter; on macOS it needs macFUSE or FUSE-T.

# Linux$ mkdir -p /mnt/bucket$ rclone mount renode:my-bucket /mnt/bucket --vfs-cache-mode writes --daemon$ fusermount -u /mnt/bucket        # unmount# Windowsrclone mount renode:my-bucket X: --vfs-cache-mode writes

The --vfs-cache-mode setting decides how faithfully the mount behaves like a disk:

• off - reads and writes stream directly; many applications fail because they cannot seek while writing.
• writes - files opened for writing are buffered on local disk and uploaded when closed. The sensible default.
• full - reads are cached too, so applications can seek around large files. Uses local disk up to --vfs-cache-max-size.

A mount is convenient for browsing, dragging files in and out, and feeding tools that only understand paths. It is not a disk: every save is a full upload, renaming a folder is a copy of every file in it, and two machines mounting the same bucket see each other's changes late or not at all. Do not run a database, a game world or an application's working directory on it. Use rclone copy in scripts instead of a mount whenever you can.

Encrypting with crypt#

A crypt remote wraps another remote and encrypts file contents - and optionally names - before they leave your machine. The storage provider sees only random-looking objects.

$ rclone config create renode-crypt crypt \    remote=renode:my-bucket/encrypted \    filename_encryption=standard \    directory_name_encryption=true \    password='a-long-passphrase-you-keep-somewhere-safe' \    --obscure

--obscure tells rclone that the password is given in plain text and must be obscured before it is written to the config file. Without the flag rclone guesses, and its documentation warns that a long passphrase made only of base64 characters can be mistaken for an already obscured one.

From then on, use renode-crypt: like any other remote: rclone copy /srv/backups renode-crypt:backups. Files appear in the bucket under encrypted/ with scrambled names.

Encrypted file names are longer than the originals, and very long names or deep paths can exceed the key length limit. filename_encryption = obfuscate produces shorter names with weaker protection; off leaves names readable and encrypts only contents. With crypt, --checksum no longer works across the boundary because the stored data differs from the source; rclone falls back to size and modification time. rclone cryptcheck verifies an encrypted remote against the source.

If what you want is encrypted, deduplicated backups with history rather than an encrypted copy, a backup tool fits better than crypt - see restic backups to S3.

Running it on a schedule#

A scheduled rclone job should be boring and loud: boring when it works, loud when it does not. A small wrapper script does both.

#!/bin/shset -euLOG=/var/log/rclone-offsite.log/usr/bin/rclone sync /srv/backups renode:my-bucket/current \    --backup-dir "renode:my-bucket/archive/$(date +%F)" \    --checksum --max-delete 100 \    --bwlimit "08:00,2M 23:00,off" \    --log-file "$LOG" --log-level INFO \  || { echo "offsite sync failed, see $LOG" | mail -s "rclone failed" you@example.com; exit 1; }

30 4 * * * /usr/local/bin/offsite-sync.sh

rclone exits with status 0 only when everything transferred. Status 1 is a syntax or usage error, 3 means a directory was not found, 4 a file was not found, and higher values cover retry exhaustion and fatal errors, so a non-zero exit always deserves a look. rclone already retries failed operations (--retries 3 by default) and low-level requests (--low-level-retries 10), so a failure that survives that is real.

Schedule the upload after whatever produces the files - the panel backup, the database dump, the archive script - with enough gap that the files are complete. Uploading a dump while it is still being written produces a truncated copy that --checksum will happily consider current until the next run. Read the log once a week, and look at the size of the last archive folder: a sudden jump means a lot changed, and an empty one may mean the source stopped producing anything.

FAQ#

rclone or the AWS CLI?

For moving files to and from S3 alone, both work. rclone has better filters, bandwidth timetables, --backup-dir, mounts, encryption and support for dozens of other storage systems, so one tool can copy between S3 and anything else. The AWS CLI is the reference implementation and the better tool for low-level S3 operations; using the AWS CLI with S3 storage covers it.

Why does rclone try to create my bucket?

By default it checks that the destination bucket exists, and creates it if it does not, before uploading. If your keys are not allowed to create buckets, that fails. no_check_bucket = true in the remote, or --s3-no-check-bucket on the command line, skips the check.

Can rclone copy directly between two S3 providers?

Yes: rclone copy renode:my-bucket other:their-bucket. Within one remote rclone uses server-side copy; between different providers it streams the data through the machine running rclone, so run it somewhere with a good connection to both.

Is rclone sync a backup?

Not on its own; it mirrors deletions and corruption. With --backup-dir and a dated folder it keeps previous versions of changed and deleted files, which is a reasonable simple backup. For snapshots with deduplication and retention policies, use restic.

How do I run rclone on a schedule?

From cron or a systemd timer, with --log-file and --log-level INFO, and alert on a non-zero exit code. Keep --max-delete on scheduled syncs. Scheduled tasks worth having covers ordering a backup job with the things it depends on.