Quickstart¶

•

Download (https://rclone.org/downloads/) the relevant binary.

•

Extract the rclone or rclone.exe binary from the archive

•

Run rclone config to setup. See rclone config docs (https://rclone.org/docs/) for more details.

See below for some expanded Linux / macOS instructions.

See the Usage section (https://rclone.org/docs/) of the docs for how to use rclone, or run rclone -h.

Script installation¶

curl https://rclone.org/install.sh | sudo bash

    

For beta installation, run:

curl https://rclone.org/install.sh | sudo bash -s beta

    

Note that this script checks the version of rclone installed first and won't re-download if not needed.

Linux installation from precompiled binary¶

curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip
unzip rclone-current-linux-amd64.zip
cd rclone-*-linux-amd64

    

Copy binary file

sudo cp rclone /usr/bin/
sudo chown root:root /usr/bin/rclone
sudo chmod 755 /usr/bin/rclone

    

Install manpage

sudo mkdir -p /usr/local/share/man/man1
sudo cp rclone.1 /usr/local/share/man/man1/
sudo mandb 

    

Run rclone config to setup. See rclone config docs (https://rclone.org/docs/) for more details.

rclone config

    

macOS installation from precompiled binary¶

cd && curl -O https://downloads.rclone.org/rclone-current-osx-amd64.zip

    

Unzip the download and cd to the extracted folder.

unzip -a rclone-current-osx-amd64.zip && cd rclone-*-osx-amd64

    

Move rclone to your $PATH. You will be prompted for your password.

sudo mkdir -p /usr/local/bin
sudo mv rclone /usr/local/bin/

    

(the mkdir command is safe to run, even if the directory already exists).

Remove the leftover files.

cd .. && rm -rf rclone-*-osx-amd64 rclone-current-osx-amd64.zip

    

Run rclone config to setup. See rclone config docs (https://rclone.org/docs/) for more details.

rclone config

    

Install from source¶

git clone https://github.com/ncw/rclone.git
cd rclone
go build
./rclone version

    

You can also build and install rclone in the GOPATH (https://github.com/golang/go/wiki/GOPATH) (which defaults to ~/go) with:

go get -u -v github.com/ncw/rclone

    

and this will build the binary in $GOPATH/bin (~/go/bin/rclone by default) after downloading the source to $GOPATH/src/github.com/ncw/rclone (~/go/src/github.com/ncw/rclone by default).

Installation with Ansible¶

Instructions

1.

git clone https://github.com/stefangweichinger/ansible-rclone.git into your local roles-directory

2.

add the role to the hosts you want rclone installed to:

    - hosts: rclone-hosts
      roles:
          - rclone

    

Configure¶

The easiest way to make the config is to run rclone with the config option:

rclone config

    

See the following for detailed instructions for

•

Alias (https://rclone.org/alias/)

•

Amazon Drive (https://rclone.org/amazonclouddrive/)

•

Amazon S3 (https://rclone.org/s3/)

•

Backblaze B2 (https://rclone.org/b2/)

•

Box (https://rclone.org/box/)

•

Cache (https://rclone.org/cache/)

•

Crypt (https://rclone.org/crypt/) - to encrypt other remotes

•

DigitalOcean Spaces (/s3/#digitalocean-spaces)

•

Dropbox (https://rclone.org/dropbox/)

•

FTP (https://rclone.org/ftp/)

•

Google Cloud Storage (https://rclone.org/googlecloudstorage/)

•

Google Drive (https://rclone.org/drive/)

•

HTTP (https://rclone.org/http/)

•

Hubic (https://rclone.org/hubic/)

•

Jottacloud (https://rclone.org/jottacloud/)

•

Mega (https://rclone.org/mega/)

•

Microsoft Azure Blob Storage (https://rclone.org/azureblob/)

•

Microsoft OneDrive (https://rclone.org/onedrive/)

•

Openstack Swift / Rackspace Cloudfiles / Memset Memstore (https://rclone.org/swift/)

•

OpenDrive (https://rclone.org/opendrive/)

•

Pcloud (https://rclone.org/pcloud/)

•

QingStor (https://rclone.org/qingstor/)

•

SFTP (https://rclone.org/sftp/)

•

Union (https://rclone.org/union/)

•

WebDAV (https://rclone.org/webdav/)

•

Yandex Disk (https://rclone.org/yandex/)

•

The local filesystem (https://rclone.org/local/)

Usage¶

Its syntax is like this

Syntax: [options] subcommand <parameters> <parameters...>

    

Source and destination paths are specified by the name you gave the storage system in the config file then the sub path, eg "drive:myfolder" to look at "myfolder" in Google drive.

You can define as many storage paths as you like in the config file.

Subcommands¶

rclone ls remote:path # lists a re
rclone copy /local/path remote:path # copies /local/path to the remote
rclone sync /local/path remote:path # syncs /local/path to the remote

    

rclone config¶

Synopsis¶

rclone config [flags]

    

Options¶

  -h, --help   help for config

    

rclone copy¶

Synopsis¶

Note that it is always the contents of the directory that is synced, not the directory so when source:path is a directory, it's the contents of source:path that are copied, not the directory name and contents.

If dest:path doesn't exist, it is created and the source:path contents go there.

For example

rclone copy source:sourcepath dest:destpath

    

Let's say there are two files in sourcepath

sourcepath/one.txt
sourcepath/two.txt

    

This copies them to

destpath/one.txt
destpath/two.txt

    

Not to

destpath/sourcepath/one.txt
destpath/sourcepath/two.txt

    

If you are familiar with rsync, rclone always works as if you had written a trailing / - meaning "copy the contents of this directory". This applies to all commands and whether you are talking about the source or destination.

Note: Use the -P/--progress flag to view real-time transfer statistics

rclone copy source:path dest:path [flags]

    

Options¶

  -h, --help   help for copy

    

rclone sync¶

Synopsis¶

Important: Since this can cause data loss, test first with the --dry-run flag to see exactly what would be copied and deleted.

Note that files in the destination won't be deleted if there were any errors at any point.

It is always the contents of the directory that is synced, not the directory so when source:path is a directory, it's the contents of source:path that are copied, not the directory name and contents. See extended explanation in the copy command above if unsure.

If dest:path doesn't exist, it is created and the source:path contents go there.

Note: Use the -P/--progress flag to view real-time transfer statistics

rclone sync source:path dest:path [flags]

    

Options¶

  -h, --help   help for sync

    

rclone move¶

Synopsis¶

If no filters are in use and if possible this will server side move source:path into dest:path. After this source:path will no longer longer exist.

Otherwise for each file in source:path selected by the filters (if any) this will move it into dest:path. If possible a server side move will be used, otherwise it will copy it (server side if possible) into dest:path then delete the original (if no errors on copy) in source:path.

If you want to delete empty source directories after move, use the --delete-empty-src-dirs flag.

Important: Since this can cause data loss, test first with the --dry-run flag.

Note: Use the -P/--progress flag to view real-time transfer statistics.

rclone move source:path dest:path [flags]

    

Options¶

      --delete-empty-src-dirs   Delete empty source dirs after move
  -h, --help                    help for move

    

rclone delete¶

Synopsis¶

rclone delete only deletes objects but leaves the directory structure alone. If you want to delete a directory and all of its contents use rclone purge

Eg delete all files bigger than 100MBytes

Check what would be deleted first (use either)

rclone --min-size 100M lsl remote:path
rclone --dry-run --min-size 100M delete remote:path

    

Then delete

rclone --min-size 100M delete remote:path

    

That reads "delete everything with a minimum size of 100 MB", hence delete all files bigger than 100MBytes.

rclone delete remote:path [flags]

    

Options¶

  -h, --help   help for delete

    

rclone purge¶

Synopsis¶

rclone purge remote:path [flags]

    

Options¶

  -h, --help   help for purge

    

rclone mkdir¶

Synopsis¶

rclone mkdir remote:path [flags]

    

Options¶

  -h, --help   help for mkdir

    

rclone rmdir¶

Synopsis¶

rclone rmdir remote:path [flags]

    

Options¶

  -h, --help   help for rmdir

    

rclone check¶

Synopsis¶

If you supply the --size-only flag, it will only compare the sizes not the hashes as well. Use this for a quick check.

If you supply the --download flag, it will download the data from both remotes and check them against each other on the fly. This can be useful for remotes that don't support hashes or if you really want to check all the data.

If you supply the --one-way flag, it will only check that files in source match the files in destination, not the other way around. Meaning extra files in destination that are not in the source will not trigger an error.

rclone check source:path dest:path [flags]

    

Options¶

      --download   Check by downloading rather than with hash.
  -h, --help       help for check
      --one-way    Check one way only, source files must exist on remote

    

rclone ls¶

Synopsis¶

Eg

$ rclone ls swift:bucket
    60295 bevajer5jef
    90613 canole
    94467 diwogej7
    37600 fubuwic

    

Any of the filtering options can be applied to this commmand.

There are several related list commands

•

ls to list size and path of objects only

•

lsl to list modification time, size and path of objects only

•

lsd to list directories only

•

lsf to list objects and directories in easy to parse format

•

lsjson to list objects and directories in JSON format

ls,lsl,lsd are designed to be human readable. lsf is designed to be human and machine readable. lsjson is designed to be machine readable.

Note that ls and lsl recurse by default - use "--max-depth 1" to stop the recursion.

The other list commands lsd,lsf,lsjson do not recurse by default - use "-R" to make them recurse.

Listing a non existent directory will produce an error except for remotes which can't have empty directories (eg s3, swift, gcs, etc - the bucket based remotes).

rclone ls remote:path [flags]

    

Options¶

  -h, --help   help for ls

    

rclone lsd¶

Synopsis¶

This command lists the total size of the directory (if known, -1 if not), the modification time (if known, the current time if not), the number of objects in the directory (if known, -1 if not) and the name of the directory, Eg

$ rclone lsd swift:
      494000 2018-04-26 08:43:20     10000 10000files
          65 2018-04-26 08:43:20         1 1File

    

Or

$ rclone lsd drive:test
          -1 2016-10-17 17:41:53        -1 1000files
          -1 2017-01-03 14:40:54        -1 2500files
          -1 2017-07-08 14:39:28        -1 4000files

    

If you just want the directory names use "rclone lsf --dirs-only".

Any of the filtering options can be applied to this commmand.

There are several related list commands

•

ls to list size and path of objects only

•

lsl to list modification time, size and path of objects only

•

lsd to list directories only

•

lsf to list objects and directories in easy to parse format

•

lsjson to list objects and directories in JSON format

ls,lsl,lsd are designed to be human readable. lsf is designed to be human and machine readable. lsjson is designed to be machine readable.

Note that ls and lsl recurse by default - use "--max-depth 1" to stop the recursion.

The other list commands lsd,lsf,lsjson do not recurse by default - use "-R" to make them recurse.

Listing a non existent directory will produce an error except for remotes which can't have empty directories (eg s3, swift, gcs, etc - the bucket based remotes).

rclone lsd remote:path [flags]

    

Options¶

  -h, --help        help for lsd
  -R, --recursive   Recurse into the listing.

    

rclone lsl¶

Synopsis¶

Eg

$ rclone lsl swift:bucket
    60295 2016-06-25 18:55:41.062626927 bevajer5jef
    90613 2016-06-25 18:55:43.302607074 canole
    94467 2016-06-25 18:55:43.046609333 diwogej7
    37600 2016-06-25 18:55:40.814629136 fubuwic

    

Any of the filtering options can be applied to this commmand.

There are several related list commands

•

ls to list size and path of objects only

•

lsl to list modification time, size and path of objects only

•

lsd to list directories only

•

lsf to list objects and directories in easy to parse format

•

lsjson to list objects and directories in JSON format

ls,lsl,lsd are designed to be human readable. lsf is designed to be human and machine readable. lsjson is designed to be machine readable.

Note that ls and lsl recurse by default - use "--max-depth 1" to stop the recursion.

The other list commands lsd,lsf,lsjson do not recurse by default - use "-R" to make them recurse.

Listing a non existent directory will produce an error except for remotes which can't have empty directories (eg s3, swift, gcs, etc - the bucket based remotes).

rclone lsl remote:path [flags]

    

Options¶

  -h, --help   help for lsl

    

rclone md5sum¶

Synopsis¶

rclone md5sum remote:path [flags]

    

Options¶

  -h, --help   help for md5sum

    

rclone sha1sum¶

Synopsis¶

rclone sha1sum remote:path [flags]

    

Options¶

  -h, --help   help for sha1sum

    

rclone size¶

Synopsis¶

rclone size remote:path [flags]

    

Options¶

  -h, --help   help for size
      --json   format output as JSON

    

rclone version¶

Synopsis¶

Eg

$ rclone version
rclone v1.41
- os/arch: linux/amd64
- go version: go1.10

    

If you supply the --check flag, then it will do an online check to compare your version with the latest release and the latest beta.

$ rclone version --check
yours:  1.42.0.6
latest: 1.42          (released 2018-06-16)
beta:   1.42.0.5      (released 2018-06-17)

    

Or

$ rclone version --check
yours:  1.41
latest: 1.42          (released 2018-06-16)
  upgrade: https://downloads.rclone.org/v1.42
beta:   1.42.0.5      (released 2018-06-17)
  upgrade: https://beta.rclone.org/v1.42-005-g56e1e820

    

rclone version [flags]

    

Options¶

      --check   Check for new version.
  -h, --help    help for version

    

rclone cleanup¶

Synopsis¶

rclone cleanup remote:path [flags]

    

Options¶

  -h, --help   help for cleanup

    

rclone dedupe¶

Synopsis¶

In the first pass it will merge directories with the same name. It will do this iteratively until all the identical directories have been merged.

The dedupe command will delete all but one of any identical (same md5sum) files it finds without confirmation. This means that for most duplicated files the dedupe command will not be interactive. You can use --dry-run to see what would happen without doing anything.

Here is an example run.

Before - with duplicates

$ rclone lsl drive:dupes
  6048320 2016-03-05 16:23:16.798000000 one.txt
  6048320 2016-03-05 16:23:11.775000000 one.txt
   564374 2016-03-05 16:23:06.731000000 one.txt
  6048320 2016-03-05 16:18:26.092000000 one.txt
  6048320 2016-03-05 16:22:46.185000000 two.txt
  1744073 2016-03-05 16:22:38.104000000 two.txt
   564374 2016-03-05 16:22:52.118000000 two.txt

    

Now the dedupe session

$ rclone dedupe drive:dupes
2016/03/05 16:24:37 Google drive root 'dupes': Looking for duplicates using interactive mode.
one.txt: Found 4 duplicates - deleting identical copies
one.txt: Deleting 2/3 identical duplicates (md5sum "1eedaa9fe86fd4b8632e2ac549403b36")
one.txt: 2 duplicates remain
  1:      6048320 bytes, 2016-03-05 16:23:16.798000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36
  2:       564374 bytes, 2016-03-05 16:23:06.731000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81
s) Skip and do nothing
k) Keep just one (choose which in next step)
r) Rename all to be different (by changing file.jpg to file-1.jpg)
s/k/r> k
Enter the number of the file to keep> 1
one.txt: Deleted 1 extra copies
two.txt: Found 3 duplicates - deleting identical copies
two.txt: 3 duplicates remain
  1:       564374 bytes, 2016-03-05 16:22:52.118000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81
  2:      6048320 bytes, 2016-03-05 16:22:46.185000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36
  3:      1744073 bytes, 2016-03-05 16:22:38.104000000, md5sum 851957f7fb6f0bc4ce76be966d336802
s) Skip and do nothing
k) Keep just one (choose which in next step)
r) Rename all to be different (by changing file.jpg to file-1.jpg)
s/k/r> r
two-1.txt: renamed from: two.txt
two-2.txt: renamed from: two.txt
two-3.txt: renamed from: two.txt

    

The result being

$ rclone lsl drive:dupes
  6048320 2016-03-05 16:23:16.798000000 one.txt
   564374 2016-03-05 16:22:52.118000000 two-1.txt
  6048320 2016-03-05 16:22:46.185000000 two-2.txt
  1744073 2016-03-05 16:22:38.104000000 two-3.txt

    

Dedupe can be run non interactively using the --dedupe-mode flag or by using an extra parameter with the same value

•

--dedupe-mode interactive - interactive as above.

•

--dedupe-mode skip - removes identical files then skips anything left.

•

--dedupe-mode first - removes identical files then keeps the first one.

•

--dedupe-mode newest - removes identical files then keeps the newest one.

•

--dedupe-mode oldest - removes identical files then keeps the oldest one.

•

--dedupe-mode largest - removes identical files then keeps the largest one.

•

--dedupe-mode rename - removes identical files then renames the rest to be different.

For example to rename all the identically named photos in your Google Photos directory, do

rclone dedupe --dedupe-mode rename "drive:Google Photos"

    

Or

rclone dedupe rename "drive:Google Photos"

    

rclone dedupe [mode] remote:path [flags]

    

Options¶

      --dedupe-mode string   Dedupe mode interactive|skip|first|newest|oldest|rename. (default "interactive")
  -h, --help                 help for dedupe

    

rclone about¶

Synopsis¶

This will print to stdout something like this:

Total:   17G
Used:    7.444G
Free:    1.315G
Trashed: 100.000M
Other:   8.241G

    

Where the fields are:

•

Total: total size available.

•

Used: total size used

•

Free: total amount this user could upload.

•

Trashed: total amount in the trash

•

Other: total amount in other storage (eg Gmail, Google Photos)

•

Objects: total number of objects in the storage

Note that not all the backends provide all the fields - they will be missing if they are not known for that backend. Where it is known that the value is unlimited the value will also be omitted.

Use the --full flag to see the numbers written out in full, eg

Total:   18253611008
Used:    7993453766
Free:    1411001220
Trashed: 104857602
Other:   8849156022

    

Use the --json flag for a computer readable output, eg

{
    "total": 18253611008,
    "used": 7993453766,
    "trashed": 104857602,
    "other": 8849156022,
    "free": 1411001220
}

    

rclone about remote: [flags]

    

Options¶

      --full   Full numbers instead of SI units
  -h, --help   help for about
      --json   Format output as JSON

    

rclone authorize¶

Synopsis¶

rclone authorize [flags]

    

Options¶

  -h, --help   help for authorize

    

rclone cachestats¶

Synopsis¶

rclone cachestats source: [flags]

    

Options¶

  -h, --help   help for cachestats

    

rclone cat¶

Synopsis¶

You can use it like this to output a single file

rclone cat remote:path/to/file

    

Or like this to output any file in dir or subdirectories.

rclone cat remote:path/to/dir

    

Or like this to output any .txt files in dir or subdirectories.

rclone --include "*.txt" cat remote:path/to/dir

    

Use the --head flag to print characters only at the start, --tail for the end and --offset and --count to print a section in the middle. Note that if offset is negative it will count from the end, so --offset -1 --count 1 is equivalent to --tail 1.

rclone cat remote:path [flags]

    

Options¶

      --count int    Only print N characters. (default -1)
      --discard      Discard the output instead of printing.
      --head int     Only print the first N characters.
  -h, --help         help for cat
      --offset int   Start printing at offset N (or from end if -ve).
      --tail int     Only print the last N characters.

    

rclone config create¶

Synopsis¶

For example to make a swift remote of name myremote using auto config you would do:

rclone config create myremote swift env_auth true

    

rclone config create <name> <type> [<key> <value>]* [flags]

    

Options¶

  -h, --help   help for create

    

rclone config delete¶

Synopsis¶

rclone config delete <name> [flags]

    

Options¶

  -h, --help   help for delete

    

rclone config dump¶

Synopsis¶

rclone config dump [flags]

    

Options¶

  -h, --help   help for dump

    

rclone config edit¶

Synopsis¶

rclone config edit [flags]

    

Options¶

  -h, --help   help for edit

    

rclone config file¶

Synopsis¶

rclone config file [flags]

    

Options¶

  -h, --help   help for file

    

rclone config password¶

Synopsis¶

For example to set password of a remote of name myremote you would do:

rclone config password myremote fieldname mypassword

    

rclone config password <name> [<key> <value>]+ [flags]

    

Options¶

  -h, --help   help for password

    

rclone config providers¶

Synopsis¶

rclone config providers [flags]

    

Options¶

  -h, --help   help for providers

    

rclone config show¶

Synopsis¶

rclone config show [<remote>] [flags]

    

Options¶

  -h, --help   help for show

    

rclone config update¶

Synopsis¶

For example to update the env_auth field of a remote of name myremote you would do:

rclone config update myremote swift env_auth true

    

rclone config update <name> [<key> <value>]+ [flags]

    

Options¶

  -h, --help   help for update

    

rclone copyto¶

Synopsis¶

This can be used to upload single files to other than their current name. If the source is a directory then it acts exactly like the copy command.

So

rclone copyto src dst

    

where src and dst are rclone paths, either remote:path or /path/to/local or C:.

This will:

if src is file
    copy it to dst, overwriting an existing file if it exists
if src is directory
    copy it to dst, overwriting existing files if they exist
    see copy command for full details

    

This doesn't transfer unchanged files, testing by size and modification time or MD5SUM. It doesn't delete files from the destination.

Note: Use the -P/--progress flag to view real-time transfer statistics

rclone copyto source:path dest:path [flags]

    

Options¶

  -h, --help   help for copyto

    

rclone copyurl¶

Synopsis¶

rclone copyurl https://example.com dest:path [flags]

    

Options¶

  -h, --help   help for copyurl

    

rclone cryptcheck¶

Synopsis¶

For it to work the underlying remote of the cryptedremote must support some kind of checksum.

It works by reading the nonce from each file on the cryptedremote: and using that to encrypt each file on the remote:. It then checks the checksum of the underlying file on the cryptedremote: against the checksum of the file it has just encrypted.

Use it like this

rclone cryptcheck /path/to/files encryptedremote:path

    

You can use it like this also, but that will involve downloading all the files in remote:path.

rclone cryptcheck remote:path encryptedremote:path

    

After it has run it will log the status of the encryptedremote:.

If you supply the --one-way flag, it will only check that files in source match the files in destination, not the other way around. Meaning extra files in destination that are not in the source will not trigger an error.

rclone cryptcheck remote:path cryptedremote:path [flags]

    

Options¶

  -h, --help      help for cryptcheck
      --one-way   Check one way only, source files must exist on destination

    

rclone cryptdecode¶

Synopsis¶

If you supply the --reverse flag, it will return encrypted file names.

use it like this

rclone cryptdecode encryptedremote: encryptedfilename1 encryptedfilename2
rclone cryptdecode --reverse encryptedremote: filename1 filename2

    

rclone cryptdecode encryptedremote: encryptedfilename [flags]

    

Options¶

  -h, --help      help for cryptdecode
      --reverse   Reverse cryptdecode, encrypts filenames

    

rclone dbhashsum¶

Synopsis¶

rclone dbhashsum remote:path [flags]

    

Options¶

  -h, --help   help for dbhashsum

    

rclone deletefile¶

Synopsis¶

rclone deletefile remote:path [flags]

    

Options¶

  -h, --help   help for deletefile

    

rclone genautocomplete¶

Synopsis¶

Options¶

  -h, --help   help for genautocomplete

    

rclone genautocomplete bash¶

Synopsis¶

This writes to /etc/bash_completion.d/rclone by default so will probably need to be run with sudo or as root, eg

sudo rclone genautocomplete bash

    

Logout and login again to use the autocompletion scripts, or source them directly

. /etc/bash_completion

    

If you supply a command line argument the script will be written there.

rclone genautocomplete bash [output_file] [flags]

    

Options¶

  -h, --help   help for bash

    

rclone genautocomplete zsh¶

Synopsis¶

This writes to /usr/share/zsh/vendor-completions/_rclone by default so will probably need to be run with sudo or as root, eg

sudo rclone genautocomplete zsh

    

Logout and login again to use the autocompletion scripts, or source them directly

autoload -U compinit && compinit

    

If you supply a command line argument the script will be written there.

rclone genautocomplete zsh [output_file] [flags]

    

Options¶

  -h, --help   help for zsh

    

rclone gendocs¶

Synopsis¶

rclone gendocs output_directory [flags]

    

Options¶

  -h, --help   help for gendocs

    

rclone hashsum¶

Synopsis¶

Run without a hash to see the list of supported hashes, eg

$ rclone hashsum
Supported hashes are:
  * MD5
  * SHA-1
  * DropboxHash
  * QuickXorHash

    

Then

$ rclone hashsum MD5 remote:path

    

rclone hashsum <hash> remote:path [flags]

    

Options¶

  -h, --help   help for hashsum

    

rclone link¶

Synopsis¶

rclone link remote:path/to/file
rclone link remote:path/to/folder/

    

If successful, the last line of the output will contain the link. Exact capabilities depend on the remote, but the link will always be created with the least constraints – e.g. no expiry, no password protection, accessible without account.

rclone link remote:path [flags]

    

Options¶

  -h, --help   help for link

    

rclone listremotes¶

Synopsis¶

When uses with the -l flag it lists the types too.

rclone listremotes [flags]

    

Options¶

  -h, --help   help for listremotes
  -l, --long   Show the type as well as names.

    

rclone lsf¶

Synopsis¶

Eg

$ rclone lsf swift:bucket
bevajer5jef
canole
diwogej7
ferejej3gux/
fubuwic

    

Use the --format option to control what gets listed. By default this is just the path, but you can use these parameters to control the output:

p - path
s - size
t - modification time
h - hash
i - ID of object if known
m - MimeType of object if known

    

So if you wanted the path, size and modification time, you would use --format "pst", or maybe --format "tsp" to put the path last.

Eg

$ rclone lsf  --format "tsp" swift:bucket
2016-06-25 18:55:41;60295;bevajer5jef
2016-06-25 18:55:43;90613;canole
2016-06-25 18:55:43;94467;diwogej7
2018-04-26 08:50:45;0;ferejej3gux/
2016-06-25 18:55:40;37600;fubuwic

    

If you specify "h" in the format you will get the MD5 hash by default, use the "--hash" flag to change which hash you want. Note that this can be returned as an empty string if it isn't available on the object (and for directories), "ERROR" if there was an error reading it from the object and "UNSUPPORTED" if that object does not support that hash type.

For example to emulate the md5sum command you can use

rclone lsf -R --hash MD5 --format hp --separator "  " --files-only .

    

Eg

$ rclone lsf -R --hash MD5 --format hp --separator "  " --files-only swift:bucket 
7908e352297f0f530b84a756f188baa3  bevajer5jef
cd65ac234e6fea5925974a51cdd865cc  canole
03b5341b4f234b9d984d03ad076bae91  diwogej7
8fd37c3810dd660778137ac3a66cc06d  fubuwic
99713e14a4c4ff553acaf1930fad985b  gixacuh7ku

    

(Though "rclone md5sum ." is an easier way of typing this.)

By default the separator is ";" this can be changed with the --separator flag. Note that separators aren't escaped in the path so putting it last is a good strategy.

Eg

$ rclone lsf  --separator "," --format "tshp" swift:bucket
2016-06-25 18:55:41,60295,7908e352297f0f530b84a756f188baa3,bevajer5jef
2016-06-25 18:55:43,90613,cd65ac234e6fea5925974a51cdd865cc,canole
2016-06-25 18:55:43,94467,03b5341b4f234b9d984d03ad076bae91,diwogej7
2018-04-26 08:52:53,0,,ferejej3gux/
2016-06-25 18:55:40,37600,8fd37c3810dd660778137ac3a66cc06d,fubuwic

    

You can output in CSV standard format. This will escape things in " if they contain ,

Eg

$ rclone lsf --csv --files-only --format ps remote:path
test.log,22355
test.sh,449
"this file contains a comma, in the file name.txt",6

    

Note that the --absolute parameter is useful for making lists of files to pass to an rclone copy with the --files-from flag.

For example to find all the files modified within one day and copy those only (without traversing the whole directory structure):

rclone lsf --absolute --files-only --max-age 1d /path/to/local > new_files
rclone copy --files-from new_files /path/to/local remote:path

    

Any of the filtering options can be applied to this commmand.

There are several related list commands

•

ls to list size and path of objects only

•

lsl to list modification time, size and path of objects only

•

lsd to list directories only

•

lsf to list objects and directories in easy to parse format

•

lsjson to list objects and directories in JSON format

ls,lsl,lsd are designed to be human readable. lsf is designed to be human and machine readable. lsjson is designed to be machine readable.

Note that ls and lsl recurse by default - use "--max-depth 1" to stop the recursion.

The other list commands lsd,lsf,lsjson do not recurse by default - use "-R" to make them recurse.

Listing a non existent directory will produce an error except for remotes which can't have empty directories (eg s3, swift, gcs, etc - the bucket based remotes).

rclone lsf remote:path [flags]

    

Options¶

      --absolute           Put a leading / in front of path names.
      --csv                Output in CSV format.
  -d, --dir-slash          Append a slash to directory names. (default true)
      --dirs-only          Only list directories.
      --files-only         Only list files.
  -F, --format string      Output format - see  help for details (default "p")
      --hash h             Use this hash when h is used in the format MD5|SHA-1|DropboxHash (default "MD5")
  -h, --help               help for lsf
  -R, --recursive          Recurse into the listing.
  -s, --separator string   Separator for the items in the format. (default ";")

    

rclone lsjson¶

Synopsis¶

The output is an array of Items, where each Item looks like this

{ "Hashes" : { "SHA-1" : "f572d396fae9206628714fb2ce00f72e94f2258f", "MD5" : "b1946ac92492d2347c6235b4d2611184", "DropboxHash" : "ecb65bb98f9d905b70458986c39fcbad7715e5f2fcc3b1f07767d7c83e2438cc" }, "ID": "y2djkhiujf83u33", "OrigID": "UYOJVTUW00Q1RzTDA", "IsDir" : false, "MimeType" : "application/octet-stream", "ModTime" : "2017-05-31T16:15:57.034468261+01:00", "Name" : "file.txt", "Encrypted" : "v0qpsdq8anpci8n929v3uu9338", "Path" : "full/path/goes/here/file.txt", "Size" : 6 }

If --hash is not specified the Hashes property won't be emitted.

If --no-modtime is specified then ModTime will be blank.

If --encrypted is not specified the Encrypted won't be emitted.

The Path field will only show folders below the remote path being listed. If "remote:path" contains the file "subfolder/file.txt", the Path for "file.txt" will be "subfolder/file.txt", not "remote:path/subfolder/file.txt". When used without --recursive the Path will always be the same as Name.

The time is in RFC3339 format with nanosecond precision.

The whole output can be processed as a JSON blob, or alternatively it can be processed line by line as each item is written one to a line.

Any of the filtering options can be applied to this commmand.

There are several related list commands

•

ls to list size and path of objects only

•

lsl to list modification time, size and path of objects only

•

lsd to list directories only

•

lsf to list objects and directories in easy to parse format

•

lsjson to list objects and directories in JSON format

ls,lsl,lsd are designed to be human readable. lsf is designed to be human and machine readable. lsjson is designed to be machine readable.

Note that ls and lsl recurse by default - use "--max-depth 1" to stop the recursion.

The other list commands lsd,lsf,lsjson do not recurse by default - use "-R" to make them recurse.

Listing a non existent directory will produce an error except for remotes which can't have empty directories (eg s3, swift, gcs, etc - the bucket based remotes).

rclone lsjson remote:path [flags]

    

Options¶

  -M, --encrypted    Show the encrypted names.
      --hash         Include hashes in the output (may take longer).
  -h, --help         help for lsjson
      --no-modtime   Don't read the modification time (can speed things up).
      --original     Show the ID of the underlying Object.
  -R, --recursive    Recurse into the listing.

    

rclone mount¶

Synopsis¶

First set up your remote using rclone config. Check it works with rclone ls etc.

Start the mount like this

rclone mount remote:path/to/files /path/to/local/mount

    

Or on Windows like this where X: is an unused drive letter

rclone mount remote:path/to/files X:

    

When the program ends, either via Ctrl+C or receiving a SIGINT or SIGTERM signal, the mount is automatically stopped.

The umount operation can fail, for example when the mountpoint is busy. When that happens, it is the user's responsibility to stop the mount manually with

# Linux
fusermount -u /path/to/local/mount
# OS X
umount /path/to/local/mount

    

Installing on Windows¶

WinFsp is an open source (https://github.com/billziss-gh/winfsp) Windows File System Proxy which makes it easy to write user space file systems for Windows. It provides a FUSE emulation layer which rclone uses combination with cgofuse (https://github.com/billziss-gh/cgofuse). Both of these packages are by Bill Zissimopoulos who was very helpful during the implementation of rclone mount for Windows.

Windows caveats¶

The easiest way around this is to start the drive from a normal command prompt. It is also possible to start a drive from the SYSTEM account (using the WinFsp.Launcher infrastructure (https://github.com/billziss-gh/winfsp/wiki/WinFsp-Service-Architecture)) which creates drives accessible for everyone on the system or alternatively using the nssm service manager (https://nssm.cc/usage).

Limitations¶

The bucket based remotes (eg Swift, S3, Google Compute Storage, B2, Hubic) won't work from the root - you will need to specify a bucket, or a path within the bucket. So swift: won't work whereas swift:bucket will as will swift:bucket/path. None of these support the concept of directories, so empty directories will have a tendency to disappear once they fall out of the directory cache.

Only supported on Linux, FreeBSD, OS X and Windows at the moment.

rclone mount vs rclone sync/copy¶

Attribute caching¶

The default is "1s" which caches files just long enough to avoid too many callbacks to rclone from the kernel.

In theory 0s should be the correct value for filesystems which can change outside the control of the kernel. However this causes quite a few problems such as rclone using too much memory (https://github.com/ncw/rclone/issues/2157), rclone not serving files to samba (https://forum.rclone.org/t/rclone-1-39-vs-1-40-mount-issue/5112) and excessive time listing directories (https://github.com/ncw/rclone/issues/2095#issuecomment-371141147).

The kernel can cache the info about a file for the time given by "--attr-timeout". You may see corruption if the remote file changes length during this window. It will show up as either a truncated file or a file with garbage on the end. With "--attr-timeout 1s" this is very unlikely but not impossible. The higher you set "--attr-timeout" the more likely it is. The default setting of "1s" is the lowest setting which mitigates the problems above.

If you set it higher ('10s' or '1m' say) then the kernel will call back to rclone less often making it more efficient, however there is more chance of the corruption issue above.

If files don't change on the remote outside of the control of rclone then there is no chance of corruption.

This is the same as setting the attr_timeout option in mount.fuse.

Filters¶

systemd¶

chunked reading¶

When --vfs-read-chunk-size-limit is also specified and greater than --vfs-read-chunk-size, the chunk size for each open file will get doubled for each chunk read, until the specified value is reached. A value of -1 will disable the limit and the chunk size will grow indefinitely.

With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.

Chunked reading will only work with --vfs-cache-mode < full, as the file will always be copied to the vfs cache before opening with --vfs-cache-mode full.

Directory Cache¶

Alternatively, you can send a SIGHUP signal to rclone for it to flush all directory caches, regardless of how old they are. Assuming only one rclone instance is running, you can reset the cache like this:

kill -SIGHUP $(pidof rclone)

    

If you configure rclone with a remote control (/rc) then you can use rclone rc to flush the whole directory cache:

rclone rc vfs/forget

    

Or individual files or directories:

rclone rc vfs/forget file=path/to/file dir=path/to/dir

    

File Buffering¶

Each open file descriptor will try to keep the specified amount of data in memory at all times. The buffered data is bound to one file descriptor and won't be shared between multiple open file descriptors of the same file.

This flag is a upper limit for the used memory per file descriptor. The buffer will only use memory for data that is downloaded but not not yet read. If the buffer is empty, only a small amount of memory will be used. The maximum memory used by rclone for buffering can be up to --buffer-size * open files.

File Caching¶

You'll need to enable VFS caching if you want, for example, to read and write simultaneously to a file. See below for more details.

Note that the VFS cache works in addition to the cache backend and you may find that you need one or the other or both.

--cache-dir string                   Directory rclone will use for caching.
--vfs-cache-max-age duration         Max age of objects in the cache. (default 1h0m0s)
--vfs-cache-mode string              Cache mode off|minimal|writes|full (default "off")
--vfs-cache-poll-interval duration   Interval to poll the cache for stale objects. (default 1m0s)

    

If run with -vv rclone will print the location of the file cache. The files are stored in the user cache file area which is OS dependent but can be controlled with --cache-dir or setting the appropriate environment variable.

The cache has 4 different modes selected by --vfs-cache-mode. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.

Note that files are written back to the remote only when they are closed so if rclone is quit or dies with open files then these won't get written back to the remote. However they will still be in the on disk cache.

--vfs-cache-mode off¶

This will mean some operations are not possible

•

Files can't be opened for both read AND write

•

Files opened for write can't be seeked

•

Existing files opened for write must have O_TRUNC set

•

Files open for read with O_TRUNC will be opened write only

•

Files open for write only will behave as if O_TRUNC was supplied

•

Open modes O_APPEND, O_TRUNC are ignored

•

If an upload fails it can't be retried

--vfs-cache-mode minimal¶

These operations are not possible

•

Files opened for write only can't be seeked

•

Existing files opened for write must have O_TRUNC set

•

Files opened for write only will ignore O_APPEND, O_TRUNC

•

If an upload fails it can't be retried

--vfs-cache-mode writes¶

This mode should support all normal file system operations.

If an upload fails it will be retried up to --low-level-retries times.

--vfs-cache-mode full¶

This may be appropriate for your needs, or you may prefer to look at the cache backend which does a much more sophisticated job of caching, including caching directory hierarchies and chunks of files.

In this mode, unlike the others, when a file is written to the disk, it will be kept on the disk after it is written to the remote. It will be purged on a schedule according to --vfs-cache-max-age.

This mode should support all normal file system operations.

If an upload or download fails it will be retried up to --low-level-retries times.

rclone mount remote:path /path/to/mountpoint [flags]

    

Options¶

      --allow-non-empty                    Allow mounting over a non-empty directory.
      --allow-other                        Allow access to other users.
      --allow-root                         Allow access to root user.
      --attr-timeout duration              Time for which file/directory attributes are cached. (default 1s)
      --daemon                             Run mount as a daemon (background mode).
      --daemon-timeout duration            Time limit for rclone to respond to kernel (not supported by all OSes).
      --debug-fuse                         Debug the FUSE internals - needs -v.
      --default-permissions                Makes kernel enforce access control based on the file mode.
      --dir-cache-time duration            Time to cache directory entries for. (default 5m0s)
      --fuse-flag stringArray              Flags or arguments to be passed direct to libfuse/WinFsp. Repeat if required.
      --gid uint32                         Override the gid field set by the filesystem. (default 502)
  -h, --help                               help for mount
      --max-read-ahead int                 The number of bytes that can be prefetched for sequential reads. (default 128k)
      --no-checksum                        Don't compare checksums on up/download.
      --no-modtime                         Don't read/write the modification time (can speed things up).
      --no-seek                            Don't allow seeking in files.
  -o, --option stringArray                 Option for libfuse/WinFsp. Repeat if required.
      --poll-interval duration             Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable. (default 1m0s)
      --read-only                          Mount read-only.
      --uid uint32                         Override the uid field set by the filesystem. (default 502)
      --umask int                          Override the permission bits set by the filesystem.
      --vfs-cache-max-age duration         Max age of objects in the cache. (default 1h0m0s)
      --vfs-cache-mode string              Cache mode off|minimal|writes|full (default "off")
      --vfs-cache-poll-interval duration   Interval to poll the cache for stale objects. (default 1m0s)
      --vfs-read-chunk-size int            Read the source objects in chunks. (default 128M)
      --vfs-read-chunk-size-limit int      If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
      --volname string                     Set the volume name (not supported by all OSes).
      --write-back-cache                   Makes kernel buffer writes before sending them to rclone. Without this, writethrough caching is used.

    

rclone moveto¶

Synopsis¶

This can be used to rename files or upload single files to other than their existing name. If the source is a directory then it acts exacty like the move command.

So

rclone moveto src dst

    

where src and dst are rclone paths, either remote:path or /path/to/local or C:.

This will:

if src is file
    move it to dst, overwriting an existing file if it exists
if src is directory
    move it to dst, overwriting existing files if they exist
    see move command for full details

    

This doesn't transfer unchanged files, testing by size and modification time or MD5SUM. src will be deleted on successful transfer.

Important: Since this can cause data loss, test first with the --dry-run flag.

Note: Use the -P/--progress flag to view real-time transfer statistics.

rclone moveto source:path dest:path [flags]

    

Options¶

  -h, --help   help for moveto

    

rclone ncdu¶

Synopsis¶

To make the user interface it first scans the entire remote given and builds an in memory representation. rclone ncdu can be used during this scanning phase and you will see it building up the directory structure as it goes along.

Here are the keys - press '?' to toggle the help on and off

 ↑,↓ or k,j to Move
 →,l to enter
 ←,h to return
 c toggle counts
 g toggle graph
 n,s,C sort by name,size,count
 d delete file/directory
 ^L refresh screen
 ? to toggle help on and off
 q/ESC/c-C to quit

    

This an homage to the ncdu tool (https://dev.yorhel.nl/ncdu) but for rclone remotes. It is missing lots of features at the moment but is useful as it stands.

Note that it might take some time to delete big files/folders. The UI won't respond in the meantime since the deletion is done synchronously.

rclone ncdu remote:path [flags]

    

Options¶

  -h, --help   help for ncdu

    

rclone obscure¶

Synopsis¶

rclone obscure password [flags]

    

Options¶

  -h, --help   help for obscure

    

rclone rc¶

Synopsis¶

A username and password can be passed in with --user and --pass.

Note that --rc-addr, --rc-user, --rc-pass will be read also for --url, --user, --pass.

Arguments should be passed in as parameter=value.

The result will be returned as a JSON object by default.

The --json parameter can be used to pass in a JSON blob as an input instead of key=value arguments. This is the only way of passing in more complicated values.

Use "rclone rc" to see a list of all possible commands.

rclone rc commands parameter [flags]

    

Options¶

  -h, --help          help for rc
      --json string   Input JSON - use instead of key=value args.
      --no-output     If set don't output the JSON result.
      --pass string   Password to use to connect to rclone remote control.
      --url string    URL to connect to rclone remote control. (default "http://localhost:5572/")
      --user string   Username to use to rclone remote control.

    

rclone rcat¶

Synopsis¶

echo "hello world" | rclone rcat remote:path/to/file
ffmpeg - | rclone rcat remote:path/to/file

    

If the remote file already exists, it will be overwritten.

rcat will try to upload small files in a single request, which is usually more efficient than the streaming/chunked upload endpoints, which use multiple requests. Exact behaviour depends on the remote. What is considered a small file may be set through --streaming-upload-cutoff. Uploading only starts after the cutoff is reached or if the file ends before that. The data must fit into RAM. The cutoff needs to be small enough to adhere the limits of your remote, please see there. Generally speaking, setting this cutoff too high will decrease your performance.

Note that the upload can also not be retried because the data is not kept around until the upload succeeds. If you need to transfer a lot of data, you're better off caching locally and then rclone move it to the destination.

rclone rcat remote:path [flags]

    

Options¶

  -h, --help   help for rcat

    

rclone rcd¶

Synopsis¶

This is useful if you are controlling rclone via the rc API.

If you pass in a path to a directory, rclone will serve that directory for GET requests on the URL passed in. It will also open the URL in the browser when rclone is run.

See the rc documentation (https://rclone.org/rc/) for more info on the rc flags.

rclone rcd <path to files to serve>* [flags]

    

Options¶

  -h, --help   help for rcd

    

rclone rmdirs¶

Synopsis¶

If you supply the --leave-root flag, it will not remove the root directory.

This is useful for tidying up remotes that rclone has left a lot of empty directories in.

rclone rmdirs remote:path [flags]

    

Options¶

  -h, --help         help for rmdirs
      --leave-root   Do not remove root directory if empty

    

rclone serve¶

Synopsis¶

rclone serve http remote:

    

Each subcommand has its own options which you can see in their help.

rclone serve <protocol> [opts] <remote> [flags]

    

Options¶

  -h, --help   help for serve

    

rclone serve ftp¶

Synopsis¶

Server options¶

If you set --addr to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.

Authentication¶

You can set a single username and password with the --user and --pass flags.

Directory Cache¶

Alternatively, you can send a SIGHUP signal to rclone for it to flush all directory caches, regardless of how old they are. Assuming only one rclone instance is running, you can reset the cache like this:

kill -SIGHUP $(pidof rclone)

    

If you configure rclone with a remote control (/rc) then you can use rclone rc to flush the whole directory cache:

rclone rc vfs/forget

    

Or individual files or directories:

rclone rc vfs/forget file=path/to/file dir=path/to/dir

    

File Buffering¶

Each open file descriptor will try to keep the specified amount of data in memory at all times. The buffered data is bound to one file descriptor and won't be shared between multiple open file descriptors of the same file.

This flag is a upper limit for the used memory per file descriptor. The buffer will only use memory for data that is downloaded but not not yet read. If the buffer is empty, only a small amount of memory will be used. The maximum memory used by rclone for buffering can be up to --buffer-size * open files.

File Caching¶

You'll need to enable VFS caching if you want, for example, to read and write simultaneously to a file. See below for more details.

Note that the VFS cache works in addition to the cache backend and you may find that you need one or the other or both.

--cache-dir string                   Directory rclone will use for caching.
--vfs-cache-max-age duration         Max age of objects in the cache. (default 1h0m0s)
--vfs-cache-mode string              Cache mode off|minimal|writes|full (default "off")
--vfs-cache-poll-interval duration   Interval to poll the cache for stale objects. (default 1m0s)

    

If run with -vv rclone will print the location of the file cache. The files are stored in the user cache file area which is OS dependent but can be controlled with --cache-dir or setting the appropriate environment variable.

The cache has 4 different modes selected by --vfs-cache-mode. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.

Note that files are written back to the remote only when they are closed so if rclone is quit or dies with open files then these won't get written back to the remote. However they will still be in the on disk cache.

--vfs-cache-mode off¶

This will mean some operations are not possible

•

Files can't be opened for both read AND write

•

Files opened for write can't be seeked

•

Existing files opened for write must have O_TRUNC set

•

Files open for read with O_TRUNC will be opened write only

•

Files open for write only will behave as if O_TRUNC was supplied

•

Open modes O_APPEND, O_TRUNC are ignored

•

If an upload fails it can't be retried

--vfs-cache-mode minimal¶

These operations are not possible

•

Files opened for write only can't be seeked

•

Existing files opened for write must have O_TRUNC set

•

Files opened for write only will ignore O_APPEND, O_TRUNC

•

If an upload fails it can't be retried

--vfs-cache-mode writes¶

This mode should support all normal file system operations.

If an upload fails it will be retried up to --low-level-retries times.

--vfs-cache-mode full¶

This may be appropriate for your needs, or you may prefer to look at the cache backend which does a much more sophisticated job of caching, including caching directory hierarchies and chunks of files.

In this mode, unlike the others, when a file is written to the disk, it will be kept on the disk after it is written to the remote. It will be purged on a schedule according to --vfs-cache-max-age.

This mode should support all normal file system operations.

If an upload or download fails it will be retried up to --low-level-retries times.

rclone serve ftp remote:path [flags]

    

Options¶

      --addr string                        IPaddress:Port or :Port to bind server to. (default "localhost:2121")
      --dir-cache-time duration            Time to cache directory entries for. (default 5m0s)
      --gid uint32                         Override the gid field set by the filesystem. (default 502)
  -h, --help                               help for ftp
      --no-checksum                        Don't compare checksums on up/download.
      --no-modtime                         Don't read/write the modification time (can speed things up).
      --no-seek                            Don't allow seeking in files.
      --pass string                        Password for authentication. (empty value allow every password)
      --passive-port string                Passive port range to use. (default "30000-32000")
      --poll-interval duration             Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable. (default 1m0s)
      --read-only                          Mount read-only.
      --uid uint32                         Override the uid field set by the filesystem. (default 502)
      --umask int                          Override the permission bits set by the filesystem. (default 2)
      --user string                        User name for authentication. (default "anonymous")
      --vfs-cache-max-age duration         Max age of objects in the cache. (default 1h0m0s)
      --vfs-cache-mode string              Cache mode off|minimal|writes|full (default "off")
      --vfs-cache-poll-interval duration   Interval to poll the cache for stale objects. (default 1m0s)
      --vfs-read-chunk-size int            Read the source objects in chunks. (default 128M)
      --vfs-read-chunk-size-limit int      If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)

    

rclone serve http¶

Synopsis¶

You can use the filter flags (eg --include, --exclude) to control what is served.

The server will log errors. Use -v to see access logs.

--bwlimit will be respected for file transfers. Use --stats to control the stats printing.

Server options¶

If you set --addr to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.

--server-read-timeout and --server-write-timeout can be used to control the timeouts on the server. Note that this is the total time for a transfer.

--max-header-bytes controls the maximum number of bytes the server will accept in the HTTP header.

Authentication¶

You can either use an htpasswd file which can take lots of users, or set a single username and password with the --user and --pass flags.

Use --htpasswd /path/to/htpasswd to provide an htpasswd file. This is in standard apache format and supports MD5, SHA1 and BCrypt for basic authentication. Bcrypt is recommended.

To create an htpasswd file:

touch htpasswd
htpasswd -B htpasswd user
htpasswd -B htpasswd anotherUser

    

The password file can be updated while rclone is running.

Use --realm to set the authentication realm.

SSL/TLS¶

--cert should be a either a PEM encoded certificate or a concatenation of that with the CA certificate. --key should be the PEM encoded private key and --client-ca should be the PEM encoded client certificate authority certificate.

Directory Cache¶

Alternatively, you can send a SIGHUP signal to rclone for it to flush all directory caches, regardless of how old they are. Assuming only one rclone instance is running, you can reset the cache like this:

kill -SIGHUP $(pidof rclone)

    

If you configure rclone with a remote control (/rc) then you can use rclone rc to flush the whole directory cache:

rclone rc vfs/forget

    

Or individual files or directories:

rclone rc vfs/forget file=path/to/file dir=path/to/dir

    

File Buffering¶

Each open file descriptor will try to keep the specified amount of data in memory at all times. The buffered data is bound to one file descriptor and won't be shared between multiple open file descriptors of the same file.

This flag is a upper limit for the used memory per file descriptor. The buffer will only use memory for data that is downloaded but not not yet read. If the buffer is empty, only a small amount of memory will be used. The maximum memory used by rclone for buffering can be up to --buffer-size * open files.

File Caching¶

You'll need to enable VFS caching if you want, for example, to read and write simultaneously to a file. See below for more details.

Note that the VFS cache works in addition to the cache backend and you may find that you need one or the other or both.

--cache-dir string                   Directory rclone will use for caching.
--vfs-cache-max-age duration         Max age of objects in the cache. (default 1h0m0s)
--vfs-cache-mode string              Cache mode off|minimal|writes|full (default "off")
--vfs-cache-poll-interval duration   Interval to poll the cache for stale objects. (default 1m0s)

    

If run with -vv rclone will print the location of the file cache. The files are stored in the user cache file area which is OS dependent but can be controlled with --cache-dir or setting the appropriate environment variable.

The cache has 4 different modes selected by --vfs-cache-mode. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.

Note that files are written back to the remote only when they are closed so if rclone is quit or dies with open files then these won't get written back to the remote. However they will still be in the on disk cache.

--vfs-cache-mode off¶

This will mean some operations are not possible

•

Files can't be opened for both read AND write

•

Files opened for write can't be seeked

•

Existing files opened for write must have O_TRUNC set

•

Files open for read with O_TRUNC will be opened write only

•

Files open for write only will behave as if O_TRUNC was supplied

•

Open modes O_APPEND, O_TRUNC are ignored

•

If an upload fails it can't be retried

--vfs-cache-mode minimal¶

These operations are not possible

•

Files opened for write only can't be seeked

•

Existing files opened for write must have O_TRUNC set

•

Files opened for write only will ignore O_APPEND, O_TRUNC

•

If an upload fails it can't be retried

--vfs-cache-mode writes¶

This mode should support all normal file system operations.

If an upload fails it will be retried up to --low-level-retries times.

--vfs-cache-mode full¶

This may be appropriate for your needs, or you may prefer to look at the cache backend which does a much more sophisticated job of caching, including caching directory hierarchies and chunks of files.

In this mode, unlike the others, when a file is written to the disk, it will be kept on the disk after it is written to the remote. It will be purged on a schedule according to --vfs-cache-max-age.

This mode should support all normal file system operations.

If an upload or download fails it will be retried up to --low-level-retries times.

rclone serve http remote:path [flags]

    

Options¶

      --addr string                        IPaddress:Port or :Port to bind server to. (default "localhost:8080")
      --cert string                        SSL PEM key (concatenation of certificate and CA certificate)
      --client-ca string                   Client certificate authority to verify clients with
      --dir-cache-time duration            Time to cache directory entries for. (default 5m0s)
      --gid uint32                         Override the gid field set by the filesystem. (default 502)
  -h, --help                               help for http
      --htpasswd string                    htpasswd file - if not provided no authentication is done
      --key string                         SSL PEM Private key
      --max-header-bytes int               Maximum size of request header (default 4096)
      --no-checksum                        Don't compare checksums on up/download.
      --no-modtime                         Don't read/write the modification time (can speed things up).
      --no-seek                            Don't allow seeking in files.
      --pass string                        Password for authentication.
      --poll-interval duration             Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable. (default 1m0s)
      --read-only                          Mount read-only.
      --realm string                       realm for authentication (default "rclone")
      --server-read-timeout duration       Timeout for server reading data (default 1h0m0s)
      --server-write-timeout duration      Timeout for server writing data (default 1h0m0s)
      --uid uint32                         Override the uid field set by the filesystem. (default 502)
      --umask int                          Override the permission bits set by the filesystem. (default 2)
      --user string                        User name for authentication.
      --vfs-cache-max-age duration         Max age of objects in the cache. (default 1h0m0s)
      --vfs-cache-mode string              Cache mode off|minimal|writes|full (default "off")
      --vfs-cache-poll-interval duration   Interval to poll the cache for stale objects. (default 1m0s)
      --vfs-read-chunk-size int            Read the source objects in chunks. (default 128M)
      --vfs-read-chunk-size-limit int      If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)

    

rclone serve restic¶

Synopsis¶

Restic (https://restic.net/) is a command line program for doing backups.

The server will log errors. Use -v to see access logs.

--bwlimit will be respected for file transfers. Use --stats to control the stats printing.

Setting up rclone for use by restic¶

Once you have set up the remote, check it is working with, for example "rclone lsd remote:". You may have called the remote something other than "remote:" - just substitute whatever you called it in the following instructions.

Now start the rclone restic server

rclone serve restic -v remote:backup

    

Where you can replace "backup" in the above by whatever path in the remote you wish to use.

By default this will serve on "localhost:8080" you can change this with use of the "--addr" flag.

You might wish to start this server on boot.

Setting up restic to use rclone¶

Note that you will need restic 0.8.2 or later to interoperate with rclone.

For the example above you will want to use "http://localhost:8080/" as the URL for the REST server.

For example:

$ export RESTIC_REPOSITORY=rest:http://localhost:8080/
$ export RESTIC_PASSWORD=yourpassword
$ restic init
created restic backend 8b1a4b56ae at rest:http://localhost:8080/
Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
$ restic backup /path/to/files/to/backup
scan [/path/to/files/to/backup]
scanned 189 directories, 312 files in 0:00
[0:00] 100.00%  38.128 MiB / 38.128 MiB  501 / 501 items  0 errors  ETA 0:00 
duration: 0:00
snapshot 45c8fdd8 saved

    

Multiple repositories¶

$ export RESTIC_REPOSITORY=rest:http://localhost:8080/user1repo/
# backup user1 stuff
$ export RESTIC_REPOSITORY=rest:http://localhost:8080/user2repo/
# backup user2 stuff

    

Server options¶

If you set --addr to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.

--server-read-timeout and --server-write-timeout can be used to control the timeouts on the server. Note that this is the total time for a transfer.

--max-header-bytes controls the maximum number of bytes the server will accept in the HTTP header.

Authentication¶

You can either use an htpasswd file which can take lots of users, or set a single username and password with the --user and --pass flags.

Use --htpasswd /path/to/htpasswd to provide an htpasswd file. This is in standard apache format and supports MD5, SHA1 and BCrypt for basic authentication. Bcrypt is recommended.

To create an htpasswd file:

touch htpasswd
htpasswd -B htpasswd user
htpasswd -B htpasswd anotherUser

    

The password file can be updated while rclone is running.

Use --realm to set the authentication realm.

SSL/TLS¶

--cert should be a either a PEM encoded certificate or a concatenation of that with the CA certificate. --key should be the PEM encoded private key and --client-ca should be the PEM encoded client certificate authority certificate.

rclone serve restic remote:path [flags]

    

Options¶

      --addr string                     IPaddress:Port or :Port to bind server to. (default "localhost:8080")
      --append-only                     disallow deletion of repository data
      --cert string                     SSL PEM key (concatenation of certificate and CA certificate)
      --client-ca string                Client certificate authority to verify clients with
  -h, --help                            help for restic
      --htpasswd string                 htpasswd file - if not provided no authentication is done
      --key string                      SSL PEM Private key
      --max-header-bytes int            Maximum size of request header (default 4096)
      --pass string                     Password for authentication.
      --realm string                    realm for authentication (default "rclone")
      --server-read-timeout duration    Timeout for server reading data (default 1h0m0s)
      --server-write-timeout duration   Timeout for server writing data (default 1h0m0s)
      --stdio                           run an HTTP2 server on stdin/stdout
      --user string                     User name for authentication.

    

rclone serve webdav¶

Synopsis¶

Webdav options¶

--etag-hash¶

If this flag is set to "auto" then rclone will choose the first supported hash on the backend or you can use a named hash such as "MD5" or "SHA-1".

Use "rclone hashsum" to see the full list.

Server options¶

If you set --addr to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.

--server-read-timeout and --server-write-timeout can be used to control the timeouts on the server. Note that this is the total time for a transfer.

--max-header-bytes controls the maximum number of bytes the server will accept in the HTTP header.

Authentication¶

You can either use an htpasswd file which can take lots of users, or set a single username and password with the --user and --pass flags.

Use --htpasswd /path/to/htpasswd to provide an htpasswd file. This is in standard apache format and supports MD5, SHA1 and BCrypt for basic authentication. Bcrypt is recommended.

To create an htpasswd file:

touch htpasswd
htpasswd -B htpasswd user
htpasswd -B htpasswd anotherUser

    

The password file can be updated while rclone is running.

Use --realm to set the authentication realm.

SSL/TLS¶

--cert should be a either a PEM encoded certificate or a concatenation of that with the CA certificate. --key should be the PEM encoded private key and --client-ca should be the PEM encoded client certificate authority certificate.

Directory Cache¶

Alternatively, you can send a SIGHUP signal to rclone for it to flush all directory caches, regardless of how old they are. Assuming only one rclone instance is running, you can reset the cache like this:

kill -SIGHUP $(pidof rclone)

    

If you configure rclone with a remote control (/rc) then you can use rclone rc to flush the whole directory cache:

rclone rc vfs/forget

    

Or individual files or directories:

rclone rc vfs/forget file=path/to/file dir=path/to/dir

    

File Buffering¶

Each open file descriptor will try to keep the specified amount of data in memory at all times. The buffered data is bound to one file descriptor and won't be shared between multiple open file descriptors of the same file.

This flag is a upper limit for the used memory per file descriptor. The buffer will only use memory for data that is downloaded but not not yet read. If the buffer is empty, only a small amount of memory will be used. The maximum memory used by rclone for buffering can be up to --buffer-size * open files.

File Caching¶

You'll need to enable VFS caching if you want, for example, to read and write simultaneously to a file. See below for more details.

Note that the VFS cache works in addition to the cache backend and you may find that you need one or the other or both.

--cache-dir string                   Directory rclone will use for caching.
--vfs-cache-max-age duration         Max age of objects in the cache. (default 1h0m0s)
--vfs-cache-mode string              Cache mode off|minimal|writes|full (default "off")
--vfs-cache-poll-interval duration   Interval to poll the cache for stale objects. (default 1m0s)

    

If run with -vv rclone will print the location of the file cache. The files are stored in the user cache file area which is OS dependent but can be controlled with --cache-dir or setting the appropriate environment variable.

The cache has 4 different modes selected by --vfs-cache-mode. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.

Note that files are written back to the remote only when they are closed so if rclone is quit or dies with open files then these won't get written back to the remote. However they will still be in the on disk cache.

--vfs-cache-mode off¶

This will mean some operations are not possible

•

Files can't be opened for both read AND write

•

Files opened for write can't be seeked

•

Existing files opened for write must have O_TRUNC set

•

Files open for read with O_TRUNC will be opened write only

•

Files open for write only will behave as if O_TRUNC was supplied

•

Open modes O_APPEND, O_TRUNC are ignored

•

If an upload fails it can't be retried

--vfs-cache-mode minimal¶

These operations are not possible

•

Files opened for write only can't be seeked

•

Existing files opened for write must have O_TRUNC set

•

Files opened for write only will ignore O_APPEND, O_TRUNC

•

If an upload fails it can't be retried

--vfs-cache-mode writes¶

This mode should support all normal file system operations.

If an upload fails it will be retried up to --low-level-retries times.

--vfs-cache-mode full¶

This may be appropriate for your needs, or you may prefer to look at the cache backend which does a much more sophisticated job of caching, including caching directory hierarchies and chunks of files.

In this mode, unlike the others, when a file is written to the disk, it will be kept on the disk after it is written to the remote. It will be purged on a schedule according to --vfs-cache-max-age.

This mode should support all normal file system operations.

If an upload or download fails it will be retried up to --low-level-retries times.

rclone serve webdav remote:path [flags]

    

Options¶

      --addr string                        IPaddress:Port or :Port to bind server to. (default "localhost:8080")
      --cert string                        SSL PEM key (concatenation of certificate and CA certificate)
      --client-ca string                   Client certificate authority to verify clients with
      --dir-cache-time duration            Time to cache directory entries for. (default 5m0s)
      --etag-hash string                   Which hash to use for the ETag, or auto or blank for off
      --gid uint32                         Override the gid field set by the filesystem. (default 502)
  -h, --help                               help for webdav
      --htpasswd string                    htpasswd file - if not provided no authentication is done
      --key string                         SSL PEM Private key
      --max-header-bytes int               Maximum size of request header (default 4096)
      --no-checksum                        Don't compare checksums on up/download.
      --no-modtime                         Don't read/write the modification time (can speed things up).
      --no-seek                            Don't allow seeking in files.
      --pass string                        Password for authentication.
      --poll-interval duration             Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable. (default 1m0s)
      --read-only                          Mount read-only.
      --realm string                       realm for authentication (default "rclone")
      --server-read-timeout duration       Timeout for server reading data (default 1h0m0s)
      --server-write-timeout duration      Timeout for server writing data (default 1h0m0s)
      --uid uint32                         Override the uid field set by the filesystem. (default 502)
      --umask int                          Override the permission bits set by the filesystem. (default 2)
      --user string                        User name for authentication.
      --vfs-cache-max-age duration         Max age of objects in the cache. (default 1h0m0s)
      --vfs-cache-mode string              Cache mode off|minimal|writes|full (default "off")
      --vfs-cache-poll-interval duration   Interval to poll the cache for stale objects. (default 1m0s)
      --vfs-read-chunk-size int            Read the source objects in chunks. (default 128M)
      --vfs-read-chunk-size-limit int      If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)

    

rclone settier¶

Synopsis¶

Note that, certain tier chages make objects not available to access immediately. For example tiering to archive in azure blob storage makes objects in frozen state, user can restore by setting tier to Hot/Cool, similarly S3 to Glacier makes object inaccessible.true

You can use it to tier single object

rclone settier Cool remote:path/file

    

Or use rclone filters to set tier on only specific files

rclone --include "*.txt" settier Hot remote:path/dir

    

Or just provide remote directory and all files in directory will be tiered

rclone settier tier remote:path/dir

    

rclone settier tier remote:path [flags]

    

Options¶

  -h, --help   help for settier

    

rclone touch¶

Synopsis¶

rclone touch remote:path [flags]

    

Options¶

  -h, --help               help for touch
  -C, --no-create          Do not create the file if it does not exist.
  -t, --timestamp string   Change the modification times to the specified time instead of the current time of day. The argument is of the form 'YYMMDD' (ex. 17.10.30) or 'YYYY-MM-DDTHH:MM:SS' (ex. 2006-01-02T15:04:05)

    

rclone tree¶

Synopsis¶

For example

$ rclone tree remote:path
/
├── file1
├── file2
├── file3
└── subdir
    ├── file4
    └── file5
1 directories, 5 files

    

You can use any of the filtering options with the tree command (eg --include and --exclude). You can also use --fast-list.

The tree command has many options for controlling the listing which are compatible with the tree command. Note that not all of them have short options as they conflict with rclone's short options.

rclone tree remote:path [flags]

    

Options¶

  -a, --all             All files are listed (list . files too).
  -C, --color           Turn colorization on always.
  -d, --dirs-only       List directories only.
      --dirsfirst       List directories before files (-U disables).
      --full-path       Print the full path prefix for each file.
  -h, --help            help for tree
      --human           Print the size in a more human readable way.
      --level int       Descend only level directories deep.
  -D, --modtime         Print the date of last modification.
  -i, --noindent        Don't print indentation lines.
      --noreport        Turn off file/directory count at end of tree listing.
  -o, --output string   Output to file instead of stdout.
  -p, --protections     Print the protections for each file.
  -Q, --quote           Quote filenames with double quotes.
  -s, --size            Print the size in bytes of each file.
      --sort string     Select sort: name,version,size,mtime,ctime.
      --sort-ctime      Sort files by last status change time.
  -t, --sort-modtime    Sort files by last modification time.
  -r, --sort-reverse    Reverse the order of the sort.
  -U, --unsorted        Leave files unsorted.
      --version         Sort files alphanumerically by version.

    

Copying single files¶

For example, suppose you have a remote with a file in called test.jpg, then you could copy just that file like this

rclone copy remote:test.jpg /tmp/download

    

The file test.jpg will be placed inside /tmp/download.

This is equivalent to specifying

rclone copy --files-from /tmp/files remote: /tmp/download

    

Where /tmp/files contains the single line

test.jpg

    

It is recommended to use copy when copying individual files, not sync. They have pretty much the same effect but copy will use a lot less memory.

Syntax of remote paths¶

/path/to/dir¶

On Windows only \ may be used instead of / in local paths only, non local paths must use /.

These paths needn't start with a leading / - if they don't then they will be relative to the current directory.

remote:path/to/dir¶

remote:/path/to/dir¶

:backend:path/to/dir¶

Eg

rclone lsd --http-url https://pub.rclone.org :http:

    

Which lists all the directories in pub.rclone.org.

Quoting and the shell¶

Here are some gotchas which may help users unfamiliar with the shell rules

Linux / OSX¶

rclone copy 'Important files?' remote:backup

    

If you want to send a ' you will need to use ", eg

rclone copy "O'Reilly Reviews" remote:backup

    

The rules for quoting metacharacters are complicated and if you want the full details you'll have to consult the manual page for your shell.

Windows¶

rclone copy "E:\folder name\folder name\folder name" remote:backup

    

If you are using the root directory on its own then don't quote it (see #464 (https://github.com/ncw/rclone/issues/464) for why), eg

rclone copy E:\ remote:backup

    

Copying files or directories with : in the names¶

So to sync a directory called sync:me to a remote called remote: use

rclone sync ./sync:me remote:path

    

or

rclone sync /full/path/to/sync:me remote:path

    

Server Side Copy¶

This means if you want to copy one folder to another then rclone won't download all the files and re-upload them; it will instruct the server to copy them in place.

Eg

rclone copy s3:oldbucket s3:newbucket

    

Will copy the contents of oldbucket to newbucket without downloading and re-uploading.

Remotes which don't support server side copy will download and re-upload in this case.

Server side copies are used with sync and copy and will be identified in the log when using the -v flag. The move command may also use them if remote doesn't support server side move directly. This is done by issuing a server side copy then a delete which is much quicker than a download and re-upload.

Server side copies will only be attempted if the remote names are the same.

This can be used when scripting to make aged backups efficiently, eg

rclone sync remote:current-backup remote:previous-backup
rclone sync /path/to/files remote:current-backup

    

Options¶

Options which use TIME use the go time parser. A duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as "300ms", "-1.5h" or "2h45m". Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".

Options which use SIZE use kByte by default. However, a suffix of b for bytes, k for kBytes, M for MBytes, G for GBytes, T for TBytes and P for PBytes may be used. These are the binary units, eg 1, 2**10, 2**20, 2**30 respectively.

--backup-dir=DIR¶

If --suffix is set, then the moved files will have the suffix added to them. If there is a file with the same path (after the suffix has been added) in DIR, then it will be overwritten.

The remote in use must support server side move or copy and you must use the same remote as the destination of the sync. The backup directory must not overlap the destination directory.

For example

rclone sync /path/to/local remote:current --backup-dir remote:old

    

will sync /path/to/local to remote:current, but for any files which would have been updated or deleted will be stored in remote:old.

If running rclone from a script you might want to use today's date as the directory name passed to --backup-dir to store the old files, or you might want to pass --suffix with today's date.

--bind string¶

--bwlimit=BANDWIDTH_SPEC¶

Single limits last for the duration of the session. To use a single limit, specify the desired bandwidth in kBytes/s, or use a suffix b|k|M|G. The default is 0 which means to not limit bandwidth.

For example, to limit bandwidth usage to 10 MBytes/s use --bwlimit 10M

It is also possible to specify a "timetable" of limits, which will cause certain limits to be applied at certain times. To specify a timetable, format your entries as "WEEKDAY-HH:MM,BANDWIDTH WEEKDAY-HH:MM,BANDWIDTH..." where: WEEKDAY is optional element. It could be writen as whole world or only using 3 first characters. HH:MM is an hour from 00:00 to 23:59.

An example of a typical timetable to avoid link saturation during daytime working hours could be:

--bwlimit "08:00,512 12:00,10M 13:00,512 18:00,30M 23:00,off"

In this example, the transfer bandwidth will be every day set to 512kBytes/sec at 8am. At noon, it will raise to 10Mbytes/s, and drop back to 512kBytes/sec at 1pm. At 6pm, the bandwidth limit will be set to 30MBytes/s, and at 11pm it will be completely disabled (full speed). Anything between 11pm and 8am will remain unlimited.

An example of timetable with WEEKDAY could be:

--bwlimit "Mon-00:00,512 Fri-23:59,10M Sat-10:00,1M Sun-20:00,off"

It mean that, the transfer bandwidh will be set to 512kBytes/sec on Monday. It will raise to 10Mbytes/s before the end of Friday. At 10:00 on Sunday it will be set to 1Mbyte/s. From 20:00 at Sunday will be unlimited.

Timeslots without weekday are extended to whole week. So this one example:

--bwlimit "Mon-00:00,512 12:00,1M Sun-20:00,off"

Is equal to this:

--bwlimit "Mon-00:00,512Mon-12:00,1M Tue-12:00,1M Wed-12:00,1M Thu-12:00,1M Fri-12:00,1M Sat-12:00,1M Sun-12:00,1M Sun-20:00,off"

Bandwidth limits only apply to the data transfer. They don't apply to the bandwidth of the directory listings etc.

Note that the units are Bytes/s, not Bits/s. Typically connections are measured in Bits/s - to convert divide by 8. For example, let's say you have a 10 Mbit/s connection and you wish rclone to use half of it - 5 Mbit/s. This is 5/8 = 0.625MByte/s so you would use a --bwlimit 0.625M parameter for rclone.

On Unix systems (Linux, MacOS, ...) the bandwidth limiter can be toggled by sending a SIGUSR2 signal to rclone. This allows to remove the limitations of a long running rclone transfer and to restore it back to the value specified with --bwlimit quickly when needed. Assuming there is only one rclone instance running, you can toggle the limiter like this:

kill -SIGUSR2 $(pidof rclone)

    

If you configure rclone with a remote control (/rc) then you can use change the bwlimit dynamically:

rclone rc core/bwlimit rate=1M

    

--buffer-size=SIZE¶

When using mount or cmount each open file descriptor will use this much memory for buffering. See the mount (/commands/rclone_mount/#file-buffering) documentation for more details.

Set to 0 to disable the buffering for the minimum memory usage.

--checkers=N¶

The default is to run 8 checkers in parallel.

-c, --checksum¶

This is useful when the remote doesn't support setting modified time and a more accurate sync is desired than just checking the file size.

This is very useful when transferring between remotes which store the same hash type on the object, eg Drive and Swift. For details of which remotes support which hash type see the table in the overview section (https://rclone.org/overview/).

Eg rclone --checksum sync s3:/bucket swift:/bucket would run much quicker than without the --checksum flag.

When using this flag, rclone won't update mtimes of remote files if they are incorrect as it would normally.

--config=CONFIG_FILE¶

Normally the config file is in your home directory as a file called .config/rclone/rclone.conf (or .rclone.conf if created with an older version). If $XDG_CONFIG_HOME is set it will be at $XDG_CONFIG_HOME/rclone/rclone.conf

If you run rclone -h and look at the help for the --config option you will see where the default location is for you.

Use this flag to override the config location, eg rclone --config=".myconfig" .config.

--contimeout=TIME¶

The connection timeout is the amount of time rclone will wait for a connection to go through to a remote object storage system. It is 1m by default.

--dedupe-mode MODE¶

--disable FEATURE,FEATURE,...¶

--disable move,copy

    

The features can be put in in any case.

To see a list of which features can be disabled use:

--disable help

    

See the overview features (/overview/#features) and optional features (/overview/#optional-features) to get an idea of which feature does what.

This flag can be useful for debugging and in exceptional circumstances (eg Google Drive limiting the total volume of Server Side Copies to 100GB/day).

-n, --dry-run¶

--ignore-checksum¶

You can use this option to skip that check. You should only use it if you have had the "corrupted on transfer" error message and you are sure you might want to transfer potentially corrupted data.

--ignore-existing¶

While this isn't a generally recommended option, it can be useful in cases where your files change due to encryption. However, it cannot correct partial transfers in case a transfer was interrupted.

--ignore-size¶

It will also cause rclone to skip verifying the sizes are the same after transfer.

This can be useful for transferring files to and from OneDrive which occasionally misreports the size of image files (see #399 (https://github.com/ncw/rclone/issues/399) for more info).

-I, --ignore-times¶

Normally rclone would skip any files that have the same modification time and are the same size (or have the same checksum if using --checksum).

--immutable¶

With this option set, files will be created and deleted as requested, but existing files will never be updated. If an existing file does not match between the source and destination, rclone will give the error Source and destination exist but do not match: immutable file modified.

Note that only commands which transfer files (e.g. sync, copy, move) are affected by this behavior, and only modification is disallowed. Files may still be deleted explicitly (e.g. delete, purge) or implicitly (e.g. sync, move). Use copy --immutable if it is desired to avoid deletion as well as modification.

This can be useful as an additional layer of protection for immutable or append-only data sets (notably backup archives), where modification implies corruption and should not be propagated.

--leave-root¶

--log-file=FILE¶

Note that if you are using the logrotate program to manage rclone's logs, then you should use the copytruncate option as rclone doesn't have a signal to rotate logs.

--log-format LIST¶

--log-level LEVEL¶

DEBUG is equivalent to -vv. It outputs lots of debug info - useful for bug reports and really finding out what rclone is doing.

INFO is equivalent to -v. It outputs information about each transfer and prints stats once a minute by default.

NOTICE is the default log level if no logging flags are supplied. It outputs very little when things are working normally. It outputs warnings and significant events.

ERROR is equivalent to -q. It only outputs error messages.

--low-level-retries NUMBER¶

A low level retry is used to retry a failing operation - typically one HTTP request. This might be uploading a chunk of a big file for example. You will see low level retries in the log with the -v flag.

This shouldn't need to be changed from the default in normal operations. However, if you get a lot of low level retries you may wish to reduce the value so rclone moves on to a high level retry (see the --retries flag) quicker.

Disable low level retries with --low-level-retries 1.

--max-backlog=N¶

This can be set arbitrarily large. It will only use memory when the queue is in use. Note that it will use in the order of N kB of memory when the backlog is in use.

Setting this large allows rclone to calculate how many files are pending more accurately and give a more accurate estimated finish time.

Setting this small will make rclone more synchronous to the listings of the remote which may be desirable.

--max-delete=N¶

--max-depth=N¶

So if you do rclone --max-depth 1 ls remote:path you will see only the files in the top level directory. Using --max-depth 2 means you will see all the files in first two directory levels and so on.

For historical reasons the lsd command defaults to using a --max-depth of 1 - you can override this with the command line flag.

You can use this command to disable recursion (with --max-depth 1).

Note that if you use this with sync and --delete-excluded the files not recursed through are considered excluded and will be deleted on the destination. Test first with --dry-run if you are not sure what will happen.

--max-transfer=SIZE¶

When the limit is reached all transfers will stop immediately.

Rclone will exit with exit code 8 if the transfer limit is reached.

--modify-window=TIME¶

The default is 1ns unless this is overridden by a remote. For example OS X only stores modification times to the nearest second so if you are reading and writing to an OS X filing system this will be 1s by default.

This command line flag allows you to override that computed default.

--no-gzip-encoding¶

There is no need to set this in normal operation, and doing so will decrease the network transfer efficiency of rclone.

--no-update-modtime¶

This can be used if the remote is being synced with another tool also (eg the Google Drive client).

-P, --progress¶

Any log messages will scroll above the static block. Log messages will push the static block down to the bottom of the terminal where it will stay.

Normally this is updated every 500mS but this period can be overridden with the --stats flag.

This can be used with the --stats-one-line flag for a simpler display.

Note: On Windows untilthis bug (https://github.com/Azure/go-ansiterm/issues/26) is fixed all non-ASCII characters will be replaced with . when --progress is in use.

-q, --quiet¶

--retries int¶

Some remotes can be unreliable and a few retries help pick up the files which didn't get transferred because of errors.

Disable retries with --retries 1.

--retries-sleep=TIME¶

The default is 0. Use 0 to disable.

--size-only¶

This can be useful transferring files from Dropbox which have been modified by the desktop sync client which doesn't set checksums of modification times in the same way as rclone.

--stats=TIME¶

This sets the interval.

The default is 1m. Use 0 to disable.

If you set the stats interval then all commands can show stats. This can be useful when running other commands, check or mount for example.

Stats are logged at INFO level by default which means they won't show at default log level NOTICE. Use --stats-log-level NOTICE or -v to make them show. See the Logging section (#logging) for more info on log levels.

Note that on macOS you can send a SIGINFO (which is normally ctrl-T in the terminal) to make the stats print immediately.

--stats-file-name-length integer¶

--stats-log-level string¶

--stats-one-line¶

--stats-unit=bits|bytes¶

This option allows the data rate to be printed in bits/second.

Data transfer volume will still be reported in bytes.

The rate is reported as a binary unit, not SI unit. So 1 Mbit/s equals 1,048,576 bits/s and not 1,000,000 bits/s.

The default is bytes.

--suffix=SUFFIX¶

See --backup-dir for more info.

--syslog¶

This can be useful for running rclone in a script or rclone mount.

--syslog-facility string¶

--tpslimit float¶

For example to limit rclone to 10 HTTP transactions per second use --tpslimit 10, or to 1 transaction every 2 seconds use --tpslimit 0.5.

Use this when the number of transactions per second from rclone is causing a problem with the cloud storage provider (eg getting you banned or rate limited).

This can be very useful for rclone mount to control the behaviour of applications using it.

See also --tpslimit-burst.

--tpslimit-burst int¶

Normally --tpslimit will do exactly the number of transaction per second specified. However if you supply --tps-burst then rclone can save up some transactions from when it was idle giving a burst of up to the parameter supplied.

For example if you provide --tpslimit-burst 10 then if rclone has been idle for more than 10*--tpslimit then it can do 10 transactions very quickly before they are limited again.

This may be used to increase performance of --tpslimit without changing the long term average number of transactions per second.

--track-renames¶

If you use this flag, and the remote supports server side copy or server side move, and the source and destination have a compatible hash, then this will track renames during sync operations and perform renaming server-side.

Files will be matched by size and hash - if both match then a rename will be considered.

If the destination does not support server-side copy or move, rclone will fall back to the default behaviour and log an error level message to the console. Note: Encrypted destinations are not supported by --track-renames.

Note that --track-renames uses extra memory to keep track of all the rename candidates.

Note also that --track-renames is incompatible with --delete-before and will select --delete-after instead of --delete-during.

--delete-(before,during,after)¶

Specifying the value --delete-before will delete all files present on the destination, but not on the source before starting the transfer of any new or updated files. This uses two passes through the file systems, one for the deletions and one for the copies.

Specifying --delete-during will delete files while checking and uploading files. This is the fastest option and uses the least memory.

Specifying --delete-after (the default value) will delay deletion of files until all new/updated files have been successfully transferred. The files to be deleted are collected in the copy pass then deleted after the copy pass has completed successfully. The files to be deleted are held in memory so this mode may use more memory. This is the safest mode as it will only delete files if there have been no errors subsequent to that. If there have been errors before the deletions start then you will get the message not deleting files as there were IO errors.

--fast-list¶

However, some remotes have a way of listing all files beneath a directory in one (or a small number) of transactions. These tend to be the bucket based remotes (eg S3, B2, GCS, Swift, Hubic).

If you use the --fast-list flag then rclone will use this method for listing directories. This will have the following consequences for the listing:

•

It will use fewer transactions (important if you pay for them)

•

It will use more memory. Rclone has to load the whole listing into memory.

•

It may be faster because it uses fewer transactions

•

It may be slower because it can't be parallelized

rclone should always give identical results with and without --fast-list.

If you pay for transactions and can fit your entire sync listing into memory then --fast-list is recommended. If you have a very big sync to do then don't use --fast-list otherwise you will run out of memory.

If you use --fast-list on a remote which doesn't support it, then rclone will just ignore it.

--timeout=TIME¶

The default is 5m. Set to 0 to disable.

--transfers=N¶

The default is to run 4 file transfers in parallel.

-u, --update¶

If an existing destination file has a modification time equal (within the computed modify window precision) to the source file's, it will be updated if the sizes are different.

On remotes which don't support mod time directly the time checked will be the uploaded time. This means that if uploading to one of these remotes, rclone will skip any files which exist on the destination and have an uploaded time that is newer than the modification time of the source file.

This can be useful when transferring to a remote which doesn't support mod times directly as it is more accurate than a --size-only check and faster than using --checksum.

--use-server-modtime¶

Use this flag to disable the extra API call and rely instead on the server's modified time. In cases such as a local to remote sync, knowing the local file is newer than the time it was last uploaded to the remote is sufficient. In those cases, this flag can speed up the process and reduce the number of API calls necessary.

-v, -vv, --verbose¶

With -vv rclone will become very verbose telling you about every file it considers and transfers. Please send bug reports with a log with this setting.

-V, --version¶

Configuration Encryption¶

If you are in an environment where that isn't possible, you can add a password to your configuration. This means that you will have to enter the password every time you start rclone.

To add a password to your rclone configuration, execute rclone config.

>rclone config
Current remotes:
e) Edit existing remote
n) New remote
d) Delete remote
s) Set configuration password
q) Quit config
e/n/d/s/q>

    

Go into s, Set configuration password:

e/n/d/s/q> s
Your configuration is not encrypted.
If you add a password, you will protect your login information to cloud services.
a) Add Password
q) Quit to main menu
a/q> a
Enter NEW configuration password:
password:
Confirm NEW password:
password:
Password set
Your configuration is encrypted.
c) Change Password
u) Unencrypt configuration
q) Quit to main menu
c/u/q>

    

Your configuration is now encrypted, and every time you start rclone you will now be asked for the password. In the same menu, you can change the password or completely remove encryption from your configuration.

There is no way to recover the configuration if you lose your password.

rclone uses nacl secretbox (https://godoc.org/golang.org/x/crypto/nacl/secretbox) which in turn uses XSalsa20 and Poly1305 to encrypt and authenticate your configuration with secret-key cryptography. The password is SHA-256 hashed, which produces the key for secretbox. The hashed password is not stored.

While this provides very good security, we do not recommend storing your encrypted rclone configuration in public if it contains sensitive information, maybe except if you use a very strong password.

If it is safe in your environment, you can set the RCLONE_CONFIG_PASS environment variable to contain your password, in which case it will be used for decrypting the configuration.

You can set this for a session from a script. For unix like systems save this to a file called set-rclone-password:

#!/bin/echo Source this file don't run it
read -s RCLONE_CONFIG_PASS
export RCLONE_CONFIG_PASS

    

Then source the file when you want to use it. From the shell you would do source set-rclone-password. It will then ask you for the password and set it in the environment variable.

If you are running rclone inside a script, you might want to disable password prompts. To do that, pass the parameter --ask-password=false to rclone. This will make rclone fail instead of asking for a password if RCLONE_CONFIG_PASS doesn't contain a valid password.

Developer options¶

--cpuprofile=FILE¶

--dump flag,flag,flag¶

--dump headers¶

Use --dump auth if you do want the Authorization: headers.

--dump bodies¶

Note that the bodies are buffered in memory so don't use this for enormous files.

--dump requests¶

--dump responses¶

--dump auth¶

--dump filters¶

--dump goroutines¶

--dump openfiles¶

--memprofile=FILE¶

--no-check-certificate=true/false¶

This option defaults to false.

This should be used only for testing.

Filtering¶

•

--delete-excluded

•

--filter

•

--filter-from

•

--exclude

•

--exclude-from

•

--include

•

--include-from

•

--files-from

•

--min-size

•

--max-size

•

--min-age

•

--max-age

•

--dump filters

See the filtering section (https://rclone.org/filtering/).

Remote control¶

•

--rc

•

and anything starting with --rc-

See the remote control section (https://rclone.org/rc/).

Logging¶

By default, rclone logs to standard error. This means you can redirect standard error and still see the normal output of rclone commands (eg rclone ls).

By default, rclone will produce Error and Notice level messages.

If you use the -q flag, rclone will only produce Error messages.

If you use the -v flag, rclone will produce Error, Notice and Info messages.

If you use the -vv flag, rclone will produce Error, Notice, Info and Debug messages.

You can also control the log levels with the --log-level flag.

If you use the --log-file=FILE option, rclone will redirect Error, Info and Debug messages along with standard error to FILE.

If you use the --syslog flag then rclone will log to syslog and the --syslog-facility control which facility it uses.

Rclone prefixes all log messages with their level in capitals, eg INFO which makes it easy to grep the log file for different kinds of information.

Exit Code¶

During the startup phase, rclone will exit immediately if an error is detected in the configuration. There will always be a log message immediately before exiting.

When rclone is running it will accumulate errors as it goes along, and only exit with a non-zero exit code if (after retries) there were still failed transfers. For every error counted there will be a high priority log message (visible with -q) showing the message and which file caused the problem. A high priority message is also shown when starting a retry so the user can see that any previous error messages may not be valid after the retry. If rclone has done a retry it will log a high priority message if the retry was successful.

List of exit codes¶

•

0 - success

•

1 - Syntax or usage error

•

2 - Error not otherwise categorised

•

3 - Directory not found

•

4 - File not found

•

5 - Temporary error (one that more retries might fix) (Retry errors)

•

6 - Less serious errors (like 461 errors from dropbox) (NoRetry errors)

•

7 - Fatal error (one that more retries won't fix, like account suspended) (Fatal errors)

•

8 - Transfer exceeded - limit set by --max-transfer reached

Environment Variables¶

Options¶

To find the name of the environment variable, first, take the long option name, strip the leading --, change - to _, make upper case and prepend RCLONE_.

For example, to always set --stats 5s, set the environment variable RCLONE_STATS=5s. If you set stats on the command line this will override the environment variable setting.

Or to always use the trash in drive --drive-use-trash, set RCLONE_DRIVE_USE_TRASH=true.

The same parser is used for the options and the environment variables so they take exactly the same form.

Config file¶

To find the name of the environment variable, you need to set, take RCLONE_CONFIG_ + name of remote + _ + name of config file option and make it all uppercase.

For example, to configure an S3 remote named mys3: without a config file (using unix ways of setting environment variables):

$ export RCLONE_CONFIG_MYS3_TYPE=s3
$ export RCLONE_CONFIG_MYS3_ACCESS_KEY_ID=XXX
$ export RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY=XXX
$ rclone lsd MYS3:
          -1 2016-09-21 12:54:21        -1 my-bucket
$ rclone listremotes | grep mys3
mys3:

    

Note that if you want to create a remote using environment variables you must create the ..._TYPE variable as above.

Other environment variables¶

•

RCLONE_CONFIG_PASS` set to contain your config file password (see Configuration Encryption (#configuration-encryption) section)

•

HTTP_PROXY, HTTPS_PROXY and NO_PROXY (or the lowercase versions thereof).

Configuring using rclone authorize¶

...
Remote config
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes
n) No
y/n> n
For this to work, you will need rclone available on a machine that has a web browser available.
Execute the following on your machine:
    rclone authorize "amazon cloud drive"
Then paste the result below:
result>

    

Then on your main desktop machine

rclone authorize "amazon cloud drive"
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize rclone for access
Waiting for code...
Got code
Paste the following into your remote machine --->
SECRET_TOKEN
<---End paste

    

Then back to the headless box, paste in the code

result> SECRET_TOKEN
--------------------
[acd12]
client_id = 
client_secret = 
token = SECRET_TOKEN
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d>

    

Configuring by copying the config file¶

So first configure rclone on your desktop machine

rclone config

    

to set up the config file.

Find the config file by running rclone -h and looking for the help for the --config option

$ rclone -h
[snip]
      --config="/home/user/.rclone.conf": Config file.
[snip]

    

Now transfer it to the remote box (scp, cut paste, ftp, sftp etc) and place it in the correct place (use rclone -h on the remote box to find out where).

Patterns¶

If the pattern starts with a / then it only matches at the top level of the directory tree, relative to the root of the remote (not necessarily the root of the local drive). If it doesn't start with / then it is matched starting at the end of the path, but it will only match a complete path element:

file.jpg  - matches "file.jpg"
          - matches "directory/file.jpg"
          - doesn't match "afile.jpg"
          - doesn't match "directory/afile.jpg"
/file.jpg - matches "file.jpg" in the root directory of the remote
          - doesn't match "afile.jpg"
          - doesn't match "directory/file.jpg"

    

Important Note that you must use / in patterns and not \ even if running on Windows.

A * matches anything but not a /.

*.jpg  - matches "file.jpg"
       - matches "directory/file.jpg"
       - doesn't match "file.jpg/something"

    

Use ** to match anything, including slashes (/).

dir/** - matches "dir/file.jpg"
       - matches "dir/dir1/dir2/file.jpg"
       - doesn't match "directory/file.jpg"
       - doesn't match "adir/file.jpg"

    

A ? matches any character except a slash /.

l?ss  - matches "less"
      - matches "lass"
      - doesn't match "floss"

    

A [ and ] together make a a character class, such as [a-z] or [aeiou] or [[:alpha:]]. See the go regexp docs (https://golang.org/pkg/regexp/syntax/) for more info on these.

h[ae]llo - matches "hello"
         - matches "hallo"
         - doesn't match "hullo"

    

A { and } define a choice between elements. It should contain a comma separated list of patterns, any of which might match. These patterns can contain wildcards.

{one,two}_potato - matches "one_potato"
                 - matches "two_potato"
                 - doesn't match "three_potato"
                 - doesn't match "_potato"

    

Special characters can be escaped with a \ before them.

\*.jpg       - matches "*.jpg"
\\.jpg       - matches "\.jpg"
\[one\].jpg  - matches "[one].jpg"

    

Patterns are case sensitive unless the --ignore-case flag is used.

Without --ignore-case (default)

potato - matches "potato"
       - doesn't match "POTATO"

    

With --ignore-case

potato - matches "potato"
       - matches "POTATO"

    

Note also that rclone filter globs can only be used in one of the filter command line flags, not in the specification of the remote, so rclone copy "remote:dir*.jpg" /path/to/dir won't work - what is required is rclone --include "*.jpg" copy remote:dir /path/to/dir

Directories¶

Eg if you add the include rule

/a/*.jpg

    

Rclone will synthesize the directory include rule

/a/

    

If you put any rules which end in / then it will only match directories.

Directory matches are only used to optimise directory access patterns - you must still match the files that you want to match. Directory matches won't optimise anything on bucket based remotes (eg s3, swift, google compute storage, b2) which don't have a concept of directory.

Differences between rsync and rclone patterns¶

Rclone always does a wildcard match so \ must always escape a \.

How the rules are used¶

Each file is matched in order, starting from the top, against the rule in the list until it finds a match. The file is then included or excluded according to the rule type.

If the matcher fails to find a match after testing against all the entries in the list then the path is included.

For example given the following rules, + being include, - being exclude,

- secret*.jpg
+ *.jpg
+ *.png
+ file2.avi
- *

    

This would include

•

file1.jpg

•

file3.png

•

file2.avi

This would exclude

•

secret17.jpg

•

non *.jpg and *.png

A similar process is done on directory entries before recursing into them. This only works on remotes which have a concept of directory (Eg local, google drive, onedrive, amazon drive) and not on bucket based remotes (eg s3, swift, google compute storage, b2).

Adding filtering rules¶

Repeating options¶

•

--include

•

--include-from

•

--exclude

•

--exclude-from

•

--filter

•

--filter-from

Important You should not use --include* together with --exclude*. It may produce different results than you expected. In that case try to use: --filter*.

Note that all the options of the same type are processed together in the order above, regardless of what order they were placed on the command line.

So all --include options are processed first in the order they appeared on the command line, then all --include-from options etc.

To mix up the order includes and excludes, the --filter flag can be used.

--exclude - Exclude files matching pattern¶

This flag can be repeated. See above for the order the flags are processed in.

Eg --exclude *.bak to exclude all bak files from the sync.

--exclude-from - Read exclude patterns from file¶

This flag can be repeated. See above for the order the flags are processed in.

Prepare a file like this exclude-file.txt

# a sample exclude rule file
*.bak
file2.jpg

    

Then use as --exclude-from exclude-file.txt. This will sync all files except those ending in bak and file2.jpg.

This is useful if you have a lot of rules.

--include - Include files matching pattern¶

This flag can be repeated. See above for the order the flags are processed in.

Eg --include *.{png,jpg} to include all png and jpg files in the backup and no others.

This adds an implicit --exclude * at the very end of the filter list. This means you can mix --include and --include-from with the other filters (eg --exclude) but you must include all the files you want in the include statement. If this doesn't provide enough flexibility then you must use --filter-from.

--include-from - Read include patterns from file¶

This flag can be repeated. See above for the order the flags are processed in.

Prepare a file like this include-file.txt

# a sample include rule file
*.jpg
*.png
file2.avi

    

Then use as --include-from include-file.txt. This will sync all jpg, png files and file2.avi.

This is useful if you have a lot of rules.

This adds an implicit --exclude * at the very end of the filter list. This means you can mix --include and --include-from with the other filters (eg --exclude) but you must include all the files you want in the include statement. If this doesn't provide enough flexibility then you must use --filter-from.

--filter - Add a file-filtering rule¶

This flag can be repeated. See above for the order the flags are processed in.

Eg --filter "- *.bak" to exclude all bak files from the sync.

--filter-from - Read filtering patterns from a file¶

This flag can be repeated. See above for the order the flags are processed in.

Prepare a file like this filter-file.txt

# a sample filter rule file
- secret*.jpg
+ *.jpg
+ *.png
+ file2.avi
- /dir/Trash/**
+ /dir/**
# exclude everything else
- *

    

Then use as --filter-from filter-file.txt. The rules are processed in the order that they are defined.

This example will include all jpg and png files, exclude any files matching secret*.jpg and include file2.avi. It will also include everything in the directory dir at the root of the sync, except dir/Trash which it will exclude. Everything else will be excluded from the sync.

--files-from - Read list of source-file names¶

Rclone will not scan any directories if you use --files-from it will just look at the files specified. Rclone will not error if any of the files are missing from the source.

This option can be repeated to read from more than one file. These are read in the order that they are placed on the command line.

Paths within the --files-from file will be interpreted as starting with the root specified in the command. Leading / characters are ignored.

For example, suppose you had files-from.txt with this content:

# comment
file1.jpg
subdir/file2.jpg

    

You could then use it like this:

rclone copy --files-from files-from.txt /home/me/pics remote:pics

    

This will transfer these files only (if they exist)

/home/me/pics/file1.jpg        → remote:pics/file1.jpg
/home/me/pics/subdir/file2.jpg → remote:pics/subdirfile1.jpg

    

To take a more complicated example, let's say you had a few files you want to back up regularly with these absolute paths:

/home/user1/important
/home/user1/dir/file
/home/user2/stuff

    

To copy these you'd find a common subdirectory - in this case /home and put the remaining files in files-from.txt with or without leading /, eg

user1/important
user1/dir/file
user2/stuff

    

You could then copy these to a remote like this

rclone copy --files-from files-from.txt /home remote:backup

    

The 3 files will arrive in remote:backup with the paths as in the files-from.txt like this:

/home/user1/important → remote:backup/user1/important
/home/user1/dir/file  → remote:backup/user1/dir/file
/home/user2/stuff     → remote:backup/stuff

    

You could of course choose / as the root too in which case your files-from.txt might look like this.

/home/user1/important
/home/user1/dir/file
/home/user2/stuff

    

And you would transfer it like this

rclone copy --files-from files-from.txt / remote:backup

    

In this case there will be an extra home directory on the remote:

/home/user1/important → remote:home/backup/user1/important
/home/user1/dir/file  → remote:home/backup/user1/dir/file
/home/user2/stuff     → remote:home/backup/stuff

    

--min-size - Don't transfer any file smaller than¶

This option controls the minimum size file which will be transferred. This defaults to kBytes but a suffix of k, M, or G can be used.

For example --min-size 50k means no files smaller than 50kByte will be transferred.

--max-size - Don't transfer any file larger than¶

This option controls the maximum size file which will be transferred. This defaults to kBytes but a suffix of k, M, or G can be used.

For example --max-size 1G means no files larger than 1GByte will be transferred.

--max-age - Don't transfer any file older than this¶

•

ms - Milliseconds

•

s - Seconds

•

m - Minutes

•

h - Hours

•

d - Days

•

w - Weeks

•

M - Months

•

y - Years

For example --max-age 2d means no files older than 2 days will be transferred.

--min-age - Don't transfer any file younger than¶

This option controls the minimum age of files to transfer. Give in seconds or with a suffix (see --max-age for list of suffixes)

For example --min-age 2d means no files younger than 2 days will be transferred.

--delete-excluded - Delete files on dest excluded from¶

Important this flag is dangerous - use with --dry-run and -v first.

When doing rclone sync this will delete any files which are excluded from the sync on the destination.

If for example you did a sync from A to B without the --min-size 50k flag

rclone sync A: B:

    

Then you repeated it like this with the --delete-excluded

rclone --min-size 50k --delete-excluded sync A: B:

    

This would delete all files on B which are less than 50 kBytes as these are now excluded from the sync.

Always test first with --dry-run and -v before using this flag.

--dump filters - dump the filters to the output¶

Useful for debugging.

--ignore-case - make searches case insensitive¶

Normally a --include "file.txt" will not match a file called FILE.txt. However if you use the --ignore-case flag then --include "file.txt" this will match a file called FILE.txt.

Quoting shell metacharacters¶

Eg linux, OSX

•

--include \*.jpg

•

--include '*.jpg'

•

--include='*.jpg'

In Windows the expansion is done by the command not the shell so this should work fine

•

--include *.jpg

Exclude directory based on a file¶

Imagine, you have the following directory structure:

dir1/file1
dir1/dir2/file2
dir1/dir2/dir3/file3
dir1/dir2/dir3/.ignore

    

You can exclude dir3 from sync by running the following command:

rclone sync --exclude-if-present .ignore dir1 remote:backup

    

Currently only one filename is supported, i.e. --exclude-if-present should not be used multiple times.

Supported parameters¶

--rc¶

--rc-addr=IP¶

--rc-cert=KEY¶

--rc-client-ca=PATH¶

--rc-htpasswd=PATH¶

--rc-key=PATH¶

--rc-max-header-bytes=VALUE¶

--rc-user=VALUE¶

--rc-pass=VALUE¶

--rc-realm=VALUE¶

--rc-server-read-timeout=DURATION¶

--rc-server-write-timeout=DURATION¶

--rc-serve¶

Default Off.

--rc-files /path/to/directory¶

If this is set then rclone will serve the files in that directory. It will also open the root in the web browser if specified. This is for implementing browser based GUIs for rclone functions.

If --rc-user or --rc-pass is set then the URL that is opened will have the authorization in the URL in the http://user:pass@localhost/ style.

Default Off.

--rc-no-auth¶

If this is set then no authorisation will be required on the server to use these methods. The alternative is to use --rc-user and --rc-pass and use these credentials in the request.

Default Off.

Accessing the remote control via the rclone rc command¶

You can use it like this

$ rclone rc rc/noop param1=one param2=two
{
    "param1": "one",
    "param2": "two"
}

    

Run rclone rc on its own to see the help for the installed remote control commands.

rclone rc also supports a --json flag which can be used to send more complicated input parameters.

$ rclone rc --json '{ "p1": [1,"2",null,4], "p2": { "a":1, "b":2 } }' rc/noop
{
    "p1": [
        1,
        "2",
        null,
        4
    ],
    "p2": {
        "a": 1,
        "b": 2
    }
}

    

Special parameters¶

Running asynchronous jobs with _async = true¶

It is recommended that potentially long running jobs, eg sync/sync, sync/copy, sync/move, operations/purge are run with the _async flag to avoid any potential problems with the HTTP request and response timing out.

Starting a job with the _async flag:

$ rclone rc --json '{ "p1": [1,"2",null,4], "p2": { "a":1, "b":2 }, "_async": true }' rc/noop
{
    "jobid": 2
}

    

Query the status to see if the job has finished. For more information on the meaning of these return parameters see the job/status call.

$ rclone rc --json '{ "jobid":2 }' job/status
{
    "duration": 0.000124163,
    "endTime": "2018-10-27T11:38:07.911245881+01:00",
    "error": "",
    "finished": true,
    "id": 2,
    "output": {
        "_async": true,
        "p1": [
            1,
            "2",
            null,
            4
        ],
        "p2": {
            "a": 1,
            "b": 2
        }
    },
    "startTime": "2018-10-27T11:38:07.911121728+01:00",
    "success": true
}

    

job/list can be used to show the running or recently completed jobs

$ rclone rc job/list
{
    "jobids": [
        2
    ]
}

    

Supported commands¶

cache/expire: Purge a remote from cache¶

Eg

rclone rc cache/expire remote=path/to/sub/folder/
rclone rc cache/expire remote=/ withData=true

    

cache/fetch: Fetch file chunks¶

The chunks= parameter specifies the file chunks to check. It takes a comma separated list of array slice indices. The slice indices are similar to Python slices: start[:end]

start is the 0 based chunk number from the beginning of the file to fetch inclusive. end is 0 based chunk number from the beginning of the file to fetch exclisive. Both values can be negative, in which case they count from the back of the file. The value "-5:" represents the last 5 chunks of a file.

Some valid examples are: ":5,-5:" -> the first and last five chunks "0,-2" -> the first and the second last chunk "0:10" -> the first ten chunks

Any parameter with a key that starts with "file" can be used to specify files to fetch, eg

rclone rc cache/fetch chunks=0 file=hello file2=home/goodbye

    

File names will automatically be encrypted when the a crypt remote is used on top of the cache.

cache/stats: Get cache stats¶

config/create: create the config for a remote.¶

•

name - name of remote

•

type - type of new remote

•

type - type of the new remote

See the config create command (https://rclone.org/commands/rclone_config_create/) command for more information on the above.

Authentication is required for this call.

config/delete: Delete a remote in the config file.¶

See the config delete command (https://rclone.org/commands/rclone_config_delete/) command for more information on the above.

Authentication is required for this call.

config/dump: Dumps the config file.¶

Where keys are remote names and values are the config parameters.

See the config dump command (https://rclone.org/commands/rclone_config_dump/) command for more information on the above.

Authentication is required for this call.

config/get: Get a remote in the config file.¶

See the config dump command (https://rclone.org/commands/rclone_config_dump/) command for more information on the above.

Authentication is required for this call.

config/listremotes: Lists the remotes in the config file.¶

See the listremotes command (https://rclone.org/commands/rclone_listremotes/) command for more information on the above.

Authentication is required for this call.

config/password: password the config for a remote.¶

•

name - name of remote

•

type - type of new remote

See the config password command (https://rclone.org/commands/rclone_config_password/) command for more information on the above.

Authentication is required for this call.

config/providers: Shows how providers are configured in the config¶

Returns a JSON object: - providers - array of objects

See the config providers command (https://rclone.org/commands/rclone_config_providers/) command for more information on the above.

Authentication is required for this call.

config/update: update the config for a remote.¶

•

name - name of remote

•

type - type of new remote

See the config update command (https://rclone.org/commands/rclone_config_update/) command for more information on the above.

Authentication is required for this call.

core/bwlimit: Set the bandwidth limit.¶

Eg

rclone rc core/bwlimit rate=1M
rclone rc core/bwlimit rate=off

    

The format of the parameter is exactly the same as passed to --bwlimit except only one bandwidth may be specified.

core/gc: Runs a garbage collection.¶

core/memstats: Returns the memory statistics¶

The most interesting values for most people are:

•

HeapAlloc: This is the amount of memory rclone is actually using

•

HeapSys: This is the amount of memory rclone has obtained from the OS

•

Sys: this is the total amount of memory requested from the OS

•

It is virtual memory so may include unused memory

core/obscure: Obscures a string passed in.¶

Returns - obscured - string

core/pid: Return PID of current process¶

core/stats: Returns stats about current transfers.¶

rclone rc core/stats

    

Returns the following values:

{
    "speed": average speed in bytes/sec since start of the process,
    "bytes": total transferred bytes since the start of the process,
    "errors": number of errors,
    "fatalError": whether there has been at least one FatalError,
    "retryError": whether there has been at least one non-NoRetryError,
    "checks": number of checked files,
    "transfers": number of transferred files,
    "deletes" : number of deleted files,
    "elapsedTime": time in seconds since the start of the process,
    "lastError": last occurred error,
    "transferring": an array of currently active file transfers:
        [
            {
                "bytes": total transferred bytes for this file,
                "eta": estimated time in seconds until file transfer completion
                "name": name of the file,
                "percentage": progress of the file transfer in percent,
                "speed": speed in bytes/sec,
                "speedAvg": speed in bytes/sec as an exponentially weighted moving average,
                "size": size of the file in bytes
            }
        ],
    "checking": an array of names of currently active file checks
        []
}

    

Values for "transferring", "checking" and "lastError" are only assigned if data is available. The value for "eta" is null if an eta cannot be determined.

core/version: Shows the current version of rclone and the go¶

This shows the current version of go and the go runtime - version - rclone version, eg "v1.44" - decomposed - version number as [major, minor, patch, subpatch] - note patch and subpatch will be 999 for a git compiled version - isGit - boolean - true if this was compiled from the git version - os - OS in use as according to Go - arch - cpu architecture in use according to Go - goVersion - version of Go runtime in use

job/list: Lists the IDs of the running jobs¶

Results - jobids - array of integer job ids

job/status: Reads the status of the job ID¶

Results - finished - boolean - duration - time in seconds that the job ran for - endTime - time the job finished (eg "2018-10-26T18:50:20.528746884+01:00") - error - error from the job or empty string for no error - finished - boolean whether the job has finished or not - id - as passed in above - startTime - time the job started (eg "2018-10-26T18:50:20.528336039+01:00") - success - boolean - true for success false otherwise - output - output of the job as would have been returned if called synchronously

operations/about: Return the space used on the remote¶

•

fs - a remote name string eg "drive:"

•

remote - a path within that remote eg "dir"

The result is as returned from rclone about --json

Authentication is required for this call.

operations/cleanup: Remove trashed files in the remote or path¶

•

fs - a remote name string eg "drive:"

See the cleanup command (https://rclone.org/commands/rclone_cleanup/) command for more information on the above.

Authentication is required for this call.

operations/copyfile: Copy a file from source remote to destination¶

This takes the following parameters

•

srcFs - a remote name string eg "drive:" for the source

•

srcRemote - a path within that remote eg "file.txt" for the source

•

dstFs - a remote name string eg "drive2:" for the destination

•

dstRemote - a path within that remote eg "file2.txt" for the destination

This returns - jobid - ID of async job to query with job/status

Authentication is required for this call.

operations/copyurl: Copy the URL to the object¶

•

fs - a remote name string eg "drive:"

•

remote - a path within that remote eg "dir"

•

url - string, URL to read from

See the copyurl command (https://rclone.org/commands/rclone_copyurl/) command for more information on the above.

Authentication is required for this call.

operations/delete: Remove files in the path¶

•

fs - a remote name string eg "drive:"

See the delete command (https://rclone.org/commands/rclone_delete/) command for more information on the above.

Authentication is required for this call.

operations/deletefile: Remove the single file pointed to¶

•

fs - a remote name string eg "drive:"

•

remote - a path within that remote eg "dir"

See the deletefile command (https://rclone.org/commands/rclone_deletefile/) command for more information on the above.

Authentication is required for this call.

operations/list: List the given remote and path in JSON format¶

•

fs - a remote name string eg "drive:"

•

remote - a path within that remote eg "dir"

•

opt - a dictionary of options to control the listing (optional)

The result is

•

list

See the lsjson command for more information on the above and examples.

Authentication is required for this call.

operations/mkdir: Make a destination directory or container¶

•

fs - a remote name string eg "drive:"

•

remote - a path within that remote eg "dir"

See the mkdir command (https://rclone.org/commands/rclone_mkdir/) command for more information on the above.

Authentication is required for this call.

operations/movefile: Move a file from source remote to destination¶

This takes the following parameters

•

srcFs - a remote name string eg "drive:" for the source

•

srcRemote - a path within that remote eg "file.txt" for the source

•

dstFs - a remote name string eg "drive2:" for the destination

•

dstRemote - a path within that remote eg "file2.txt" for the destination

This returns - jobid - ID of async job to query with job/status

Authentication is required for this call.

operations/purge: Remove a directory or container and all of its¶

This takes the following parameters

•

fs - a remote name string eg "drive:"

•

remote - a path within that remote eg "dir"

See the purge command (https://rclone.org/commands/rclone_purge/) command for more information on the above.

Authentication is required for this call.

operations/rmdir: Remove an empty directory or container¶

•

fs - a remote name string eg "drive:"

•

remote - a path within that remote eg "dir"

See the rmdir command (https://rclone.org/commands/rclone_rmdir/) command for more information on the above.

Authentication is required for this call.

operations/rmdirs: Remove all the empty directories in the path¶

•

fs - a remote name string eg "drive:"

•

remote - a path within that remote eg "dir"

•

leaveRoot - boolean, set to true not to delete the root

See the rmdirs command (https://rclone.org/commands/rclone_rmdirs/) command for more information on the above.

Authentication is required for this call.

operations/size: Count the number of bytes and files in remote¶

•

fs - a remote name string eg "drive:path/to/dir"

Returns

•

count - number of files

•

bytes - number of bytes in those files

See the size command (https://rclone.org/commands/rclone_size/) command for more information on the above.

Authentication is required for this call.

options/blocks: List all the option blocks¶

options/get: Get all the options¶

This shows the internal names of the option within rclone which should map to the external options very easily with a few exceptions.

options/set: Set an option¶

•

option block name containing an object with

•

key: value

Repeated as often as required.

Only supply the options you wish to change. If an option is unknown it will be silently ignored. Not all options will have an effect when changed like this.

rc/error: This returns an error¶

rc/list: List all the registered remote control commands¶

rc/noop: Echo the input to the output parameters¶

rc/noopauth: Echo the input to the output parameters requiring auth¶

Authentication is required for this call.

sync/copy: copy a directory from source remote to destination remote¶

•

srcFs - a remote name string eg "drive:src" for the source

•

dstFs - a remote name string eg "drive:dst" for the destination

This returns - jobid - ID of async job to query with job/status

See the copy command (https://rclone.org/commands/rclone_copy/) command for more information on the above.

Authentication is required for this call.

sync/move: move a directory from source remote to destination remote¶

•

srcFs - a remote name string eg "drive:src" for the source

•

dstFs - a remote name string eg "drive:dst" for the destination

•

deleteEmptySrcDirs - delete empty src directories if set

This returns - jobid - ID of async job to query with job/status

See the move command (https://rclone.org/commands/rclone_move/) command for more information on the above.

Authentication is required for this call.

sync/sync: sync a directory from source remote to destination remote¶

•

srcFs - a remote name string eg "drive:src" for the source

•

dstFs - a remote name string eg "drive:dst" for the destination

This returns - jobid - ID of async job to query with job/status

See the sync command (https://rclone.org/commands/rclone_sync/) command for more information on the above.

Authentication is required for this call.

vfs/forget: Forget files or directories in the directory cache.¶

If no paths are passed in then it will forget all the paths in the directory cache.

rclone rc vfs/forget

    

Otherwise pass files or dirs in as file=path or dir=path. Any parameter key starting with file will forget that file and any starting with dir will forget that dir, eg

rclone rc vfs/forget file=hello file2=goodbye dir=home/junk

    

vfs/poll-interval: Get the status or update the value of the¶

Without any parameter given this returns the current status of the poll-interval setting.

When the interval=duration parameter is set, the poll-interval value is updated and the polling function is notified. Setting interval=0 disables poll-interval.

rclone rc vfs/poll-interval interval=5m

    

The timeout=duration parameter can be used to specify a time to wait for the current poll function to apply the new value. If timeout is less or equal 0, which is the default, wait indefinitely.

The new poll-interval value will only be active when the timeout is not reached.

If poll-interval is updated or disabled temporarily, some changes might not get picked up by the polling function, depending on the used remote.

vfs/refresh: Refresh the directory cache.¶

If no paths are passed in then it will refresh the root directory.

rclone rc vfs/refresh

    

Otherwise pass directories in as dir=path. Any parameter key starting with dir will refresh that directory, eg

rclone rc vfs/refresh dir=home/junk dir2=data/misc

    

If the parameter recursive=true is given the whole directory tree will get refreshed. This refresh will use --fast-list if enabled.

Accessing the remote control via HTTP¶

Each endpoint takes an JSON object and returns a JSON object or an error. The JSON objects are essentially a map of string names to values.

All calls must made using POST.

The input objects can be supplied using URL parameters, POST parameters or by supplying "Content-Type: application/json" and a JSON blob in the body. There are examples of these below using curl.

The response will be a JSON blob in the body of the response. This is formatted to be reasonably human readable.

Error returns¶

{
    "error": "Expecting string value for key \"remote\" (was float64)",
    "input": {
        "fs": "/tmp",
        "remote": 3
    },
    "status": 400
    "path": "operations/rmdir",
}

    

The keys in the error response are - error - error string - input - the input parameters to the call - status - the HTTP status code - path - the path of the call

CORS¶

Using POST with URL parameters only¶

curl -X POST 'http://localhost:5572/rc/noop?potato=1&sausage=2'

    

Response

{
    "potato": "1",
    "sausage": "2"
}

    

Here is what an error response looks like:

curl -X POST 'http://localhost:5572/rc/error?potato=1&sausage=2'

    

{
    "error": "arbitrary error on input map[potato:1 sausage:2]",
    "input": {
        "potato": "1",
        "sausage": "2"
    }
}

    

Note that curl doesn't return errors to the shell unless you use the -f option

$ curl -f -X POST 'http://localhost:5572/rc/error?potato=1&sausage=2'
curl: (22) The requested URL returned error: 400 Bad Request
$ echo $?
22

    

Using POST with a form¶

curl --data "potato=1" --data "sausage=2" http://localhost:5572/rc/noop

    

Response

{
    "potato": "1",
    "sausage": "2"
}

    

Note that you can combine these with URL parameters too with the POST parameters taking precedence.

curl --data "potato=1" --data "sausage=2" "http://localhost:5572/rc/noop?rutabaga=3&sausage=4"

    

Response

{
    "potato": "1",
    "rutabaga": "3",
    "sausage": "4"
}

    

Using POST with a JSON blob¶

curl -H "Content-Type: application/json" -X POST -d '{"potato":2,"sausage":1}' http://localhost:5572/rc/noop

    

response

{
    "password": "xyz",
    "username": "xyz"
}

    

This can be combined with URL parameters too if required. The JSON blob takes precedence.

curl -H "Content-Type: application/json" -X POST -d '{"potato":2,"sausage":1}' 'http://localhost:5572/rc/noop?rutabaga=3&potato=4'

    

{
    "potato": 2,
    "rutabaga": "3",
    "sausage": 1
}

    

Debugging rclone with pprof¶

To use these, first install go (https://golang.org/doc/install).

Debugging memory use¶

go tool pprof -web http://localhost:5572/debug/pprof/heap

    

This should open a page in your browser showing what is using what memory.

You can also use the -text flag to produce a textual summary

$ go tool pprof -text http://localhost:5572/debug/pprof/heap
Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
      flat  flat%   sum%        cum   cum%
 1024.03kB 66.62% 66.62%  1024.03kB 66.62%  github.com/ncw/rclone/vendor/golang.org/x/net/http2/hpack.addDecoderNode
     513kB 33.38%   100%      513kB 33.38%  net/http.newBufioWriterSize
         0     0%   100%  1024.03kB 66.62%  github.com/ncw/rclone/cmd/all.init
         0     0%   100%  1024.03kB 66.62%  github.com/ncw/rclone/cmd/serve.init
         0     0%   100%  1024.03kB 66.62%  github.com/ncw/rclone/cmd/serve/restic.init
         0     0%   100%  1024.03kB 66.62%  github.com/ncw/rclone/vendor/golang.org/x/net/http2.init
         0     0%   100%  1024.03kB 66.62%  github.com/ncw/rclone/vendor/golang.org/x/net/http2/hpack.init
         0     0%   100%  1024.03kB 66.62%  github.com/ncw/rclone/vendor/golang.org/x/net/http2/hpack.init.0
         0     0%   100%  1024.03kB 66.62%  main.init
         0     0%   100%      513kB 33.38%  net/http.(*conn).readRequest
         0     0%   100%      513kB 33.38%  net/http.(*conn).serve
         0     0%   100%  1024.03kB 66.62%  runtime.main

    

Debugging go routine leaks¶

See all active go routines using

curl http://localhost:5572/debug/pprof/goroutine?debug=1

    

Or go to http://localhost:5572/debug/pprof/goroutine?debug=1 in your browser.

Other profiles to look at¶

Here is how to use some of them:

•

Memory: go tool pprof http://localhost:5572/debug/pprof/heap

•

Go routines: curl http://localhost:5572/debug/pprof/goroutine?debug=1

•

30-second CPU profile: go tool pprof http://localhost:5572/debug/pprof/profile

•

5-second execution trace: wget http://localhost:5572/debug/pprof/trace?seconds=5

See the net/http/pprof docs (https://golang.org/pkg/net/http/pprof/) for more info on how to use the profiling and for a general overview see the Go team's blog post on profiling go programs (https://blog.golang.org/profiling-go-programs).

The profiling hook is zero overhead unless it is used (https://stackoverflow.com/q/26545159/164234).

Features¶

Hash¶

To use the verify checksums when transferring between cloud storage systems they must support a common hash type.

† Note that Dropbox supports its own custom hash (https://www.dropbox.com/developers/reference/content-hash). This is an SHA256 sum of all the 4MB block SHA256s.

‡ SFTP supports checksums if the same login has shell access and md5sum or sha1sum as well as echo are in the remote's PATH.

†† WebDAV supports modtimes when used with Owncloud and Nextcloud only.

‡‡ Microsoft OneDrive Personal supports SHA1 hashes, whereas OneDrive for business and SharePoint server support Microsoft's own QuickXorHash (https://docs.microsoft.com/en-us/onedrive/developer/code-snippets/quickxorhash).

ModTime¶

All cloud storage systems support some kind of date on the object and these will be set when transferring from the cloud storage system.

Case Insensitive¶

This can cause problems when syncing between a case insensitive system and a case sensitive system. The symptom of this is that no matter how many times you run the sync it never completes fully.

The local filesystem and SFTP may or may not be case sensitive depending on OS.

•

Windows - usually case insensitive, though case is preserved

•

OSX - usually case insensitive, though it is possible to format case sensitive

•

Linux - usually case sensitive, but there are case insensitive file systems (eg FAT formatted USB keys)

Most of the time this doesn't cause any problems as people tend to avoid files whose name differs only by case even on case sensitive systems.

Duplicate files¶

This confuses rclone greatly when syncing - use the rclone dedupe command to rename or remove duplicates.

MIME Type¶

Some cloud storage systems support reading (R) the MIME type of objects and some support writing (W) the MIME type of objects.

The MIME type can be important if you are serving files directly to HTTP from the storage system.

If you are copying from a remote which supports reading (R) to a remote which supports writing (W) then rclone will preserve the MIME types. Otherwise they will be guessed from the extension, or the remote itself may assign the MIME type.

Optional Features¶

Purge¶

† Note Swift and Hubic implement this in order to delete directory markers but they don't actually have a quicker way of deleting files other than deleting them individually.

‡ StreamUpload is not supported with Nextcloud

Copy¶

If the server doesn't support Copy directly then for copy operations the file is downloaded then re-uploaded.

Move¶

If the server isn't capable of Move then rclone simulates it with Copy then delete. If the server doesn't support Copy then rclone will download the file and re-upload it.

DirMove¶

CleanUp¶

If the server can't do CleanUp then rclone cleanup will return an error.

ListR¶

StreamUpload¶

LinkSharing¶

About¶

If the server can't do About then rclone about will return an error.

Alias¶

Paths may be as deep as required or a local path, eg remote:directory/subdirectory or /directory/subdirectory.

During the initial setup with rclone config you will specify the target remote. The target remote can either be a local path or another remote.

Subfolders can be used in target remote. Asume a alias remote named backup with the target mydrive:private/backup. Invoking rclone mkdir backup:desktop is exactly the same as invoking rclone mkdir mydrive:private/backup/desktop.

There will be no special handling of paths containing .. segments. Invoking rclone mkdir backup:../desktop is exactly the same as invoking rclone mkdir mydrive:private/backup/../desktop. The empty path is not allowed as a remote. To alias the current directory use . instead.

Here is an example of how to make a alias called remote for local folder. First run:

 rclone config

    

This will guide you through an interactive setup process:

No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> remote
Type of storage to configure.
Choose a number from below, or type in your own value
 1 / Alias for a existing remote
   \ "alias"
 2 / Amazon Drive
   \ "amazon cloud drive"
 3 / Amazon S3 (also Dreamhost, Ceph, Minio)
   \ "s3"
 4 / Backblaze B2
   \ "b2"
 5 / Box
   \ "box"
 6 / Cache a remote
   \ "cache"
 7 / Dropbox
   \ "dropbox"
 8 / Encrypt/Decrypt a remote
   \ "crypt"
 9 / FTP Connection
   \ "ftp"
10 / Google Cloud Storage (this is not Google Drive)
   \ "google cloud storage"
11 / Google Drive
   \ "drive"
12 / Hubic
   \ "hubic"
13 / Local Disk
   \ "local"
14 / Microsoft Azure Blob Storage
   \ "azureblob"
15 / Microsoft OneDrive
   \ "onedrive"
16 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
   \ "swift"
17 / Pcloud
   \ "pcloud"
18 / QingCloud Object Storage
   \ "qingstor"
19 / SSH/SFTP Connection
   \ "sftp"
20 / Webdav
   \ "webdav"
21 / Yandex Disk
   \ "yandex"
22 / http Connection
   \ "http"
Storage> 1
Remote or path to alias.
Can be "myremote:path/to/dir", "myremote:bucket", "myremote:" or "/local/path".
remote> /mnt/storage/backup
Remote config
--------------------
[remote]
remote = /mnt/storage/backup
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name                 Type
====                 ====
remote               alias
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

    

Once configured you can then use rclone like this,

List directories in top level in /mnt/storage/backup

rclone lsd remote:

    

List all the files in /mnt/storage/backup

rclone ls remote:

    

Copy another local directory to the alias directory called source

rclone copy /home/source remote:source

    

Standard Options¶

--alias-remote¶

•

Config: remote

•

Env Var: RCLONE_ALIAS_REMOTE

•

Type: string

•

Default: ""

Amazon Drive¶

Status¶

For the history on why rclone no longer has a set of Amazon Drive API keys see the forum (https://forum.rclone.org/t/rclone-has-been-banned-from-amazon-drive/2314).

If you happen to know anyone who works at Amazon then please ask them to re-instate rclone into the Amazon Drive developer program - thanks!

Setup¶

The configuration process for Amazon Drive may involve using an oauth proxy (https://github.com/ncw/oauthproxy). This is used to keep the Amazon credentials out of the source code. The proxy runs in Google's very secure App Engine environment and doesn't store any credentials which pass through it.

Since rclone doesn't currently have its own Amazon Drive credentials so you will either need to have your own client_id and client_secret with Amazon Drive, or use a a third party ouath proxy in which case you will need to enter client_id, client_secret, auth_url and token_url.

Note also if you are not using Amazon's auth_url and token_url, (ie you filled in something for those) then if setting up on a remote machine you can only use the copying the config method of configuration (https://rclone.org/remote_setup/#configuring-by-copying-the-config-file) - rclone authorize will not work.

Here is an example of how to make a remote called remote. First run:

 rclone config

    

This will guide you through an interactive setup process:

No remotes found - make a new one
n) New remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
n/r/c/s/q> n
name> remote
Type of storage to configure.
Choose a number from below, or type in your own value
 1 / Amazon Drive
   \ "amazon cloud drive"
 2 / Amazon S3 (also Dreamhost, Ceph, Minio)
   \ "s3"
 3 / Backblaze B2
   \ "b2"
 4 / Dropbox
   \ "dropbox"
 5 / Encrypt/Decrypt a remote
   \ "crypt"
 6 / FTP Connection
   \ "ftp"
 7 / Google Cloud Storage (this is not Google Drive)
   \ "google cloud storage"
 8 / Google Drive
   \ "drive"
 9 / Hubic
   \ "hubic"
10 / Local Disk
   \ "local"
11 / Microsoft OneDrive
   \ "onedrive"
12 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
   \ "swift"
13 / SSH/SFTP Connection
   \ "sftp"
14 / Yandex Disk
   \ "yandex"
Storage> 1
Amazon Application Client Id - required.
client_id> your client ID goes here
Amazon Application Client Secret - required.
client_secret> your client secret goes here
Auth server URL - leave blank to use Amazon's.
auth_url> Optional auth URL
Token server url - leave blank to use Amazon's.
token_url> Optional token URL
Remote config
Make sure your Redirect URL is set to "http://127.0.0.1:53682/" in your custom config.
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes
n) No
y/n> y
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize rclone for access
Waiting for code...
Got code
--------------------
[remote]
client_id = your client ID goes here
client_secret = your client secret goes here
auth_url = Optional auth URL
token_url = Optional token URL
token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxx","token_type":"bearer","refresh_token":"xxxxxxxxxxxxxxxxxx","expiry":"2015-09-06T16:07:39.658438471+01:00"}
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

    

See the remote setup docs (https://rclone.org/remote_setup/) for how to set it up on a machine with no Internet browser available.

Note that rclone runs a webserver on your local machine to collect the token as returned from Amazon. This only runs from the moment it opens your browser to the moment you get back the verification code. This is on http://127.0.0.1:53682/ and this it may require you to unblock it temporarily if you are running a host firewall.

Once configured you can then use rclone like this,

List directories in top level of your Amazon Drive

rclone lsd remote:

    

List all the files in your Amazon Drive

rclone ls remote:

    

To copy a local directory to an Amazon Drive directory called backup

rclone copy /home/source remote:backup

    

Modified time and MD5SUMs¶

It does store MD5SUMs so for a more accurate sync, you can use the --checksum flag.

Deleting files¶

Using with non .com Amazon accounts¶

Standard Options¶

--acd-client-id¶

•

Config: client_id

•

Env Var: RCLONE_ACD_CLIENT_ID

•

Type: string

•

Default: ""

--acd-client-secret¶

•

Config: client_secret

•

Env Var: RCLONE_ACD_CLIENT_SECRET

•

Type: string

•

Default: ""

Advanced Options¶

--acd-auth-url¶

•

Config: auth_url

•

Env Var: RCLONE_ACD_AUTH_URL

•

Type: string

•

Default: ""

--acd-token-url¶

•

Config: token_url

•

Env Var: RCLONE_ACD_TOKEN_URL

•

Type: string

•

Default: ""

--acd-checkpoint¶

•

Config: checkpoint

•

Env Var: RCLONE_ACD_CHECKPOINT

•

Type: string

•

Default: ""

--acd-upload-wait-per-gb¶

Sometimes Amazon Drive gives an error when a file has been fully uploaded but the file appears anyway after a little while. This happens sometimes for files over 1GB in size and nearly every time for files bigger than 10GB. This parameter controls the time rclone waits for the file to appear.

The default value for this parameter is 3 minutes per GB, so by default it will wait 3 minutes for every GB uploaded to see if the file appears.

You can disable this feature by setting it to 0. This may cause conflict errors as rclone retries the failed upload but the file will most likely appear correctly eventually.

These values were determined empirically by observing lots of uploads of big files for a range of file sizes.

Upload with the "-v" flag to see more info about what rclone is doing in this situation.

•

Config: upload_wait_per_gb

•

Env Var: RCLONE_ACD_UPLOAD_WAIT_PER_GB

•

Type: Duration

•

Default: 3m0s

--acd-templink-threshold¶

Files this size or more will be downloaded via their "tempLink". This is to work around a problem with Amazon Drive which blocks downloads of files bigger than about 10GB. The default for this is 9GB which shouldn't need to be changed.

To download files above this threshold, rclone requests a "tempLink" which downloads the file through a temporary URL directly from the underlying S3 storage.

•

Config: templink_threshold

•

Env Var: RCLONE_ACD_TEMPLINK_THRESHOLD

•

Type: SizeSuffix

•

Default: 9G

Limitations¶

Amazon Drive has rate limiting so you may notice errors in the sync (429 errors). rclone will automatically retry the sync up to 3 times by default (see --retries flag) which should hopefully work around this problem.

Amazon Drive has an internal limit of file sizes that can be uploaded to the service. This limit is not officially published, but all files larger than this will fail.

At the time of writing (Jan 2016) is in the area of 50GB per file. This means that larger files are likely to fail.

Unfortunately there is no way for rclone to see that this failure is because of file size, so it will retry the operation, as any other failure. To avoid this problem, use --max-size 50000M option to limit the maximum size of uploaded files. Note that --max-size does not split files into segments, it only ignores files over this size.

Amazon S3 Storage Providers¶

•

AWS S3

•

Ceph

•

DigitalOcean Spaces

•

Dreamhost

•

IBM COS S3

•

Minio

•

Wasabi

Paths are specified as remote:bucket (or remote: for the lsd command.) You may put subdirectories in too, eg remote:bucket/path/to/dir.

Once you have made a remote (see the provider specific section above) you can use it like this:

See all buckets

rclone lsd remote:

    

Make a new bucket

rclone mkdir remote:bucket

    

List the contents of a bucket

rclone ls remote:bucket

    

Sync /home/local/directory to the remote bucket, deleting any excess files in the bucket.

rclone sync /home/local/directory remote:bucket

    

AWS S3¶

rclone config

    

This will guide you through an interactive setup process.

No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> remote
Type of storage to configure.
Choose a number from below, or type in your own value
 1 / Alias for a existing remote
   \ "alias"
 2 / Amazon Drive
   \ "amazon cloud drive"
 3 / Amazon S3 Compliant Storage Providers (AWS, Ceph, Dreamhost, IBM COS, Minio)
   \ "s3"
 4 / Backblaze B2
   \ "b2"
[snip]
23 / http Connection
   \ "http"
Storage> s3
Choose your S3 provider.
Choose a number from below, or type in your own value
 1 / Amazon Web Services (AWS) S3
   \ "AWS"
 2 / Ceph Object Storage
   \ "Ceph"
 3 / Digital Ocean Spaces
   \ "DigitalOcean"
 4 / Dreamhost DreamObjects
   \ "Dreamhost"
 5 / IBM COS S3
   \ "IBMCOS"
 6 / Minio Object Storage
   \ "Minio"
 7 / Wasabi Object Storage
   \ "Wasabi"
 8 / Any other S3 compatible provider
   \ "Other"
provider> 1
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own value
 1 / Enter AWS credentials in the next step
   \ "false"
 2 / Get AWS credentials from the environment (env vars or IAM)
   \ "true"
env_auth> 1
AWS Access Key ID - leave blank for anonymous access or runtime credentials.
access_key_id> XXX
AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
secret_access_key> YYY
Region to connect to.
Choose a number from below, or type in your own value
   / The default endpoint - a good choice if you are unsure.
 1 | US Region, Northern Virginia or Pacific Northwest.
   | Leave location constraint empty.
   \ "us-east-1"
   / US East (Ohio) Region
 2 | Needs location constraint us-east-2.
   \ "us-east-2"
   / US West (Oregon) Region
 3 | Needs location constraint us-west-2.
   \ "us-west-2"
   / US West (Northern California) Region
 4 | Needs location constraint us-west-1.
   \ "us-west-1"
   / Canada (Central) Region
 5 | Needs location constraint ca-central-1.
   \ "ca-central-1"
   / EU (Ireland) Region
 6 | Needs location constraint EU or eu-west-1.
   \ "eu-west-1"
   / EU (London) Region
 7 | Needs location constraint eu-west-2.
   \ "eu-west-2"
   / EU (Frankfurt) Region
 8 | Needs location constraint eu-central-1.
   \ "eu-central-1"
   / Asia Pacific (Singapore) Region
 9 | Needs location constraint ap-southeast-1.
   \ "ap-southeast-1"
   / Asia Pacific (Sydney) Region
10 | Needs location constraint ap-southeast-2.
   \ "ap-southeast-2"
   / Asia Pacific (Tokyo) Region
11 | Needs location constraint ap-northeast-1.
   \ "ap-northeast-1"
   / Asia Pacific (Seoul)
12 | Needs location constraint ap-northeast-2.
   \ "ap-northeast-2"
   / Asia Pacific (Mumbai)
13 | Needs location constraint ap-south-1.
   \ "ap-south-1"
   / South America (Sao Paulo) Region
14 | Needs location constraint sa-east-1.
   \ "sa-east-1"
region> 1
Endpoint for S3 API.
Leave blank if using AWS to use the default endpoint for the region.
endpoint> 
Location constraint - must be set to match the Region. Used when creating buckets only.
Choose a number from below, or type in your own value
 1 / Empty for US Region, Northern Virginia or Pacific Northwest.
   \ ""
 2 / US East (Ohio) Region.
   \ "us-east-2"
 3 / US West (Oregon) Region.
   \ "us-west-2"
 4 / US West (Northern California) Region.
   \ "us-west-1"
 5 / Canada (Central) Region.
   \ "ca-central-1"
 6 / EU (Ireland) Region.
   \ "eu-west-1"
 7 / EU (London) Region.
   \ "eu-west-2"
 8 / EU Region.
   \ "EU"
 9 / Asia Pacific (Singapore) Region.
   \ "ap-southeast-1"
10 / Asia Pacific (Sydney) Region.
   \ "ap-southeast-2"
11 / Asia Pacific (Tokyo) Region.
   \ "ap-northeast-1"
12 / Asia Pacific (Seoul)
   \ "ap-northeast-2"
13 / Asia Pacific (Mumbai)
   \ "ap-south-1"
14 / South America (Sao Paulo) Region.
   \ "sa-east-1"
location_constraint> 1
Canned ACL used when creating buckets and/or storing objects in S3.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Choose a number from below, or type in your own value
 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
   \ "private"
 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access.
   \ "public-read"
   / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access.
 3 | Granting this on a bucket is generally not recommended.
   \ "public-read-write"
 4 / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access.
   \ "authenticated-read"
   / Object owner gets FULL_CONTROL. Bucket owner gets READ access.
 5 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
   \ "bucket-owner-read"
   / Both the object owner and the bucket owner get FULL_CONTROL over the object.
 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
   \ "bucket-owner-full-control"
acl> 1
The server-side encryption algorithm used when storing this object in S3.
Choose a number from below, or type in your own value
 1 / None
   \ ""
 2 / AES256
   \ "AES256"
server_side_encryption> 1
The storage class to use when storing objects in S3.
Choose a number from below, or type in your own value
 1 / Default
   \ ""
 2 / Standard storage class
   \ "STANDARD"
 3 / Reduced redundancy storage class
   \ "REDUCED_REDUNDANCY"
 4 / Standard Infrequent Access storage class
   \ "STANDARD_IA"
 5 / One Zone Infrequent Access storage class
   \ "ONEZONE_IA"
storage_class> 1
Remote config
--------------------
[remote]
type = s3
provider = AWS
env_auth = false
access_key_id = XXX
secret_access_key = YYY
region = us-east-1
endpoint = 
location_constraint = 
acl = private
server_side_encryption = 
storage_class = 
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> 

    

--fast-list¶

--update and --use-server-modtime¶

For many operations, the time the object was last uploaded to the remote is sufficient to determine if it is "dirty". By using --update along with --use-server-modtime, you can avoid the extra API call and simply upload files whose local modtime is newer than the time it was last uploaded.

Modified time¶

Multipart uploads¶

Buckets and Regions¶

Authentication¶

The different authentication methods are tried in this order:

•

Directly in the rclone configuration file (env_auth = false in the config file):

•

access_key_id and secret_access_key are required.

•

session_token can be optionally set when using AWS STS.

•

Runtime configuration (env_auth = true in the config file):

•

Export the following environment variables before running rclone:

•

Or, use a named profile (https://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html):

•

Or, run rclone in an ECS task with an IAM role (AWS only).

•

Or, run rclone on an EC2 instance with an IAM role (AWS only).

If none of these option actually end up providing rclone with AWS credentials then S3 interaction will be non-authenticated (see below).

S3 Permissions¶

•

ListBucket

•

DeleteObject

•

GetObject

•

PutObject

•

PutObjectACL

Example policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::USER_SID:user/USER_NAME"
            },
            "Action": [
                "s3:ListBucket",
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
              "arn:aws:s3:::BUCKET_NAME/*",
              "arn:aws:s3:::BUCKET_NAME"
            ]
        }
    ]
}

    

Notes on above:

1.

This is a policy that can be used when creating bucket. It assumes that USER_NAME has been created.

2.

The Resource entry must include both resource ARNs, as one implies the bucket and the other implies the bucket's objects.

For reference, here's an Ansible script (https://gist.github.com/ebridges/ebfc9042dd7c756cd101cfa807b7ae2b) that will generate one or more buckets that will work with rclone sync.

Key Management System (KMS)¶

A proper fix is being worked on in issue #1824 (https://github.com/ncw/rclone/issues/1824).

Glacier¶

2017/09/11 19:07:43 Failed to sync: failed to open source object: Object in GLACIER, restore first: path/to/file

    

In this case you need to restore (http://docs.aws.amazon.com/AmazonS3/latest/user-guide/restore-archived-objects.html) the object(s) in question before using rclone.

Standard Options¶

--s3-provider¶

•

Config: provider

•

Env Var: RCLONE_S3_PROVIDER

•

Type: string

•

Default: ""

•

Examples:

--s3-env-auth¶

•

Config: env_auth

•

Env Var: RCLONE_S3_ENV_AUTH

•

Type: bool

•

Default: false

•

Examples:

--s3-access-key-id¶

•

Config: access_key_id

•

Env Var: RCLONE_S3_ACCESS_KEY_ID

•

Type: string

•

Default: ""

--s3-secret-access-key¶

•

Config: secret_access_key

•

Env Var: RCLONE_S3_SECRET_ACCESS_KEY

•

Type: string

•

Default: ""

--s3-region¶

•

Config: region

•

Env Var: RCLONE_S3_REGION

•

Type: string

•

Default: ""

•

Examples:

--s3-region¶

•

Config: region

•

Env Var: RCLONE_S3_REGION

•

Type: string

•

Default: ""

•

Examples:

--s3-endpoint¶

•

Config: endpoint

•

Env Var: RCLONE_S3_ENDPOINT

•

Type: string

•

Default: ""

--s3-endpoint¶

•

Config: endpoint

•

Env Var: RCLONE_S3_ENDPOINT

•

Type: string

•

Default: ""

•

Examples:

--s3-endpoint¶

•

Config: endpoint

•

Env Var: RCLONE_S3_ENDPOINT

•

Type: string

•

Default: ""

•

Examples:

--s3-location-constraint¶

•

Config: location_constraint

•

Env Var: RCLONE_S3_LOCATION_CONSTRAINT

•

Type: string

•

Default: ""

•

Examples:

--s3-location-constraint¶

•

Config: location_constraint

•

Env Var: RCLONE_S3_LOCATION_CONSTRAINT

•

Type: string

•

Default: ""

•

Examples:

--s3-location-constraint¶

•

Config: location_constraint

•

Env Var: RCLONE_S3_LOCATION_CONSTRAINT

•

Type: string

•

Default: ""

--s3-acl¶

For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl

Note that this ACL is applied when server side copying objects as S3 doesn't copy the ACL from the source but rather writes a fresh one.

•

Config: acl

•

Env Var: RCLONE_S3_ACL

•

Type: string

•

Default: ""

•

Examples:

--s3-server-side-encryption¶

•

Config: server_side_encryption

•

Env Var: RCLONE_S3_SERVER_SIDE_ENCRYPTION

•

Type: string

•

Default: ""

•

Examples:

--s3-sse-kms-key-id¶

•

Config: sse_kms_key_id

•

Env Var: RCLONE_S3_SSE_KMS_KEY_ID

•

Type: string

•

Default: ""

•

Examples:

--s3-storage-class¶

•

Config: storage_class

•

Env Var: RCLONE_S3_STORAGE_CLASS

•

Type: string

•

Default: ""

•

Examples:

Advanced Options¶

--s3-chunk-size¶

Any files larger than this will be uploaded in chunks of this size. The default is 5MB. The minimum is 5MB.

Note that "--s3-upload-concurrency" chunks of this size are buffered in memory per transfer.

If you are transferring large files over high speed links and you have enough memory, then increasing this will speed up the transfers.

•

Config: chunk_size

•

Env Var: RCLONE_S3_CHUNK_SIZE

•

Type: SizeSuffix

•

Default: 5M

--s3-disable-checksum¶

•

Config: disable_checksum

•

Env Var: RCLONE_S3_DISABLE_CHECKSUM

•

Type: bool

•

Default: false

--s3-session-token¶

•

Config: session_token

•

Env Var: RCLONE_S3_SESSION_TOKEN

•

Type: string

•

Default: ""

--s3-upload-concurrency¶

This is the number of chunks of the same file that are uploaded concurrently.

If you are uploading small numbers of large file over high speed link and these uploads do not fully utilize your bandwidth, then increasing this may help to speed up the transfers.

•

Config: upload_concurrency

•

Env Var: RCLONE_S3_UPLOAD_CONCURRENCY

•

Type: int

•

Default: 2

--s3-force-path-style¶

If this is true (the default) then rclone will use path style access, if false then rclone will use virtual path style. See the AWS S3 docs (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro) for more info.

Some providers (eg Aliyun OSS or Netease COS) require this set to false.

•

Config: force_path_style

•

Env Var: RCLONE_S3_FORCE_PATH_STYLE

•

Type: bool

•

Default: true

--s3-v2-auth¶

If this is false (the default) then rclone will use v4 authentication. If it is set then rclone will use v2 authentication.

Use this only if v4 signatures don't work, eg pre Jewel/v10 CEPH.

•

Config: v2_auth

•

Env Var: RCLONE_S3_V2_AUTH

•

Type: bool

•

Default: false

Anonymous access to public buckets¶

[anons3]
type = s3
provider = AWS
env_auth = false
access_key_id = 
secret_access_key = 
region = us-east-1
endpoint = 
location_constraint = 
acl = private
server_side_encryption = 
storage_class = 

    

Then use it as normal with the name of the public bucket, eg

rclone lsd anons3:1000genomes

    

You will be able to list and copy data but not upload it.

Ceph¶

To use rclone with Ceph, configure as above but leave the region blank and set the endpoint. You should end up with something like this in your config:

[ceph]
type = s3
provider = Ceph
env_auth = false
access_key_id = XXX
secret_access_key = YYY
region =
endpoint = https://ceph.endpoint.example.com
location_constraint =
acl =
server_side_encryption =
storage_class =

    

Note also that Ceph sometimes puts / in the passwords it gives users. If you read the secret access key using the command line tools you will get a JSON blob with the / escaped as \/. Make sure you only write / in the secret access key.

Eg the dump from Ceph looks something like this (irrelevant keys removed).

{
    "user_id": "xxx",
    "display_name": "xxxx",
    "keys": [
        {
            "user": "xxx",
            "access_key": "xxxxxx",
            "secret_key": "xxxxxx\/xxxx"
        }
    ],
}

    

Because this is a json dump, it is encoding the / as \/, so if you use the secret key as xxxxxx/xxxx it will work fine.

Dreamhost¶

To use rclone with Dreamhost, configure as above but leave the region blank and set the endpoint. You should end up with something like this in your config:

[dreamobjects]
type = s3
provider = DreamHost
env_auth = false
access_key_id = your_access_key
secret_access_key = your_secret_key
region =
endpoint = objects-us-west-1.dream.io
location_constraint =
acl = private
server_side_encryption =
storage_class =

    

DigitalOcean Spaces¶

To connect to DigitalOcean Spaces you will need an access key and secret key. These can be retrieved on the "Applications & API (https://cloud.digitalocean.com/settings/api/tokens)" page of the DigitalOcean control panel. They will be needed when promted by rclone config for your access_key_id and secret_access_key.

When prompted for a region or location_constraint, press enter to use the default value. The region must be included in the endpoint setting (e.g. nyc3.digitaloceanspaces.com). The defualt values can be used for other settings.

Going through the whole process of creating a new remote by running rclone config, each prompt should be answered as shown below:

Storage> s3
env_auth> 1
access_key_id> YOUR_ACCESS_KEY
secret_access_key> YOUR_SECRET_KEY
region>
endpoint> nyc3.digitaloceanspaces.com
location_constraint>
acl>
storage_class>

    

The resulting configuration file should look like:

[spaces]
type = s3
provider = DigitalOcean
env_auth = false
access_key_id = YOUR_ACCESS_KEY
secret_access_key = YOUR_SECRET_KEY
region =
endpoint = nyc3.digitaloceanspaces.com
location_constraint =
acl =
server_side_encryption =
storage_class =

    

Once configured, you can create a new Space and begin copying files. For example:

rclone mkdir spaces:my-new-space
rclone copy /path/to/files spaces:my-new-space

    

IBM COS (S3)¶

To configure access to IBM COS S3, follow the steps below:

1.

Run rclone config and select n for a new remote.

2.

Enter the name for the configuration

3.

Select "s3" storage.

4.

Select IBM COS as the S3 Storage Provider.

5.

Enter the Access Key and Secret.

6.

Specify the endpoint for IBM COS. For Public IBM COS, choose from the option below. For On Premise IBM COS, enter an enpoint address.

7.

Specify a IBM COS Location Constraint. The location constraint must match endpoint when using IBM Cloud Public. For on-prem COS, do not make a selection from this list, hit enter

8.

Specify a canned ACL. IBM Cloud (Strorage) supports "public-read" and "private". IBM Cloud(Infra) supports all the canned ACLs. On-Premise COS supports all the canned ACLs.

9.

Review the displayed configuration and accept to save the "remote" then quit. The config file should look like this

10.

Execute rclone commands

Minio¶

It is very easy to install and provides an S3 compatible server which can be used by rclone.

To use it, install Minio following the instructions here (https://docs.minio.io/docs/minio-quickstart-guide).

When it configures itself Minio will print something like this

Endpoint:  http://192.168.1.106:9000  http://172.23.0.1:9000
AccessKey: USWUXHGYZQYFYFFIT3RE
SecretKey: MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
Region:    us-east-1
SQS ARNs:  arn:minio:sqs:us-east-1:1:redis arn:minio:sqs:us-east-1:2:redis
Browser Access:
   http://192.168.1.106:9000  http://172.23.0.1:9000
Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio http://192.168.1.106:9000 USWUXHGYZQYFYFFIT3RE MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
Object API (Amazon S3 compatible):
   Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
   Java:       https://docs.minio.io/docs/java-client-quickstart-guide
   Python:     https://docs.minio.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.minio.io/docs/dotnet-client-quickstart-guide
Drive Capacity: 26 GiB Free, 165 GiB Total

    

These details need to go into rclone config like this. Note that it is important to put the region in as stated above.

env_auth> 1
access_key_id> USWUXHGYZQYFYFFIT3RE
secret_access_key> MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
region> us-east-1
endpoint> http://192.168.1.106:9000
location_constraint>
server_side_encryption>

    

Which makes the config file look like this

[minio]
type = s3
provider = Minio
env_auth = false
access_key_id = USWUXHGYZQYFYFFIT3RE
secret_access_key = MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
region = us-east-1
endpoint = http://192.168.1.106:9000
location_constraint =
server_side_encryption =

    

So once set up, for example to copy files into a bucket

rclone copy /path/to/files minio:bucket

    

Wasabi¶

Wasabi provides an S3 interface which can be configured for use with rclone like this.

No remotes found - make a new one
n) New remote
s) Set configuration password
n/s> n
name> wasabi
Type of storage to configure.
Choose a number from below, or type in your own value
 1 / Amazon Drive
   \ "amazon cloud drive"
 2 / Amazon S3 (also Dreamhost, Ceph, Minio)
   \ "s3"
[snip]
Storage> s3
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own value
 1 / Enter AWS credentials in the next step
   \ "false"
 2 / Get AWS credentials from the environment (env vars or IAM)
   \ "true"
env_auth> 1
AWS Access Key ID - leave blank for anonymous access or runtime credentials.
access_key_id> YOURACCESSKEY
AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
secret_access_key> YOURSECRETACCESSKEY
Region to connect to.
Choose a number from below, or type in your own value
   / The default endpoint - a good choice if you are unsure.
 1 | US Region, Northern Virginia or Pacific Northwest.
   | Leave location constraint empty.
   \ "us-east-1"
[snip]
region> us-east-1
Endpoint for S3 API.
Leave blank if using AWS to use the default endpoint for the region.
Specify if using an S3 clone such as Ceph.
endpoint> s3.wasabisys.com
Location constraint - must be set to match the Region. Used when creating buckets only.
Choose a number from below, or type in your own value
 1 / Empty for US Region, Northern Virginia or Pacific Northwest.
   \ ""
[snip]
location_constraint>
Canned ACL used when creating buckets and/or storing objects in S3.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Choose a number from below, or type in your own value
 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
   \ "private"
[snip]
acl>
The server-side encryption algorithm used when storing this object in S3.
Choose a number from below, or type in your own value
 1 / None
   \ ""
 2 / AES256
   \ "AES256"
server_side_encryption>
The storage class to use when storing objects in S3.
Choose a number from below, or type in your own value
 1 / Default
   \ ""
 2 / Standard storage class
   \ "STANDARD"
 3 / Reduced redundancy storage class
   \ "REDUCED_REDUNDANCY"
 4 / Standard Infrequent Access storage class
   \ "STANDARD_IA"
storage_class>
Remote config
--------------------
[wasabi]
env_auth = false
access_key_id = YOURACCESSKEY
secret_access_key = YOURSECRETACCESSKEY
region = us-east-1
endpoint = s3.wasabisys.com
location_constraint =
acl =
server_side_encryption =
storage_class =
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

    

This will leave the config file looking like this.

[wasabi]
type = s3
provider = Wasabi
env_auth = false
access_key_id = YOURACCESSKEY
secret_access_key = YOURSECRETACCESSKEY
region =
endpoint = s3.wasabisys.com
location_constraint =
acl =
server_side_encryption =
storage_class =

    

Aliyun OSS / Netease NOS¶

Note this is a pretty standard S3 setup, except for the setting of force_path_style = false in the advanced config.

# rclone config
e/n/d/r/c/s/q> n
name> oss
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 3 / Amazon S3 Compliant Storage Providers (AWS, Ceph, Dreamhost, IBM COS, Minio)
   \ "s3"
Storage> s3
Choose your S3 provider.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 8 / Any other S3 compatible provider
   \ "Other"
provider> other
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Enter a boolean value (true or false). Press Enter for the default ("false").
Choose a number from below, or type in your own value
 1 / Enter AWS credentials in the next step
   \ "false"
 2 / Get AWS credentials from the environment (env vars or IAM)
   \ "true"
env_auth> 1
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
access_key_id> xxxxxxxxxxxx
AWS Secret Access Key (password)
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
secret_access_key> xxxxxxxxxxxxxxxxx
Region to connect to.
Leave blank if you are using an S3 clone and you don't have a region.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Use this if unsure. Will use v4 signatures and an empty region.
   \ ""
 2 / Use this only if v4 signatures don't work, eg pre Jewel/v10 CEPH.
   \ "other-v2-signature"
region> 1
Endpoint for S3 API.
Required when using an S3 clone.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
endpoint> oss-cn-shenzhen.aliyuncs.com
Location constraint - must be set to match the Region.
Leave blank if not sure. Used when creating buckets only.
Enter a string value. Press Enter for the default ("").
location_constraint>
Canned ACL used when creating buckets and/or storing objects in S3.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
   \ "private"
acl> 1
Edit advanced config? (y/n)
y) Yes
n) No
y/n> y
Chunk size to use for uploading
Enter a size with suffix k,M,G,T. Press Enter for the default ("5M").
chunk_size>
Don't store MD5 checksum with object metadata
Enter a boolean value (true or false). Press Enter for the default ("false").
disable_checksum>
An AWS session token
Enter a string value. Press Enter for the default ("").
session_token>
Concurrency for multipart uploads.
Enter a signed integer. Press Enter for the default ("2").
upload_concurrency>
If true use path style access if false use virtual hosted style.
Some providers (eg Aliyun OSS or Netease COS) require this.
Enter a boolean value (true or false). Press Enter for the default ("true").
force_path_style> false
Remote config
--------------------
[oss]
type = s3
provider = Other
env_auth = false
access_key_id = xxxxxxxxx
secret_access_key = xxxxxxxxxxxxx
endpoint = oss-cn-shenzhen.aliyuncs.com
acl = private
force_path_style = false
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

    

Backblaze B2¶

Paths are specified as remote:bucket (or remote: for the lsd command.) You may put subdirectories in too, eg remote:bucket/path/to/dir.

Here is an example of making a b2 configuration. First run

rclone config

    

This will guide you through an interactive setup process. You will need your account number (a short hex number) and key (a long hex number) which you can get from the b2 control panel.

No remotes found - make a new one
n) New remote
q) Quit config
n/q> n
name> remote
Type of storage to configure.
Choose a number from below, or type in your own value
 1 / Amazon Drive
   \ "amazon cloud drive"
 2 / Amazon S3 (also Dreamhost, Ceph, Minio)
   \ "s3"
 3 / Backblaze B2
   \ "b2"
 4 / Dropbox
   \ "dropbox"
 5 / Encrypt/Decrypt a remote
   \ "crypt"
 6 / Google Cloud Storage (this is not Google Drive)
   \ "google cloud storage"
 7 / Google Drive
   \ "drive"
 8 / Hubic
   \ "hubic"
 9 / Local Disk
   \ "local"
10 / Microsoft OneDrive
   \ "onedrive"
11 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
   \ "swift"
12 / SSH/SFTP Connection
   \ "sftp"
13 / Yandex Disk
   \ "yandex"
Storage> 3
Account ID or Application Key ID
account> 123456789abc
Application Key
key> 0123456789abcdef0123456789abcdef0123456789
Endpoint for the service - leave blank normally.
endpoint>
Remote config
--------------------
[remote]
account = 123456789abc
key = 0123456789abcdef0123456789abcdef0123456789
endpoint =
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

    

This remote is called remote and can now be used like this

See all buckets

rclone lsd remote:

    

Create a new bucket

rclone mkdir remote:bucket

    

List the contents of a bucket

rclone ls remote:bucket

    

Sync /home/local/directory to the remote bucket, deleting any excess files in the bucket.

rclone sync /home/local/directory remote:bucket

    

Application Keys¶

You can use these with rclone too.

Follow Backblaze's docs to create an Application Key with the required permission and add the Application Key ID as the account and the Application Key itself as the key.

Note that you must put the Application Key ID as the account - you can't use the master Account ID. If you try then B2 will return 401 errors.

--fast-list¶

Modified time¶

Modified times are used in syncing and are fully supported except in the case of updating a modification time on an existing object. In this case the object will be uploaded again as B2 doesn't have an API method to set the modification time independent of doing an upload.

SHA1 checksums¶

Large files (bigger than the limit in --b2-upload-cutoff) which are uploaded in chunks will store their SHA1 on the object as X-Bz-Info-large_file_sha1 as recommended by Backblaze.

For a large file to be uploaded with an SHA1 checksum, the source needs to support SHA1 checksums. The local disk supports SHA1 checksums so large file transfers from local disk will have an SHA1. See the overview (/overview/#features) for exactly which remotes support SHA1.

Sources which don't support SHA1, in particular crypt will upload large files without SHA1 checksums. This may be fixed in the future (see #1767 (https://github.com/ncw/rclone/issues/1767)).

Files sizes below --b2-upload-cutoff will always have an SHA1 regardless of the source.

Transfers¶

Note that uploading big files (bigger than 200 MB by default) will use a 96 MB RAM buffer by default. There can be at most --transfers of these in use at any moment, so this sets the upper limit on the memory used.

Versions¶

Old versions of files, where available, are visible using the --b2-versions flag.

If you wish to remove all the old versions then you can use the rclone cleanup remote:bucket command which will delete all the old versions of files, leaving the current ones intact. You can also supply a path and only old versions under that path will be deleted, eg rclone cleanup remote:bucket/path/to/stuff.

Note that cleanup does not remove partially uploaded files from the bucket.

When you purge a bucket, the current and the old versions will be deleted then the bucket will be deleted.

However delete will cause the current versions of the files to become hidden old versions.

Here is a session showing the listing and retrieval of an old version followed by a cleanup of the old versions.

Show current version and all the versions with --b2-versions flag.

$ rclone -q ls b2:cleanup-test
        9 one.txt
$ rclone -q --b2-versions ls b2:cleanup-test
        9 one.txt
        8 one-v2016-07-04-141032-000.txt
       16 one-v2016-07-04-141003-000.txt
       15 one-v2016-07-02-155621-000.txt

    

Retrieve an old version

$ rclone -q --b2-versions copy b2:cleanup-test/one-v2016-07-04-141003-000.txt /tmp
$ ls -l /tmp/one-v2016-07-04-141003-000.txt
-rw-rw-r-- 1 ncw ncw 16 Jul  2 17:46 /tmp/one-v2016-07-04-141003-000.txt

    

Clean up all the old versions and show that they've gone.

$ rclone -q cleanup b2:cleanup-test
$ rclone -q ls b2:cleanup-test
        9 one.txt
$ rclone -q --b2-versions ls b2:cleanup-test
        9 one.txt

    

Data usage¶

All copy commands send the following 4 requests:

/b2api/v1/b2_authorize_account
/b2api/v1/b2_create_bucket
/b2api/v1/b2_list_buckets
/b2api/v1/b2_list_file_names

    

The b2_list_file_names request will be sent once for every 1k files in the remote path, providing the checksum and modification time of the listed files. As of version 1.33 issue #818 (https://github.com/ncw/rclone/issues/818) causes extra requests to be sent when using B2 with Crypt. When a copy operation does not require any files to be uploaded, no more requests will be sent.

Uploading files that do not require chunking, will send 2 requests per file upload:

/b2api/v1/b2_get_upload_url
/b2api/v1/b2_upload_file/

    

Uploading files requiring chunking, will send 2 requests (one each to start and finish the upload) and another 2 requests for each chunk:

/b2api/v1/b2_start_large_file
/b2api/v1/b2_get_upload_part_url
/b2api/v1/b2_upload_part/
/b2api/v1/b2_finish_large_file

    

Versions¶

Listing without --b2-versions

$ rclone -q ls b2:cleanup-test
        9 one.txt

    

And with

$ rclone -q --b2-versions ls b2:cleanup-test
        9 one.txt
        8 one-v2016-07-04-141032-000.txt
       16 one-v2016-07-04-141003-000.txt
       15 one-v2016-07-02-155621-000.txt

    

Showing that the current version is unchanged but older versions can be seen. These have the UTC date that they were uploaded to the server to the nearest millisecond appended to them.

Note that when using --b2-versions no file write operations are permitted, so you can't upload files or delete them.

Standard Options¶

--b2-account¶

•

Config: account

•

Env Var: RCLONE_B2_ACCOUNT

•

Type: string

•

Default: ""

--b2-key¶

•

Config: key

•

Env Var: RCLONE_B2_KEY

•

Type: string

•

Default: ""

--b2-hard-delete¶

•

Config: hard_delete

•

Env Var: RCLONE_B2_HARD_DELETE

•

Type: bool

•

Default: false

Advanced Options¶

--b2-endpoint¶

•

Config: endpoint

•

Env Var: RCLONE_B2_ENDPOINT

•

Type: string

•

Default: ""

--b2-test-mode¶

This is for debugging purposes only. Setting it to one of the strings below will cause b2 to return specific errors:

•

"fail_some_uploads"

•

"expire_some_account_authorization_tokens"

•

"force_cap_exceeded"

These will be set in the "X-Bz-Test-Mode" header which is documented in the b2 integrations checklist (https://www.backblaze.com/b2/docs/integration_checklist.html).

•

Config: test_mode

•

Env Var: RCLONE_B2_TEST_MODE

•

Type: string

•

Default: ""