rclone

Overview

rclone is a command-line tool for managing cloud storage files, offering an extensive array of functionalities as an alternative to traditional web storage interfaces provided by cloud providers. It is compatible with over 70 cloud storage solutions, including Baidu AI Cloud Object Storage (BOS), enterprise and consumer file storage services, and standard transfer protocols. Widely adopted on Linux, Windows, and Mac, rclone also enables third-party development of backup, recovery, and business process solutions using its command-line capabilities or API.

Install

Go to Here to download and install based on the OS used.

Usage

Configuration

Enter the command rclone config in the command line, followed by the following steps:

1No remotes found, make a new one?
2n) New remote
3s) Set configuration password
4q) Quit config
5n/s/q> n
6Enter name for new remote.
7 name> myBOS  #Specify a name for the new remote connection here
8 Storage> 4  #Please confirm to select the option starting with Amazon S3 Compliant Storage Providers. The specific number may vary
9 provider> 34  #Please confirm to select Any other S3 compatible provider(Other). The specific number may vary
10env_auth> 1
11 access_key_id> abcde #Account Access Key, which can be queried at https://console.bce.baidu.com/iam/#/iam/accesslist
12 secret_access_key> ABCDE #Account Secret Key, which can be queried at https://console.bce.baidu.com/iam/#/iam/accesslist
13 region> 1  #Be sure to select the option that uses the V4 signature
14 endpoint> http://s3.bj.bcebos.com  #BOS service domain compatible with the S3 protocol. Different endpoints correspond to different S3 domains. For details, see https://cloud.baidu.com/doc/BOS/s/xjwvyq9l4
15 location_constraint>  ##If unsure, press Enter directly without input
16 acl> 1  #Select the permission mode you want. The default is private
17Edit advanced config?
18y) Yes
19n) No (default)
20y/n> n
21Current remotes:
22Name                 Type
23====                 ====
24myBOS                s3
25e) Edit existing remote
26n) New remote
27d) Delete remote
28r) Rename remote
29c) Copy remote
30s) Set configuration password
31q) Quit config
32 e/n/d/r/c/s/q> q  #If the input is correct, select q to exit the configuration

The configuration settings can be adjusted as required, and you can exit based on the prompts after completing the configuration.

Specify access mode

View configuration file location

rclone config file

Specify the access mode as Virtual-hosted-style

1$ vi ~/.config/rclone/rclone.conf

Add the following configuration content to the configuration file

1$ force_path_style = false

General flag

• --help/-h Help
• -v Display more detailed step information during execution
• -vv Display debug log alignment error
• --interactive/-i Ask before execution

bucket

• View all buckets: rclone lsd myBOS:
• Create a bucket named new-bu: rclone mkdir myBOS:new-bu
• Delete a bucket named new-bu: rclone rmdir myBOS:new-bu
• Forcibly delete a bucket named new-bu (used when there are files in the bucket): rclone purge myBOS:new-bu

• Please note that the custom bucket name for the above creation and deletion operations must meet the bucket name definition rules of BOS, namely:

  1. Only include lowercase letters, numbers, and “-”.
  2. Start and end with lowercase letters or numbers
  3. The length is between 4-63 characters.
  4. Names cannot be duplicated across different buckets

• Bucket acl permission settings:

  • Select yes in the advanced config at the end of rclone config.
  • All newly created buckets will be automatically generated according to the bucket ACL permissions, with no impact on other buckets

object

• List all objects under the bucket named new-bu: rclone ls myBOS:new-bu

• File upload:

  • rclone copyto src dest ## src is the local address of the file to be uploaded, and dest is the destination BOS location
  • An example is as follows: rclone copyto /Users/Desktop/baidu.pptx myBOS:new-bu/baidu.pptx
  • To upload directly from a URL address to the cloud, you can use rclone copyurl https://example.com myBOS:new-bu

• File download:

  • rclone copyto src dest ## src is the location of the destination BOS to be downloaded, and dest is the destination save address
  • An example is as follows: rclone copyto myBOS:new-bu/baidu.pptx /Users/Desktop/baidu.pptx
• File deletion: rclone delete myBOS:new-bu/baidu.pptx

Folder synchronization to bucket

• The folder synchronization command is rclone sync src dest, where src represents the synchronization source and dest represents the target cloud storage location

  • For example, synchronize the local folder Desktop to new-bu: rclone sync ~/Desktop myBOS:new-bu

• Use flags to implement special rule synchronization measures:

  • --size-only Target files are modified only when the source and target file sizes are different, with no additional transactions
  • --checksum Modifications are performed only when the source and target file sizes or MD5 checksums are different, with no additional transactions
  • --update --use-server-modtime No additional transaction is applied, and the modification time becomes the object upload time, which applies to many operations, but may miss files created with timestamps in the past.

Clean up unfinished multipart upload files

• Clean up unfinished multipart upload files before 24 hours: rclone cleanup myBOS:test-bu
• Clean up the unfinished multipart upload files before 1 hour, 1 minute and 1 second, with the time adjustable: rclone backend cleanup myBOS:test-bu -o max-age=1h1m1s

Multithreaded flag

• --multi-thread-write-buffer-size=SIZE

  • Set buffer size to SIZE, 128k by default
  • Adjustable due to disk read-write limitations, recommended to be a multiple of 16k

• --multi-thread-chunk-size=SizeSuffix

  • The part size for multithreaded transmissions is usually determined by the backend.
  • For backends like local or SMB that implement OpenWriterAt but not OpenChunkWriter, there is no predefined part size. In such cases, the value specified by this option (default: 64 Mi) is used.

• --multi-thread-cutoff=SIZE

  • Rclone uses multithreading when transmitting files larger than the specified SIZE, with a default of 5GB
  • This is applicable to supported backends as targets but works with any backend as a source.

• --multi-thread-streams=N

  • This command sets the number of streams used during multithreaded transmission
  • Set the value to 0 to disable multithreaded transmission. The default value for rclone is 4, but a setting of 10 is recommended for BOS.
• The above flags are for specific command adjustments. For permanent settings, select yes in rclone config when reaching advanced config

More usage

For details, see https://rclone.org/commands/

FAQs

1. Selecting a non-V4 signature during the region configuration step may cause an error where files with Chinese filenames cannot be located in the cloud.
2. Once non-V4 signature is selected in the region step, modifications via rclone config cannot be saved to the config file. You need to query the specific config file location by using rclone config file to directly modify the config file.
3. If you experience an issue where only downloads are functional but uploads are not, it may be due to the AKSK permissions being set to read-only. Verify whether the AKSK has write permissions enabled.