Scenarios Overview

The process of data cloud can be divided into two scenarios: "local data cloud" and "migration from other clouds to BOS". Each scenario has stock and incremental data. object Storage BOS of Baidu AI Cloud provides a variety of data migration methods to minimize additional overhead on the premise of data consistency and seamless switching. You can flexibly choose the data migration method according to your own business situation.

Cloud on Local Data

Stock Data

The characteristic of stock data is that the amount of data is generally relatively large and the migration takes a long time, but the data is basically fixed and rarely modified. When local data goes to the cloud, you can choose to migrate online or offline according to your own network conditions.

Online Mode: BOS CMD Tools

The online way of cloud on local data is suitable for the situation where the customer's local network conditions are good (bandwidth is abundant). In this case, we recommend that the data be uploaded to the cloud through the object upload function of BOS CMD tool.

BOS CMD is a command line tool provided by BOS. It provides rich functions and is convenient for users to operate and manage BOS resources. BOS CMD supports three object upload methods: batch upload, synchronous upload (sync) and single file upload. Using batch upload and synchronous upload (sync) can efficiently solve the problem of local existing data on the cloud online. Here, let's first introduce the batch upload method:

You can use bulk uploads by executing the following command on a machine that already has BOS CMD installed:

$ bcecmd bos cp<local-path><bos-path> --recursive [--storage-class [STORAGE_CLASS]] [--restart] [--quiet] [--yes] [--disable-bar] 

And local-path is the directory where files are uploaded locally. For specific operation and precautions, please refer to: [BOS CMD Object Upload Function](https://cloud.baidu.com/doc/BOS/s/hjwvyqds3 #object%E4%B8%8A%E4%BC%A0)

Offline Mode 1: Offline Migration Box

If the network environment on the customer side is not good (bandwidth is small, network is unstable) and the bandwidth cost is very expensive, the transmission is carried out under limited bandwidth and if the speed is not particularly ideal, it is recommended to use Offline Migration Box for offline transmission.

Offline Migration Box is a TB-level data transmission solution provided by Baidu AI Cloud BOS. It uses storage devices to transmit large amounts of data between Baidu AI Cloud and customers. It solves the problems of high network cost, long transmission time and safety in large-scale data transmission, and has the advantages of simplicity, rapidity, safety and low cost.

Compared with another offline method-"local data offline transmission" provided by BOS,, the Offline Migration Box provides you with a hard disk by BOS. Therefore, the Offline Migration Box is suitable for cases where the migration amount is greater than or equal to 20TB.

Offline Migration Box is currently in the open beta test period, so you need to fill in the open beta test application before using it: Application for Open Beta Test of Offline Migration Box

For specific operation and precautions, please refer to: Offline Migration Box Operation Process

Offline Mode 2: Offline Transmission for Local Data

BOS also provides an offline mode of "offline transmission for local data". The difference between the offline transmission and "Offline Migration Box" is that offline transmission only supports customers to provide their own hard disks. Therefore, offline transmission is suitable for the case where the migration amount is less than 20TB.

Before using offline transmission, please contact us through Ticket or the account manager. After you copy the data to the hard disk, please send the hard disk to the following address:

For the address for hard disk sent back, you can contact the customer manager to inform our interface person. Please choose to guarantee the price when sending the hard disk, and pay the postage at your own expense. You can also choose to pay when sending it back. It can be reached in one day in the Beijing and in two days in different cities. Before the hard disk is sent back, the hard disk will be emptied according to Baidu's security requirements. Please pay attention.

Incremental Data

BOS CMD Sync Function

The synchronous upload (sync) function of BOS CMD can conveniently and quickly solve the problem of incremental data in the process of local data uploaded to the cloud. Sync supports batch operation by default, and can synchronize the local directory to BOS. If there are files with the same name on BOS end and the modification time is newer than the local file, synchronized upload ignores this file and only synchronize the new or the changed files. Sync command will list local and BOS side files at the same time, and perform different actions for different situations to ensure accurate and safe upload of incremental data.

You can use synchronous upload (sync) by executing the following command on a machine that already has BOS CMD installed:

$ bcecmd bos sync<local_dir> bos:/<bucket_name>/[prefix] [--exclude EXCLUDE] [--include INCLUDE] [--delete] [--exclude-delete EXCLUDE-DELETE] [--dryrun] [--yes] [--quiet] [--storage-class STORAGE-CLASS] [--sync-type SYNC-TYPE] [--concurrency CONCURRENCY] [--restart] 

And local_DIR is the local synchronization directory. For specific operation and precautions, please refer to: [BOS CMD Synchronous Upload(sync) Function](https://cloud.baidu.com/doc/BOS/s/hjwvyqds3 #object%E4%B8%8A%E4%BC%A0)

Migrate from Another Cloud

Stock Data

For the scene of customers migrating from other clouds to BOS, object Storage BOS of Baidu AI Cloud provides two migration methods, namely BOS Import tool and online data import function of console. BOS Import tool is characterized by reliability, safety and stability, and is suitable for data migration over 1TB. The online data import function of the console is characterized by convenient and fast operation under the condition of small migration amount, and is suitable for data migration below 1TB.

BOS Import Tools

BOS Import tool is a tool launched by object Storage BOS of Baidu AI Cloud to migrate data from other cloud storage to BOS. It has many advantages such as "good expandability" and "support for hot start and stop of migration tasks". The tool is implemented based on C-S structure, and the client end is responsible for reading the file list, task scheduling and statistics on progress, and all parameters/configuration related to migration are implemented on the client end; the migration task is completed on the server end, and the client end notifies the server through RPC request which files are migrated.

For specific operation method and precautions, please refer to: BOS Import Tool

Online Data Import Function of Console

When users are ready to use Baidu AI Cloud BOS, they often need to migrate data from other cloud storage to Baidu AI Cloud BOS. If the amount of data migrated is large, the traditional manual migration method of "download and upload" is often time-consuming and laborious, and it is difficult to meet the business requirements of rapid migration of users.

Therefore, BOS provides online data import function for the vast number of users. Users only need to configure the import rules, and the system will automatically grab data from the source directory and store it to the designated path on BOS, thus greatly improving the migration efficiency of business data.

The online data import function needs to be operated on the BOS console. For specific operations and precautions, please refer to: Online Data Import

Incremental Data

Image Forward-to-origin

Image forward-to-origin is a bucket-level configuration, which is mainly used to help users to carry out data hot migration without stopping service and solve the problem of incremental data migration from other clouds to BOS.

After a bucket is configured with image forward-to-origin, when the user accesses an object in a BOS and the BOS finds that the object does not exist in the BOS, the BOS will request the object to the forward-to-origin address and store the data returned by the source station into the BOS while returning to the requesting user.

For specific operation method and precautions, please refer to: [BOS Image forward-to-Origin](https://cloud.baidu.com/doc/BOS/s/0jwvyr6bw #%E8%AE%BE%E7%BD%AE%E9%95%9C%E5%83%8F%E5%9B%9E%E6%BA%90)