Migrating Local Data to BOS

Overview

Practical steps for migrating local data to the cloud: If your data is stored in a self-managed IDC or a locally managed server room, follow this guide for cloud data migration.

Requirement scenarios

Requirement Scenario 1: Migration of historical stock data to the cloud

Stock data typically involves a large volume and takes a long time to migrate. However, the data is usually fixed with minimal changes. Based on your network conditions, you can choose between online and offline migration methods for transferring local data to the cloud.

Requirement Scenario 2: Migration of incremental data to the cloud

During historical stock data migration, new data may still be written to your source location. For such incremental data, BOS provides specific cloud migration solutions.

Solution overview

BOS offers several methods for data migration, such as command-line CMD tools, MoonBox devices, and hard disk delivery. Choose the migration option that best suits your requirements.

Practical operations

Operations for historical stock data

Method 1: Online migration to the cloud using BOS CMD tool

The online method for migrating local data to the cloud works well when the customer’s local network conditions are favorable (sufficient bandwidth). In such cases, we recommend using the object upload feature of the BOS CMD tool for cloud data migration.

BOS CMD is a command-line tool provided by BOS that offers extensive features for managing and operating BOS resources. It supports three object upload methods: batch uploads, sync uploads, and single-file uploads. Batch and sync uploads are particularly effective for migrating local stock data to the cloud. Let’s first explore the batch upload method:

You can execute the following command on a machine with BOS CMD installed to use batch upload:

1$ bcecmd bos cp <local-path> <bos-path> --recursive [--storage-class [STORAGE_CLASS]] [--restart] [--quiet] [--yes] [--disable-bar]

Where local-path is the directory of the local files to be uploaded. For specific operations and precautions, please refer to: BOS CMD Object Upload Function

Method 2: Cloud migration via MoonBox copy and physical delivery

If the customer's network is poor (low bandwidth or unstable), and the cost of bandwidth is high, making data transmission slow, using MoonBox for offline transmission is recommended.

MoonBox is a TB-level data transfer solution offered by Baidu AI Cloud Object Storage (BOS). By using physical storage devices, MoonBox enables large-scale data transfers between Baidu AI Cloud and customers efficiently, securely, and at a low cost, addressing high network costs, long transmission times, and security concerns.

Compared to another offline migration option, “Local Data Offline Transmission,” provided by BOS, MoonBox includes BOS-provided hard disks and is more suitable for migrating data volumes of 20 TB or larger.

MoonBox is in the public beta period. You need to fill in a public beta application before using it: MoonBox Public Beta Application

For specific operation procedures and precautions, please refer to: MoonBox Operation Process

Method 3: Cloud migration via self-owned hard disk delivery

BOS also offers an offline method called "Local Data Offline Transmission." Unlike "MoonBox," this method requires customers to provide their own hard drives. As such, offline transmission is more suitable for migrating data volumes of less than 20 TB.

Before using offline transmission, please contact us through work order or account manager first. After you copy the data to the hard disk, please send the hard disk to the following address:

To arrange the return address for the hard disk, you can contact the account manager to notify our designated contact. For hard drive delivery, please choose insured shipping, and you will need to cover the postage. The return delivery will also be on a cash-on-delivery basis. It takes approximately one day for intra-city delivery within Beijing and around two days for other locations. Please be aware that, according to Baidu's security policies, the hard drives will be uniformly wiped before being returned.

Operations for source incremental data

Method 1: BOS CMD sync function

The synchronous upload (sync) feature of BOS CMD provides a convenient and efficient way to handle incremental data during the migration process of local data to the cloud. By default, synchronous upload supports batch operations and enables syncing of local directories to BOS. If a file with the same name exists on the BOS side and its modification time is more recent than the local file, the sync function will skip uploading that file, only transferring new or modified files. The sync command compares files on both the local and BOS sides, performing appropriate actions for different scenarios to ensure accurate and secure incremental data upload.

You can execute the following command on a machine with BOS CMD installed to use synchronous upload (sync):

1$ bcecmd bos sync <local_dir> bos:/<bucket_name>/[prefix] [--exclude EXCLUDE] [--include INCLUDE] [--delete] [--exclude-delete EXCLUDE-DELETE] [--dryrun] [--yes] [--quiet] [--storage-class STORAGE-CLASS] [--sync-type SYNC-TYPE] [--concurrency CONCURRENCY] [--restart]

Where local_DIR is the local synchronization directory. For specific operations and precautions, please refer to: BOS CMD Synchronous Upload (sync) Function

Related products

• Data Flow Platform CloudFlow - MoonBox: MoonBox is suitable for the transmission and migration of hundreds of TB-level data to the cloud. It uses specific storage devices to transfer large amounts of data between Baidu AI Cloud and customers. It solves the problems of high network costs, long transmission time, and security in large-scale data transmission, and has the advantages of simplicity, speed, security, and low cost.

MoonBox provides mobile storage devices from BOS with features such as large capacity, user-friendly interface, military-grade protection, and encryption. Each MoonBox device has a calibrated capacity of 96 TB and an actual usable capacity of approximately 83 TB, making it ideal for scenarios requiring local IDC data migration to the cloud at scales reaching hundreds of terabytes.

• Baidu AI Cloud Object Storage (BOS) - Cloud Migration via Hard Disk Delivery: The difference between it and “MoonBox” is that the devices need to be prepared by the customer themselves. You can write data to your own hard disks, send the hard disks in batches to Baidu AI Cloud computer rooms, and we will transfer the data to the cloud for free. This solution is suitable for local IDC data migration to the cloud scenarios at the hundreds of TB level.
• The synchronous upload (sync) function of BOS CMD: Conveniently and quickly solve the problem of incremental data during the process of local data migration to the cloud. Synchronous upload supports batch operations by default, and can synchronize local directories to BOS. If there is a file with the same name on the BOS side and its modification time is newer than that of the local file, synchronous upload will ignore this file and only synchronize new files or modified files. The sync command will list files on both the local and BOS sides, and perform different actions for different situations to ensure the accurate and secure upload of incremental data.