Migrating Data from Third-Party Clouds to BOS
Overview
If you need to transfer data from another cloud service provider's object storage to BOS, BOS offers several migration solutions. Choose the method that best suits your requirements.
Requirement scenarios
Requirement Scenario 1: Migration of historical data from other cloud service providers
Historical data typically involves large volumes. When migrating historical stock data from other clouds to BOS, Baidu AI Cloud Object Storage (BOS) offers two migration options: the BOS Import tool and the CloudFlow data flow platform.
Requirement Scenario 2: Migration of incremental data from other cloud service providers
During the migration of historical data, the source might continue to generate new data. BOS provides migration and synchronization solutions for such incremental data. You can choose from the BOS Import tool, the CloudFlow data flow platform, or BOS mirror back-to-origin.
Solution overview
For cases where customers migrate from other clouds to BOS, Baidu AI Cloud Object Storage (BOS) offers three migration methods: the CloudFlow data flow platform, the BOS Import tool, and mirror back-to-origin.
- Data Flow Platform CloudFlow (recommended): It is a fully managed migration platform for storage data launched by Baidu AI Cloud Storage. It supports visual creation of migration tasks and viewing the progress of migration tasks. You can initiate cross-cloud migration of historical stock data and incremental data from other cloud service providers through the data flow platform, and also migrate local self-built object storage clusters to the cloud. CloudFlow supports data migration in public network and dedicated line environments.
- BOS Import Script Tool: It is reliable, secure, and stable. You can download and install it from the BOS official website to your local machine, and initiate data migration after configuration. It is suitable for data migration of more than 1 TB;
- BOS mirror back-to-origin: You can configure mirror back-to-origin for the bucket. When you access an object in the bucket that does not exist in BOS, BOS will immediately request the object from the pre-configured back-to-origin address to ensure normal access. For details, please refer to Bucket Trash Function1. Files moved to the bucket trash will continue to be charged until they are completely deleted by the user.
Practical operations
Cross-cloud migration of historical stock data
Method 1: CloudFlow data flow platform (recommended)
Data Flow Platform CloudFlow is a fully managed migration platform for storage data launched by Baidu AI Cloud Storage. It realizes data migration from third-party clouds to Baidu AI Cloud through visual configuration. It supports migration of third-party object storage data to Baidu AI Cloud object storage, and allows viewing the progress and results of migration tasks. For failed migration tasks, it supports re-initiating migration retries, and also supports operations such as interrupting and deleting migration tasks.
The data flow platform supports the following functions:
- Task creation: Before starting a data migration process, you need to create a corresponding migration task first;
- Supported sources: The data flow platform supports migration from Alibaba Cloud OSS, Tencent Cloud COS, Kingsoft Cloud KS3, Volcano Cloud TOS, Qiniu Cloud KODO, Huawei Cloud OBS, AWS China region, and local self-built S3 clusters to Baidu AI Cloud Object Storage (BOS).
- Task management: After the migration task is created, you can manage the task, such as interrupting, deleting, downloading failed files, restarting, etc.;
- Task progress viewing: After the migration task is created, you can view the migration progress of the task, such as the number of files that have been migrated, the number of successfully migrated files in progress, the number of failed files, the number of files to be migrated, etc.;
- Supports both stock data migration and incremental data migration.
You can log in to the Baidu AI Cloud BOS console, click Data Migration in the left navigation to create a migration task, or directly log in to the CloudFlow Console to create a migration task and initiate data migration. For detailed migration steps, please refer to CloudFlow Help Document.
Method 2: BOS import tool
The BOS Import tool, launched by Baidu AI Cloud Object Storage (BOS), facilitates data migration from other cloud storage services to BOS. It boasts features like "excellent scalability" and "support for hot start and stop of migration tasks." Based on the C-S structure, the client handles tasks such as reading the file list, scheduling tasks, and progress updates, while all migration-related parameters and configurations are set on the client side. Migration tasks are executed on the server side, with the client using RPC requests to notify the server about files to migrate.
For specific usage methods and precautions, please refer to: BOS Import Tool
Cross-cloud migration of incremental data at the source
You can achieve cross-cloud synchronization of incremental data by setting up mirror back-to-origin rules or utilizing the CloudFlow data flow platform.
Method 1: CloudFlow data flow platform
Data Flow Platform CloudFlow is a fully managed migration platform for storage data launched by Baidu AI Cloud Storage. It realizes data migration from third-party clouds to Baidu AI Cloud through visual configuration. It supports cross-cloud migration and synchronization for both existing and incremental data.
Method 2: Mirror back-to-origin
Mirror back-to-origin is a bucket-level configuration primarily designed to help users perform hot data migration without service interruptions and to handle the migration of incremental data from other clouds to BOS.
When mirror back-to-origin is configured for a bucket, if a user accesses (GetObject) an object in BOS and the object is not found, BOS will retrieve the object from the back-to-origin address. It then saves the data obtained from the source to BOS and delivers it to the requesting user.
For specific usage methods and precautions, please refer to: BOS mirror back-to-origin
Related products
- Data Flow Platform CloudFlow - Cross-cloud Migration is a fully managed migration platform for storage data launched by Baidu AI Cloud Storage. It realizes data migration from third-party clouds to Baidu AI Cloud through visual configuration. It supports migration of third-party object storage data to Baidu AI Cloud object storage, and allows viewing the progress and results of migration tasks. For failed migration tasks, it supports re-initiating migration retries, and also supports operations such as interrupting and deleting migration tasks. Support migration from Alibaba Cloud OSS, Tencent Cloud COS, Kingsoft Cloud KS3, Volcano Cloud TOS, Qiniu Cloud KODO, Huawei Cloud CBS, AWS China region, and local self-built S3 clusters to Baidu AI Cloud Object Storage (BOS).
