Historical Inventory Data

The historical inventory data refers to the data generated at this time. The data features a large volume and long migration time, but it is basically fixed and rarely modified. During the local data cloudification, you can select the online or offline cloudification according to your network conditions.

Online mode: BOS CMD tool

The online mode of local data cloudification is suitable for customers with excellent local network conditions (abundant bandwidth). In this case, we recommend that you implement the data cloudification through the object upload feature of the BOS CMD tool.

BOS CMD is a command-line tool provided by BOS, which provides rich functions to facilitate users to operate and manage BOS resources. BOS CMD supports three object upload modes, i.e., batch upload, synchronous upload (sync), and single file upload. The batch upload and synchronous upload (sync) can effectively solve the problem in the online cloudification of the local stock data. Here, we first introduce the batch upload mode:

You can use the batch upload by executing the following command on the machine with BOS CMD installed:

$ bcecmd bos cp <local-path> <bos-path> --recursive [--storage-class [STORAGE_CLASS]] [--restart] [--quiet] [--yes] [--disable-bar]

local-path is the directory of local upload files. For the specific operations and precautions, see BOS CMD Object Upload Function

Offline mode 1: Offline Migration Box

The Offline Migration Box is a physical delivery cloudification mode launched by BOS, which can accomplish the migration cloudification of the local IDC data.

Application scenarios

• Your network environment is poor (narrow bandwidth and unstable network), and the bandwidth cost is very expensive.
• You want to complete the data cloudification as soon as possible, but the cost of ET mode is too expensive.
• You have no hard disk or removable data storage and transmission devices.

The Offline Migration Box is a TB-level data transmission solution provided by Baidu AI Cloud BOS, which uses the storage device to transmit large amounts of data between Baidu AI Cloud and the customer. It solves the problems such as high network cost, long transmission time, and security during large-scale data transmission and features simplicity, high speed, security, and low cost.

The Offline Migration Box is provided with a mobile storage device by BOS, which features large capacity, visual operation, and military-level protection encryption. The nominal capacity of a single Offline Migration Box is 96 TB, and the actual available capacity is about 83 TB. It is applicable for the migration scenarios of 100 TB local IDC data cloudification.

How to Use

The Offline Migration Box is in the open beta test stage. You need to fill in the application for open beta test before use: Apply for the open beta test of Offline Migration box, or submit the Ticket to contact us. We match the appropriate service time and service cycle for you according to the current equipment usage. Please tell us your estimated data volume (such as 300TB), IDC city, and expected service time.

Product introduction

For detailed introductions, operation process, and precautions of Offline Migration Box, see Introduction and Operation Process of Offline Migration Box.

Offline mode 2: Self-owned hard disk delivery

BOS also provides an offline mode - self-owned hard disk delivery. The difference from the Offline Migration Box is that the device needs to be prepared by customers. You can write the data to the self-owned hard disk, and deliver the hard disk to the Baidu AI Cloud data center. We implement data cloudification for you free of charge. This scheme is applicable for the scenario of 100 TB level local IDC data cloudification. BOS is provided with multiple fixed devices in the data center for hard disk cloudification. Up to 16 hard disks to mount and data cloudification are supported for each device at the same time. Compared with the Offline Migration box, the self-owned hard disk delivery mode is higher speed.

Application scenarios

The self-owned hard disk delivery mode is applicable for scenarios where customers have more hard disks and urgent requirements for data cloudification. For example,

• The local IDC needs to be decommissioned, and it is necessary to find a suitable storage location of data on the cloud as soon as possible.
• The local data scale is large, and users require data backup and cloudification as soon as possible. However, the cost of the ET mode is too expensive, so users hope to find other ways with higher cost performance.

Usage Limitations

• It only supports the national standard 3.5’’ or 2.5’’ mobile hard disk.
• It only supports the USB interface hard disk. Many 3.5’’ hard disks with large capacity are only provided with the SATA interface. You need to purchase the corresponding hard disk box and deliver it together with the hard disk to the Baidu data center. The hard disk box can be purchased through major online shopping malls. You can search for "SATA to USB Hard Disk Box" during purchase. A hard disk needs to match with a hard disk box during uploading. Please purchase the corresponding number of hard disk boxes according to the number of your hard disks. If the number of hard disk boxes is less than that of hard disks, you can implement the data copy on the cloud of the next batch of hard disks only when the data in the hard disk on the current hard disk box is uploaded successfully.
• It only supports the ext4, ntfs, and xfs file systems. You can view the file system details supported by Linux through man 5 fs.
• Upon the completion of the data copy, the hard disks and hard disk boxes are delivered back to you. All data in the hard disk is formatted before delivery. If the data is important, prepare the data backup in advance;
• After the hard disks are delivered to the Baidu data center, the technical personnel comprehensively check whether there is any bad track for the hard disk immediately. If there is any bad track resulting in the hard disk being unreadable, this hard disk is directly formatted before the return.

Usage cost

• All freight and insurance costs for hard disk delivery shall be borne by customers, including logistics cost for delivery of hard disk to Baidu data center and logistics cost for delivery back of hard disk after successful uploading. After successful uploading, BOS delivers back the hard disk in cash on delivery.
• We recommend that you have your hard disk insured.
• The data center server usage fee, network bandwidth fee, and technical personnel support service fee generated during data uploading are free of charge.

Operating process

Step 1: If you use your own hard disk to deliver the data cloudification, please contact us through the Ticket or account manager, and inform of your data volume (such as 30 TB in total), number of hard disks (such as 30 portable hard disks), hard disk box included or not (if it is a SATA interface hard disk, please prepare your hard disk box in advance), the estimated delivery time of hard disk to the Baidu data center and how long you want to upload it.

Step 2: You need to copy the data to the hard disk and carry out the data check and data backup of important data.

Step 3: Deliver the hard disk and hard disk box (if included) to the following address:

Step 4: Inform us of the bucket name you need to upload, expected directory structure, and your AK/SK through your account manager or Ticket, and we upload the data for you. It is recommended that you create the bucket and subdirectory in advance and then create AK/SK, but do not use the default AK/SK.

Step 5: The technical personnel of BOS receives the hard disk and then starts the data uploading after check. After successful uploading, we confirm the number of files with you through the Ticket or account manager. After the confirmation, all data on the hard disk is formatted, and then the hard disk and hard disk box (if included) are delivered back to you.

Step 6: After the successful uploading, you can manage and use the data through the BOS console, API, and SDK, and delete the previously provided AK/SK?.

Incremental Data

The incremental data refers to the data generated at this time or in the future. This data is generally the online data generated by your business, which is real time.

BOS CMD Sync feature

The BOS CMD sync function can easily and quickly solve the problem of incremental data during local data cloudification. The synchronous upload supports the batch operation by default. You can synchronize the local directory to BOS. If the file with the same name already exists on the BOS side and the modification time is later than that of the local file, this file is ignored for synchronous upload, and only the new file or modified file is synchronized. The sync command lists the local and BOS side files at the same time and performs different actions for different situations to ensure the accurate and secure upload of incremental data.

You can use the synchronous upload (sync) by executing the following command on the machine with BOS CMD installed:

$ bcecmd bos sync <local_dir> bos:/<bucket_name>/[prefix] [--exclude EXCLUDE] [--include INCLUDE] [--delete] [--exclude-delete EXCLUDE-DELETE] [--dryrun] [--yes] [--quiet] [--storage-class STORAGE-CLASS] [--sync-type SYNC-TYPE] [--concurrency CONCURRENCY] [--restart]

Where local_DIR is the local synchronization directory. For specific operation and precautions, see BOS CMD Synchronous Upload Function (Sync)