Metadata Conversion Service

Overview

The BOS metadata conversion service allows the direct transformation of a standard BOS bucket into a hierarchical namespace (HNS) bucket without the need for additional tools. HNS buckets retain most features of flat buckets and are compatible with most HDFS file system APIs. They also enable atomic directory operations such as renaming and deleting while optimizing the latency of list and head requests, making them ideal for data lake computing in big data and AI scenarios.

Prerequisites

1. Currently, the metadata conversion service and HNS buckets are available only in Beijing, Suzhou, Guangzhou, and Baoding regions.
2. To enable the BOS metadata conversion feature, you need to submit a support ticket for access approval.

Usage restrictions

1. Buckets with trash or Versioning features enabled, or those containing archived objects, symlinks, or object tags, are not eligible for conversion.
2. Once a flat bucket is successfully converted through metadata conversion, the hierarchical namespace cannot be disabled. The process is irreversible, and only the metadata organization format changes during conversion, with no data being moved.
3. The completion time of a metadata conversion task depends on the number of objects in the bucket and the availability of backend resources. Currently, there is no precise guarantee for the task's completion time.

Risk statement

• Once the conversion task begins and until it is completed, avoid incremental writes, overwrites, or deletions of existing data to prevent data loss. If archived storage class files are encountered during conversion, the task will fail and automatically abort.
• Ensure that the bucket being converted does not contain files and folders with the same name (e.g., both a file named dir and a folder named dir/). Conflicts arising from such scenarios are linked to the order of import during task execution. The service resolves conflicts through renaming; for more details, refer to the conversion rules.
• Refer to the conversion rules to verify there are no files with specially reserved filenames in the BOS bucket, and ensure no such files are added during conversion. Otherwise, files with the same name may overwrite each other, leading to file loss.
• Before starting a bucket metadata conversion, back up your data using methods such as replication. Once the conversion task is completed, verify the integrity of the bucket data.

Conversion rules

• For files ending with / before conversion (e.g., a/b/): If the file is empty (size = 0), a directory with the same name (e.g., a/b) will be directly created post-conversion. If the file contains data, it will be renamed with a special suffix .slash along with the conversion timestamp. For instance, the file a/b/ will become a/b.slash.1716487553. The newly converted filename can be viewed through the ListObjects API.

• In HNS buckets, files and directories with the same name cannot coexist. Thus, if the original bucket contains both a file a/b and a directory a/b/, a conflict will occur during conversion due to the conversion order. The primary rule prioritizes the creation and preservation of directories.

  • Scenario I: During the creation of directory a/b, if a file with the same name (a/b) already exists, it will be renamed with a special suffix .file along with the conversion timestamp. For example, the file a/b will become a/b.file.1716487553. If the new filename already exists, a numeric suffix will be tried once (e.g., a/b.file.1716487553.1).
  • Scenario II: During the creation of file a/b, if a directory with the same name (a/b) already exists, the file will be attempted under a new name with the special suffix .file and a conversion timestamp (e.g., a/b.file.1716487553). If this name also exists, a numeric suffix will be added, retrying up to three times (e.g., a/b.file.1716487553.1).