Hierarchical namespace

Function overview:

Hierarchical namespace is a metadata management feature provided by Baidu AI Cloud. It supports seamless directory-level mv and rename operations, optimizing common read operations like list and head. This significantly enhances data processing efficiency and performance, making it ideal for data lake computing scenarios in the big data domain.

Feature activation

• Step 1: Submit a request outlining your requirements, and the system will grant the necessary access permissions.
• Step 2: Once permissions are granted, create a bucket with the hierarchical namespace feature enabled, using BOS tools or by accessing the console.

Feature architecture:

Advantages:

1. Directly leverage the advantages of object storage in big data scenarios: automated data management, low storage costs, and strong data security

2. Natively object-based hierarchical namespace delivers high performance for file semantics in big data computing scenarios

   • Support high-performance atomic operations such as Rename, Delete Dir, and Delete Object
   • Significantly lower List/Head request latency compared to flat object storage
3. Big data analytics directly based on object storage (“All in Object”), eliminating additional costs for data replication, transfer, storage, and third-party services
4. No extra costs for O&M, management, or learning - fully utilize the ecosystem and tools of object storage

Application scenarios:

Big data analytics and computing

As the leading cloud storage solution, Baidu AI Cloud Object Storage (BOS) offers a suite of high-performance big data computing solutions. In big data applications, BOS excels as the underlying storage system for on-premises or cloud-based computing clusters like MapReduce, Hive, and Spark offline data warehouse clusters. Its built-in hierarchical namespace functionality significantly decreases the latency of large-scale batch operations (e.g., Rename, List, Head) by roughly 70%, effectively overcoming traditional object storage latency bottlenecks in big data computing.

Usage method:

Currently, the hierarchical namespace feature is only supported for newly created buckets in the following regions: North China - Beijing, South China - Guangzhou. Once enabled, the hierarchical namespace feature cannot be disabled. For detailed steps to enable hierarchical namespace via the console, refer to [Console Operation Guide - Instructions for Hierarchical Namespace Buckets](BOS/Console Operation Guide/Hierarchical Namespace Bucket Operation Instructions.md).

You can create, rename, or delete directories within an HNS-enabled bucket using the methods provided below.

Feature support and limitations

Support

• File and directory semantics and operations
• Atomic directory operations and millisecond-level rename
• Optimized read performance
• Most features of flat object storage

Restriction

HNS-enabled buckets support nearly the same features as flat-namespace buckets: basic object-level operations (e.g., upload, download, copy); bucket-level configuration features (e.g., lifecycle management, event notification, log management). Currently, a small number of features are not supported compared to flat-namespace buckets. Submit a ticket if you have requirements for these features. Details are as follows:

Bucket-related features

1. Bucket trash is not supported

Object-related features

1. Archive storage class files are not supported
2. Uploading non-empty directories (i.e., files with non-empty content whose names end with /) is not supported
3. File symlink functionality is not supported
4. Object tagging is not supported

API adjustments

Due to the hierarchical namespace feature, some BOS APIs have compatibility and semantic adjustments. Details are as follows:

ListObjects

For flat-namespace buckets, the request parameters of the ListObjects API under hierarchical namespace are adjusted. See the table below for details. For information about ListObjects in flat namespaces, refer to ListObjects API Description.

Special note

ListObjects returns all sub-directories and objects under the parent directory, sorted lexicographically.

It is no longer possible to list all files under a specific prefix (directory) in the directory of an HNS-enabled bucket. Only sub-directories and sub-files can be listed based on the parent directory information provided.