List files in the storage space
List files in the storage space
The BOS SDK allows users to list objects in the following two ways:
- Simple listing
- Advanced listing via parameters
In addition, users can also simulate folders while listing files.
Simple listing
To quickly and easily list the needed files, users can use the listObjects method, which returns a ListObjectsResponse object containing the results of the listObject request. Users can retrieve details of all objects using the getContents method in the ListObjectsResponse.
1void listObjects(Client& client, const std::string bucketName) {
2 ListObjectsRequest listObjectsRequest(bucketName);
3 // Obtain all object information under the specified bucket
4 ListObjectsResponse listObjectsResponse;
5 int ret = client.list_objects(listObjectsRequest, &listObjectsResponse);
6 if (ret != 0) return;
7 if (listObjectResponse.is_fail()) {
8 return;
9 }
10 // Traverse all objects
11 for (const ObjectSummary& objectSummary : listObjectsResponse.contents()) {
12 std::cout << "ObjectKey: " << objectSummary.key << std::endl;
13 }
14}
Note:
- By default, when a bucket contains more than 1,000 objects, only the first 1,000 will be returned. The is_truncated value in the result will be True, and next_marker will indicate the starting point for the subsequent fetch.
- To increase the number of objects returned, users can utilize the marker parameter to read objects in batches.
Advanced listing via parameters
In addition to simple listing, users can achieve various flexible queries by adjusting the parameters of ListObjectsRequest. The available parameters for ListObjectsRequest include:
| Parameters | Function | Usage method |
|---|---|---|
| prefix | Restrict the returned object keys to those starting with a specific prefix | set_prefix(const std::string& prefix) |
| delimiter | A character that groups object names. All objects with the specified prefix and the first occurrence of the delimiter in their names will be treated as a group element: CommonPrefixes. | set_delimiter(const std::string& delimiter) |
| marker | Set the result to start returning from the first object in alphabetical order after the marker | set_marker(const std::string& marker) |
| max_keys | Set the maximum number of objects to return, with a default and upper limit of 1,000. Specifying a value greater than 1,000 will default to 1,000. | set_max_keys(int maxKeys) |
Note:
- If there is an object named with the prefix, when querying only with prefix, all returned keys will still include the object named with the Prefix. For details, see [Recursively List All Files in a Directory](#Simulate folder function).
- If there is an object named with the prefix, when querying with the combination of prefix and delimiter, there will be Null in all returned keys, and the key names do not contain the prefix. For details, see [View Files and Subdirectories in a Directory](#Simulate folder function).
The following are several examples to illustrate the method of listing via parameters:
Specify the maximum number of returned items
1//Specify the maximum number of returned items as 500
2ListObjectsRequest listObjectsRequest("bucketName");
3listObjectsRequest.set_max_keys(500);
4ListObjectsResponse listObjectsResponse;
5int ret = client.list_objects(listObjectsRequest, &listObjectsResponse);
6if (ret != 0) return;
7 ...//Response exception handling
8for(const ObjectSummary& objectSummary :listObjectsResponse.contents()) {
9 std::cout << "ObjectKey: " << objectSummary.key << std::endl;
10}
Return objects with a specified prefix
1//Specify to return objects with the prefix “test"
2ListObjectsRequest listObjectsRequest("bucketName");
3listObjectsRequest.set_prefix("test");
4ListObjectsResponse listObjectsResponse;
5int ret = client.list_objects(listObjectsRequest, &listObjectsResponse);
6if (ret != 0) return;
7 ...//Response exception handling
8for(const ObjectSummary& objectSummary :listObjectsResponse.contents()) {
9 std::cout << "ObjectKey: " << objectSummary.key << std::endl;
10}
Return from after a specified object
1//Users can define to exclude a certain object and start returning from after it
2ListObjectsRequest listObjectsRequest("bucketName");
3listObjectsRequest.set_marker("object");
4ListObjectsResponse listObjectsResponse;
5int ret = client.list_objects(listObjectsRequest, &listObjectsResponse);
6if (ret != 0) return;
7 ...//Response exception handling
8for(const ObjectSummary& objectSummary :listObjectsResponse.contents()) {
9 std::cout << "ObjectKey: " << objectSummary.key << std::endl;
10}
Pagination to get all objects
1//Users can set a maximum of 500 records per page
2ListObjectsRequest listObjectsRequest("bucketName");
3listObjectsRequest.set_max_keys(500);
4ListObjectsResponse listObjectsResponse;
5int ret = client.list_objects(listObjectsRequest, &listObjectsResponse);
6if (ret != 0) return;
7 ...//Response exception handling
8bool isTruncated = true;
9while (isTruncated) {
10 ListObjectsResponse listObjectsResponse;
11 client.list_objects(listObjectsRequest, &listObjectsResponse);
12 isTruncated = listObjectsResponse.is_truncated();
13 if (listObjectsResponse.next_marker() != "") {
14 listObjectsRequest.set_marker(listObjectsResponse.next_marker());
15 }
16}
Results of paginated restoration of all objects after a specific object
1//Users can set a maximum of 500 records per page and start getting from after a specific object
2ListObjectsRequest listObjectsRequest("bucketName");
3listObjectsRequest.set_max_keys(500);
4listObjectsRequest.set_marker("object");
5ListObjectsResponse listObjectsResponse;
6int ret = client.list_objects(listObjectsRequest, &listObjectsResponse);
7if (ret != 0) return;
8bool isTruncated = true;
9while (isTruncated) {
10 ListObjectsResponse listObjectsResponse;
11 client.list_objects(listObjectsRequest, &listObjectsResponse);
12 isTruncated = listObjectsResponse.is_truncated();
13 if (listObjectsResponse.next_marker() != "") {
14 listObjectsRequest.set_marker(listObjectsResponse.next_marker());
15 }
16}
Results of paginated restoration of all objects after a specific object
1//Users can set pagination to get objects with a specified prefix, with a maximum of 500 records per page
2ListObjectsRequest listObjectsRequest("bucketName");
3listObjectsRequest.set_max_keys(500);
4listObjectsRequest.set_prefix("object");
5ListObjectsResponse listObjectsResponse;
6int ret = client.list_objects(listObjectsRequest, &listObjectsResponse);
7if (ret != 0) return;
8bool isTruncated = true;
9while (isTruncated) {
10 ListObjectsResponse listObjectsResponse;
11 client.list_objects(listObjectsRequest, &listObjectsResponse);
12 isTruncated = listObjectsResponse.is_truncated();
13 if (listObjectsResponse.next_marker() != "") {
14 listObjectsRequest.set_marker(listObjectsResponse.next_marker());
15 }
16}
The parameters available for calling in the parsing class of ListObjectsResponse returned by the list_object method are:
| Parameters | Description |
|---|---|
| name | Bucket name |
| prefix | The matched objects from prefix to the first Delimiter character are returned as a group of elements |
| marker | Starting point of this query |
| max_keys | Maximum number of requests returned |
| is_truncated | Indicate whether all queries are returned; false - all results are returned this time; true - not all results are returned this time |
| contents | A container for the returned objects |
| +key | Object name |
| +last_modified | Last modification time of the object |
| +etag | The HTTP protocol entity tag of object. |
| +storage_class | Storage class of object |
| +size | The size of the object content (in bytes) |
| +owner_id | User ID of bucket owner |
| +owner_display_name | Name of bucket owner |
Simulate folder function
In BOS storage, there is no concept of folders. All elements are stored as objects. However, when using data, BOS users often need to manage files in folders. Therefore, BOS provides the ability to create simulated folders, which essentially involves creating an object with a size of 0. This object can be uploaded and downloaded, but the console will display objects ending with “/” as folders.
By combining the delimiter and prefix parameters, users can simulate folder functionalities. The combined effect of delimiter and prefix is as follows:
If prefix is set to a folder name, files starting with the prefix can be listed, including all files and subfolders (directory) recursively listed under the folder. File names are displayed in contents. If delimiter is set to /, the return values only list the files and subfolders (directory) under the folder. The names of subfolders (directory) under the folder are returned in the CommonPrefixes. The files and folders recursively listed under the subfolders are not displayed.
The following are several application methods:
List all files in the bucket
When users need to get all files under a bucket, they can refer to [Pagination to Get All Objects](#Advanced listing via parameters)
Recursively list all files in a directory
You can obtain all files in the a directory by setting the Prefix parameter:
1// Construct a ListObjectsRequest request
2ListObjectsRequest listObjectsRequest("bucketName");
3 // Recursively list all files under the fun directory
4listObjectsRequest.set_prefix("fun/");
5ListObjectsResponse listObjectsResponse;
6int ret = client.list_objects(listObjectsRequest, &listObjectsResponse);
7if (ret != 0) return;
8 ...//Response exception handling
9std::cout << "ObjectKeys:" << std::endl;
10 // Traverse all objects
11for(const ObjectSummary& objectSummary :listObjectsResponse.contents()) {
12 std::cout << objectSummary.key << std::endl;
13}
Output:
1ObjectKeys:
2fun/
3fun/movie/001.avi
4fun/movie/007.avi
5fun/test.jpg
View files and subdirectories in a directory
With the combination of Prefix and Delimiter, it can list files and subdirectories under the directory:
1// Construct a ListObjectsRequest request
2ListObjectsRequest listObjectsRequest("bucketName");
3 // "/" is the delimiter for folders
4listObjectsRequest.set_delimiter("/");
5 //List all files and folders under the “fun” directory
6listObjectsRequest.set_prefix("fun/");
7ListObjectsResponse listObjectsResponse;
8int ret = client.list_objects(listObjectsRequest, &listObjectsResponse);
9if (ret != 0) return;
10 // Traverse all objects
11std::cout << "\nObjectKeys:" << std::endl;
12for(const ObjectSummary& objectSummary :listObjectsResponse.contents()) {
13 std::cout << objectSummary.key << std::endl;
14}
15
16 // Traverse all CommonPrefix
17std::cout << "CommonPrefixs:" << std::endl;
18for (const std::string& commonPrefix : listObjectsResponse.common_prefixes()) {
19 std::cout << commonPrefix << std::endl;
20}
Output:
1ObjectKeys:
2fun/
3fun/test.jpg
4CommonPrefixs:
5fun/movie/
In the returned results, the list contents of ObjectSummary provides the files under the “fun” directory. The list in common_prefixes shows all subfolders under the fun directory. It can be seen that the two files fun/movie/001.avi and fun/movie/007.avi are not listed because they belong to the movie directory under the fun folder.
List storage properties of objects in a bucket
After users complete the upload, if they need to view the storage class properties of all objects in a specified bucket, it can be achieved through the following code:
1public void listObjectsStorageClass(){
2 ListObjectsRequest listObjectsRequest("bucketName");
3 ListObjectsResponse listObjectsResponse;
4 int ret = client.list_objects(listObjectsRequest, &listObjectsResponse);
5 std::vector<ObjectSummary> objectList = listObjectResponse.contents();
6 for(int i = 0; i<objectList.size(); i++) {
7 std::cout << objectList[i].storage_class() << std::endl;
8 }
9}
