百度智能云

All Product Document

          Object Storage

          File Management

          Upload File

          The basic data unit for user operationS is an object in BOS. The object contains key, meta, and data, of which the key is the name of the object, the meta is the user's description of this object which is composed of a series of Name-Value pairs. The data is that of the object.

          BOS C++ SDK provides abundant file upload interfaces. You can upload files in the following ways:

          • Simple upload
          • Additional upload
          • Multipart upload
          • Resumeable upload

          Simple upload

          In the simple upload scenario, BOS allows you to upload objects in the form of specified file, data stream, file descriptor, and string by referring to the following codes:

          int PutObjectDemo(Client& client,const std::string& bucketName, const std::string objectKey){
              // Get the file data flow
              FileInputStream inputStream("/path/to/test.zip"); // bcesdk/util/util.h
          	int ret = 0;
              // Upload objects with the file name as parameter
              ret = client.upload_file(bucketName, objectKey, "/path/to/test.zip");
              // Upload objects in the form of data stream
              ret = client.upload_file(bucketName, objectKey, inputStream);
              // Upload objects in the form of file descriptor    
              fd_t fd = open("/path/to/test.zip", O_RDWR, 0666);//Under linux, fd_t is defined in common.h
              ret = client.upload_file(bucketName, objectKey, fd);
              // Upload objects in the form of string
              std::string data = "this is data";
          	ret = client.put_object(bucketName, objectKey, data);
              return ret;
          }

          The object is uploaded to BOS in the form of file. The put_object and upload_file functions can be used to upload the object of no more than 5GB. To upload the large files (greater than 5GB), you need to use upload_super_file, whose reference codes are as follows:

          int PutLargeObjectDemo(Client& client,const std::string& bucketName, const std::string objectKey){
              std::string fileName = "/path/to/test.zip"
              return client.upload_super_file(bucketName, objectKey, fileName);// The third parameter can be also the fd_t form.
          }

          Set Object Metadata

          The object metadata describes the file attribute when the user uploads the file to BOS, mainly including HTTP standard attribute (HTTP headers) and user-defined metadata.

          • Set HTTP Header of Object

          In essence, BOS C++ SDK calls the backend HTTP interface. Therefore, users can customize the HTTP header of objects when they upload files. The common HTTP headers are described as follows:

          Name Description Default Value
          Content-MD5 File data check. After setting, BOS enables the MD5 check of the file content to compare the MD5 provided by you with the MD5 of the file. If they are not consistent with each other, an error is thrown. None
          Content-Type MIME of the file, which defines the file type and webpage coding, and determines in what form and coding the browser reads the file. If it is not specified, BOS automatically generates it according to the file extension. If the file has no extension, enter the default value. application/octet-stream
          Content-Disposition Indicate how the MIME user agent displays, opens, or downloads attached files and file names. None
          Content-Length Length of the file uploaded, which is truncated if it exceeds the length of the stream/file and is the actual value if it is less than the length of stream/file. Time length of the stream/file
          Expires Cache expiration time None
          Cache-Control Specify the cache behavior of the webpage when this object is downloaded. None

          The reference code is as follows:

          ...
          // Initialize meta
          ObjectMetaData meta;
          
          //Set the ContentType
          meta.set_content_type("application/json");
          
          // Set cache-control
          meta.set_cache_control("no-cache");
          
          //Set x-bce-storage-class
          meta.set_storage_class("STANDARD");
          
          ret = client.upload_file(bucketName, objectKey, content, meta);
          ...
          • User-defined Meta Information

          BOS can use the custom metadata to describe objects, as shown in the following code segment:

          // Set the value of custom metadata name to my-data.
          meta.set_user_meta("name", "my-data");
              
          //Upload Object
          ret = client.upload_file(bucketName, objectKey, file_name, meta);

          Tips:

          • In the above codes, the user customizes metadata with a "name" and the value of "my-data".
          • When users download this object, they can also get this metadata together.
          • An object can have multiple similar parameters, but the total size of all user metadata cannot exceed 2 KB.

          Set Copy Attribute of Object

          BOS also provides the CopyObject API to copy an existing object to another object. During the copy, it judges the Etag or modification status of the source object and decides whether the object is copied according to the judgment results. The detailed parameter interpretation is as follows:

          Name Type Description Required?
          x-bce-copy-source-if-match std::string If the Etag of the source object is equal to the Etag provided by the user, the copy operation is performed. Otherwise, the copy fails. No
          x-bce-copy-source-if-none-match std::string If the Etag of the source object is not equal to the Etag provided by the user, the copy operation is performed. Otherwise, the copy fails. No
          x-bce-copy-source-if-unmodified-since std::string If the source object is not modified after the x-bce-copy-source-if-unmodified-since, perform the copy. Otherwise, the copy fails. No
          x-bce-copy-source-if-modified-since std::string If the source object is modified after the x-bce-copy-source-if-unmodified-since, perform the copy. Otherwise, the copy fails. No

          The corresponding code example is as follows:

          // Initialize BosClient
          Client client = ...;
          
          // Create a CopyObjectRequest object.
          CopyObjectRequest copyObjectRequest(destBucketName, destKey, srcBucketName, srcKey);
          CopyObjectResponse copyObjectResponse;
          
          // Set a new metadata.
          StringMap& userMetadata = *(meta.mutable_user_meta());//StringMap == map<string, string>
          userMetadata.clear();
          userMetadata["<user-meta-key>"] = "<user-meta-value>";
          
          copyObjectRequest.set_meta(&meta, false);// If the second parameter (is_own) is true, delete the meta when it is destructed by copyObjectRequest.
          //copy-source-if-match
          copyObjectRequest.set_if_match("111111111183bf192b57a4afc76fa632");
          //copy-source-if-none-match
          copyObjectRequest.set_if_none_match("111111111183bf192b57a4afc76fa632");
                  
          std::string gmtDate = TimeUtil::now_gmttime();//The current time in the GMT format 
          
          //copy-source-if-modified-since
          copyObjectRequest.set_if_modified_since(gmtDate);
          
          //copy-source-if-unmodified-since
          copyObjectRequest.set_if_unmodified_since(gmtDate);
          
          // Copy the Object
          client.copy_object(copyObjectRequest, copyObjectResponse;);
          
          std::cout << "ETag: " << copyObjectResponse.etag() << " LastModified: " <<      copyObjectResponse.last_modified() << std::endl;

          Set Storage Type During Object Upload

          BOS supports standard storage, infrequent access, and cold storage. The objects are uploaded and stored as the infrequent access type by specifying StorageClass. The corresponding parameters of three storage types are as follows:

          Storage Type Parameters
          Standard Storage STANDRAD
          Infrequent access STANDARD_IA
          Cold storage COLD
          Archive storage ARCHIVE

          Taking the inrequent access as an example, the codes are as follows:

          void print_common_response(BceResponse &result) {
              printf("status:%d\n", result.status_code());
              if (result.is_ok()) {
                  printf("request-id:%s\n", result.request_id().c_str());
                  printf("debug-id:%s\n", result.debug_id().c_str());
              }
              if (result.is_fail()) {
                  printf("error-message:%s\n", result.error().message().c_str());
              }
          }
          int putObjectStorageClass(){
              std::string filename = "file.txt";  
              FileInputStream file(filename); 
              PutObjectRequest request(bucket, object, &file); 
              request.mutable_meta()->set_storage_class("STANDARD_IA");
              PutObjectResponse result;
              client.put_object(request, &result);
              print_common_response(result);  
              printf("etag: %s\n", result.etag().c_str());
          }

          Additional upload

          In the simple upload mode described above, the objects created is Normal type, and users can no longer append the object. Therefore, it is inconvenient to use in scenarios with frequent data replication, such as log, video monitoring, and live streaming.

          Therefore, BOS especially supports AppendObject, that is, upload files in the AppendObject way. The object type created through the AppendObject operation is an appendable object, and the data can be appended to this object. The size of AppendObject is limited to 0~5G.

          The code example of the upload files through AppendObject is as follows:

          int AppendObjectDemo(Client& client,const std::string& bucketName, const std::string& objectKey) {
          
              // Get the data stream
              FileInputStream inputStream("/path/to/test.zip");
          
              // Upload objects in the form of the data stream
              AppendObjectRequest appendObjectFromInputStreamRequest(bucketName, objectKey, &inputStream);
              AppendObjectResponse appendObjectFromInputStreamResponse;
              int ret = client.append_object(appendObjectFromInputStreamRequest, &appendObjectFromInputStreamResponse);
             
              // Upload the Object in the form of string
              std::string data = "this is data";
              AppendObjectRequest appendObjecFromStringtRequest(bucketName, objectKey, data);
              AppendObjectResponse appendObjectFromStringResponse; 
              
              ret = client.appendObject(appendObjecFromStringtRequest, &appendObjectFromStringResponse);
          
              // Print ETag
              std::cout << appendObjectFromInputStreamResponse.etag() << std::endl;
              // Print NextAppendOffset
              std::cout << appendObjectFromInputStreamResponse.next_append_offset() << std::endl;
          
          
              // Example of AppendUpload, for which you need to add the location of next AppendObject in the request.
              long long nextAppendOffset = appendObjectFromInputStreamResponse.next_append_offset();
              AppendObjectRequest appendObjectFromStringRequest(bucketName,objectKey,data);
              appendObjectFromStringRequest.set_offset(nextAppendOffset);
              AppendObjectResponse appendObjectFromStringResponse;
              ret = client.append_object(appendObjectFromStringRequest, &appendObjectFromStringResponse);
              return ret;
          }

          Multipart upload

          In addition to uploading files to BOS through the simple upload and AppendUpload, BOS also provides another upload mode, i.e., Multipart Upload. Users can use the Multipart Upload mode in the following application scenarios (but not limited to this). For example,

          • Support the resumable upload.
          • Upload the files larger than 5 GB.
          • The network environment is poor, and it is often disconnected from the BOS service.
          • Upload files in a streaming way.
          • The size of the file uploaded cannot be determined before file upload.

          The following introduces the implementation of Multipart Upload step by step. Assume that there is a file with the local path of /path/to/file.zip . Because the file is relatively large, it is transferred to BOS in multiparts.

          Initialize multipart upload

          Use the initiateMultipartUpload method to initialize a multipart upload event:

          // Start Multipart Upload
          InitMultiUploadRequest initMultiUploadRequest(bucketName, objectKey);
          InitMultiUploadResponse initMultiUploadResponse;
          int ret = client.init_multipart_upload(initMultiUploadRequest, &initMultiUploadResponse);
          //Exception handling
          ...    
          // Print UploadId
          std::cout << "UploadId: " << initMultiUploadResponse.upload_id() << std::endl;

          The return result of initMultiUploadResponse contains UploadId , which is the unique identifier to distinguish the MultipartUpload event. We use it in subsequent operations.

          • Initialization of Infrequent Access Object Uploaded

          Initialize a MultipartUpload event of the infrequent access:

          void putMultiUploadStorageClass(){
              ObjectMetaData meta;
              meta.set_storage_class("STANDARD_IA");
              InitMultiUploadRequest initMultiUploadRequest(bucketName, objectKey);
              InitMultiUploadResponse initMultiUploadResponse;
              initMultiUploadRequest.set_meta(&meta);
              client.init_multipart_upload(initMultiUploadRequest, &initMultiUploadResponse);
          }
          • Initialization of Cold Storage Object Uploading

          Initialize a MultipartUpload event of infrequent access:

          void putMultiUploadStorageClass(){
              ObjectMetaData meta;
              meta.set_storage_class("COLD)");
              InitMultiUploadRequest initMultiUploadRequest(bucketName, objectKey);
              InitMultiUploadResponse initMultiUploadResponse;
              initMultiUploadRequest.set_meta(&meta);
              client.init_multipart_upload(initMultiUploadRequest, &initMultiUploadResponse);
          }

          Multipart upload

          Then, upload the file in multiparts.

          // Set each part to 5 MB.
          long partSize = 1024 * 1024 * 5L;
          
          std::string partFileName = "/path/to/file.zip";
          FileInputStream file(partFileName);
          // Compute the number of multiparts.
          int partCount = static_cast<int>(file.get_size() / partSize);
          if (file.get_size() % partSize != 0){
              partCount++;
          }
          
          // Create a vector to save the ETag and part_number of each part after the multipart upload.
          std::vectorvector<part_t> partEtags;
          
          for(int i = 0; i < partCount; i++){
              // Redirect to the beginning of each part.
              long skipBytes = partSize * i;
          
              // Compute the size of each part.
              long size = partSize < file.get_size() - skipBytes ?
                      partSize : file.get_size() - skipBytes;
              // Set the file read range [skipBytes, skipBytes+size).
              file.seek(skipBytes);
          ????file.set_size(file.get_size()- size);
              
              // Create UploadPartRequest and upload parts.
              UploadPartRequest uploadPartRequest(bucketName, objectKey, file, i + 1, initMultiUploadResponse.upload_id());
             
              UploadPartResponse uploadPartResponse;
              int ret = client.upload_part(uploadPartRequest, &uploadPartResponse);
              // Save the returned PartETag to the List.
              part_t partInfo;
              partInfo.part_number = i+1;
              partInfo.etag = uploadPartResponse.etag();
              partEtags.push_back(partInfo);
          }

          The core of the above codes is to call the upload_part method to upload each part concurrently. However, attention should be paid to the following aspects:

          • The upload_part method requires that the size of all parts other than the last one should be greater than or equal to 5 MB. However, the Upload Part interface cannot immediately check the size of parts uploaded but can only check the size of parts upon completing the Multipart Upload.
          • To ensure that there is no error in the data during network transmission, it is recommended that you use the Content-MD5 value returned for each part by BOS to verify the correctness of part data uploaded after upload_part. When all parts data is combined into an object, it does not contain the MD5 value.
          • The part number ranges from 1 to 10,000. If it is out of this range, BOS returns the error code of InvalidArgument.
          • When you upload a part each time, you need to locate the stream to the corresponding location at the beginning of MultipartUpload this time.
          • After you upload a part every time, the return results of BOS contain an object ETag , which is the combination of Etag and PartNumber of parts uploaded and used in the subsequent steps of MultiPartUpload. Therefore, it needs to be saved. In general, these objects ETag are saved in the vector.

          Complete multipart upload

          As shown in the following code segment below, complete the multipart upload:

          CompleteMultipartUploadRequest completeMultipartUploadRequest(bucketName, objectKey, initMultiUploadResponse.upload_id());
          
          // Add the part information, i.e., part merging order.
          for (part_t partInfo : partEtags) {
              completeMultipartUploadRequest.add_part(partInfo.part_number, partInfo.etag);
          }
              
          // Complete the multipart upload.
          CompleteMultipartUploadResponse completeMultipartUploadResponse;
          int ret = client.complete_multipart_upload(completeMultipartUploadRequest, &completeMultipartUploadResponse);
              
          // Print the ETag of objects.
          std::cout << completeMultipartUploadResponse.etag() << std::endl;

          partETags in the codes above is the part_t list saved in Step 2. After it receives the Part list submitted by users, BOS verifies the validity of each data part one by one. When all data parts are verified, BOS combines these data parts into a complete object.

          Cancelling MultiPartUpload Event

          Users can use the abortMultipartUpload method to cancel the multipart upload.

          AbortMultipartUploadRequest abortMultipartUploadRequest(bucketName, objectKey, uploadId);
          AbortMultipartUploadResponse abortMultipartUploadResponse;
          // Cancel the multipart upload
          int ret = client.abort_multipart_upload(abortMultipartUploadRequest, &abortMultipartUploadResponse);

          Get uncompleted multipart upload event

          Users can use the list_multipart_uploads method to get the uncompleted multipart upload event in the bucket.

          ListMultipartUploadsRequest listMultipartUploadsRequest(bucketName);
          ListMultipartUploadsResponse listMultipartUploadsResponse;
          // Get all multipart upload events in the bucket.
          int ret = client.list_multipart_uploads(listMultipartUploadsRequest, &
          listMultipartUploadsResponse);
          
          if (ret != 0) {
              return ret;
          }    
          
          // Go through all multipart upload events.
          for (const MultipartUploadSummary& multipartUpload : listMultipartUploadsResponse.uploads()) {
              std::cout << "Key: " << multipartUpload.key << 
                  " UploadId: " << multipartUpload.upload_id << std::endl;
          }

          Notes:

          1. If the number of multipart upload events in the bucket is more than 1,000, it only returns 1,000 objects by default, the is_truncated value in the return results is True, and it returns next_marker as the starting point for the next reading.
          2. If you want to return more MultiPartUpload events, you can use the set_marker function to set the batch reading of the markers.

          Get information on all parts uploaded

          Users can use the listParts method to get all uploaded parts in a MultipartUpload event.

          ListPartsRequest listPartsRequest(bucketName, objectKey, uploadId);
              
          // Get the information on all parts uploaded.
          ListPartsResponse listPartsResponse;
          int ret = client.list_parts(listPartsRequest, &listPartsResponse);
          
          if (ret != 0) {
              return ret;
          }    
          
          // Go through all parts.
          for (consr PartSummary& part : listPartsResponse.parts()) {
              std::cout << "PartNumber: " << part.part_number << " ETag: " << part.etag;
          }

          If you want to view the storage class of an object, use the following codes:

          public void listPartsStorageClass(){
              ListPartsRequest listPartsRequest(bucketName, objectKey, uploadId);
              
              // Get the information on all parts uploaded.
              ListPartsResponse listPartsResponse;
              int ret = client.list_parts(listPartsRequest, &listPartsResponse);
          
              if (ret != 0) {
                  return ret;
              }    
              std::string storageClass = listPartsResponse.storage_class();
          }

          Resumable upload

          If the network is unstable or the program crashes when users upload the large files to BOS, the whole upload fails, and the parts uploaded before failure are also invalid. Therefore, users have to perform the upload again. It wastes resources. Furthermore, the upload cannot be completed after multiple retries in case of network instability. Based on the scenarios above, BOS provides the resumable upload capability:

          • When your network environment is normal, it is recommended to partition the object into parts with a size of 5 MB in the three-step upload way. Please see Multipart Upload.
          • When your network environment is very poor, it is recommended to resume the upload in an append_object way, and a small data size 256 KB is appended every time. Please see Append Upload.

          Tips:

          • The resumable upload is the encapsulation and reinforcement of the multipart upload.
          • When the file is large, or the network environment is poor, the multipart upload is recommended.

          Download File

          BOS C++ SDK provides abundant file download interfaces. Thus, users can download files from BOS in the following ways:

          • Simple streaming download
          • Download the files to the local folder
          • Download the files to the string in memory
          • Resumeable download
          • Range download
          • Download progress bar

          Simple streaming download

          Users can output the object to a file stream through the following codes:

          void getObject(Client& client, const std::string& bucketName, const std::string& objectKey) {
              
              // Get the file output stream.
              FileOutputStream outStream("test.txt");
          
              // Method 1: Get it through the request and response.
              //Initialize the request
              GetObjectRequest getObjectRequest(bucketName, objectKey);
              // Use outStream as the output object.
              GetObjectResponse getObjectResponse(&outStream);
              int ret = client.get_object(getObjectRequest, &getObjectResponse);
          
              // Method 2: Write the object to the outStream file stream through the convenient interface.
              ret = client.download_file(bucketName, objectKey, outStream);
              ...
          }

          Download objects to file directly

          Users can download the object to the specified file through the following codes directly:

          // Download objects to the file.
          std::string localFileName = "test.txt";
          ObjectMetadata objectMetadata;
          
          // Set the read range of objects.
          int64_t start = 0;
          int64_t length = 100;
          //Download the Object
          int ret = client.download_file(bucketName, objectKey, localFileName, start, length, &objectMetadata); // Save the object to test.txt and return objectMetadata.
          ...

          When the method above is used to directly download objects to the file, the method returns the object ObjectMetadata.

          Download objects to string in memory

          This method is to directly save objects in the string and not to write them to the disk.

          std::string inMemoryData;
          // Method 1: Get the object through the request and response.
          GetObjectRequest getObjectRequest(bucketName, objectKey);
          //Use inMemoryData as the output object
          GetObjectResponse getObjectResponse(&inMemoryData);
          int ret = client.get_object(getObjectRequest, &getObjectResponse);
          
          if (ret != 0) {
              return ret;
          }
          
          //Method 2, get the object through a convenient interface
          ret = client.get_object(bucketName, objectKey, &inMemoryData);
          // Print out
          std::cout << inMemoryData << std::endl;

          Range download

          To perform more functions, you can specify the download range by using GetObjectRequest to get objects more exactly. If the specified download range is 0-100, it returns 101 bytes of data from the 0th byte to 100th byte, including the 100th byte, i.e., [0, 100].

          GetObjectRequest getObjectRequest(bucketName, objectKey);
          // Initialize the file output stream.
          FileOutputStream outStream("test.txt");
          GetObjectResponse getObjectResponse(&outStream);
          // Set the byte data of the download range [0, 100].
          getObjectRequest.set_range(0, 100);
          int ret = client.get_object(getObjectRequest, &getObjectResponse);
          
          if (ret != 0) {
              return ret;
          }
          return 0;

          Users can set the range of returned objects through the set_range method of getObjectRequest. Users can also use this feature to realize the segmented download and perform the resumable download.

          Other usage method

          Get Storage Type of Objects

          The storage class attribute of objects includes STANDARD (standard storage), STANDARD_IA (infrequent access) andCOLD (cold storage), which can be implemented through the following codes:

          int getObjectStorageClass(){
              HeadObjectRequest request(bucketName, objectKey);
              HeadObjectResponse response;
              int ret = client.head_object(request, &response);
              if (ret != 0) {
                  return ret;
              }
              ObjectMetadata& meta = response.meta();
              std::string storageClass = meta.storage_class();
              std::cout << storageClass << std::endl;
              return 0;
          }

          Get ObjectMetadata Only

          Users can only get ObjectMetadata, but do not get the entity of objects by the head_object method.

          The parameters that can be called in the ObjectMetadata parsing class include:

          Parameters Description
          content_type Object type
          content_length Object size
          content_md5 Object MD5
          etag HTTP protocol entity tag of the object
          storage_class Storage type of the object
          user_meta If the userMetadata custom meta is specified in PutObject, it returns this item.
          expires Cache expiration date during the object download
          content_disposition Set the browser to download files. The optional values are inline, attachment; filename="download.txt".
          cache_control Download the Cache setting of the Object. The common values are private, no-cache, max-age, and must-revalidate
          content_range If a range is available, it returns the data range of the object.

          Change File Storage Class

          As mentioned above, BOS supports the assignment of three storage classes, i.e., STANDARD (standard storage), STANDARD_IA (infrequent access), and COLD (cold storage). Furthermore, BOS C++ SDK also supports users to change the storage class of specific files. The parameters involved are as follows:

          Parameters Description
          x-bce-storage-class Specify the storage class of object. STANDARD_IA represents the infrequent access, and COLD represents the cold storage. If the storage class is not specified, it is the standard storage class by default.

          Examples are as follows:

          // Change the standard storage to the infrequent access
          CopyObjectRequest copyObjectRequest(destBucketName, destKey, srcBucketName, srcKey);
          copyObjectRequest.mutable_meta()->set_storage_class("STANDARD_IA");
          CopyObjectResponse copyObjectResponse;
          int ret = client.copy_object(copyObjectRequest, &copyObjectResponse);
          
          // Change the infrequent access to the cold storage
          CopyObjectRequest copyObjectRequest(destBucketName, destKey, srcBucketName, srcKey);
          copyObjectRequest.mutable_meta()->set_storage_class("COLD");
          CopyObjectResponse copyObjectResponse;
          ret = client.copy_object(copyObjectRequest, &copyObjectResponse);

          Get File Download URL

          Users can get the URL of the specified object through the following codes:

          std::string generatePresignedUrl(Client& client, const std::string& bucketName, const std::string& objectKey, int expirationInSeconds) {
          
             // Specify the bucket name of the object to be got by users, object name, and effective duration of the URL.   
             return  client.generate_url(bucketName, objectKey, expirationInSeconds);
          }

          Descriptions:

          • Before they call this function, users need to manually set the endpoint as the domain name of the affiliated region. At present, Baidu AI Cloud provides multi-region support. See Instruction for Region Selection. At present, it supports three regions, including "North China - Beijing", "South China - Guangzhou", and "East China - Suzhou". Beijing region: http://bj.bcebos.com, Guangzhou region: http://gz.bcebos.com, Suzhou region:http://su.bcebos.com.
          • expirationInSeconds is the effective duration of the specified URL, whose time is counted from the current time. It is an optional parameter. When it is not configured, the system default is 1,800 seconds. If you want to set it as the permanent non-expiration time, you can set theexpirationInSecondsparameter to -1, but cannot set to other negative numbers.
          • If it is expected that the got file is public-read, the corresponding URL link can be quickly spliced through simple rules: http://$region.bcebos.com/$bucket/$object

          Enumerate Files in Storage Space

          BOS SDK supports users to list objects in the following two ways:

          • Simple list
          • Complex list through parameters

          Furthermore, users can simulate the folders while they list the files.

          Simple list

          When users want to list the required files simply and quickly, they can return the ListObjectsResponse object by the listObjects method. The ListObjectsResponse object contains the return result of this listObject request. Users can get the description information of all objects through the getContents method in ListObjectsResponse.

          void listObjects(Client& client, const std::string bucketName) {
              ListObjectsRequest listObjectsRequest(bucketName);
              // Get all object information in the specified bucket.
              ListObjectsResponse listObjectsResponse;
              int ret = client.list_objects(listObjectsRequest, &listObjectsResponse);
          
              if (ret != 0) return;
              if (listObjectResponse.is_fail()) {
                  return;
              }
              // Go through all objects
              for (const ObjectSummary& objectSummary : listObjectsResponse.contents()) {
                  std::cout << "ObjectKey: " << objectSummary.key << std::endl;
              }
          }

          Notes:

          1. If the number of objects in the bucket is more than 1,000, it only returns 1,000 objects by default, the is_truncated value in the return results is True, and it returns next_marker as the starting point for the next reading.
          2. If you want to increase the number of objects returned, you can use the marker parameter for batch reading.

          Complex list through parameters

          In addition to the simple list above, users can also set the parameters of ListObjectsRequest to perform various flexible query features. The parameters of ListObjectsRequest that can be set are as follows:

          Parameters Feature How to Use
          prefix Specify that the object key returned must use Prefix as the prefix. set_prefix(const std::string& prefix)
          delimiter It is a character used to group the object names. All names contain the specified prefix and appear for the first time. The objects between delimiter characters are used as a set of elements: CommonPrefixes set_delimiter(const std::string& delimiter)
          marker It set that the results are returned from the first one in alphabetical order after the marker. set_marker(const std::string& marker)
          max_keys Specify the maximum number of objects returned this time, maximum 1,000, and 1,000 by default. If the specified value is greater than 1,000, 1,000 is taken. set_max_keys(int maxKeys).

          Notes:

          1. If an object is named with the prefix, all keys returned still contain the object named with prefix when it is queried with the prefix only. For more information, please see List All Files in Directory Recursively.
          2. If an object is named with the prefix, all keys returned contain Null, and the name of keys does not contain prefix when it is queried with the combination of prefix and delimiter. For more information, see View Files and Subdirectories in Directory.

          Next, we illustrate the parameter list method with? several cases:

          Specify Maximum Number of Objects Returned

          // Specify that the maximum number of objects returned is 500.
          ListObjectsRequest listObjectsRequest("bucketName");
          listObjectsRequest.set_max_keys(500);
          ListObjectsResponse listObjectsResponse;
          int ret = client.list_objects(listObjectsRequest, &listObjectsResponse);
          if (ret != 0) return;
          ...//Response exception handling
          for(const ObjectSummary& objectSummary :listObjectsResponse.contents()) {
              std::cout << "ObjectKey: " << objectSummary.key << std::endl;
          }

          Return Objects with Specified Prefix.

          // Specify to return the objects with prefix Test.
          ListObjectsRequest listObjectsRequest("bucketName");
          listObjectsRequest.set_prefix("test");
          ListObjectsResponse listObjectsResponse;
          int ret = client.list_objects(listObjectsRequest, &listObjectsResponse);
          if (ret != 0) return;
          ...//Response exceptional handling
          for(const ObjectSummary& objectSummary :listObjectsResponse.contents()) {
              std::cout << "ObjectKey: " << objectSummary.key << std::endl;
          }

          Return from Specified Object

          // Users can define that an object is not included and it returns after this object.
          ListObjectsRequest listObjectsRequest("bucketName");
          listObjectsRequest.set_marker("object");
          ListObjectsResponse listObjectsResponse;
          int ret = client.list_objects(listObjectsRequest, &listObjectsResponse);
          if (ret != 0) return;
          ...//Response exceptional handling
          for(const ObjectSummary& objectSummary :listObjectsResponse.contents()) {
              std::cout << "ObjectKey: " << objectSummary.key << std::endl;
          }

          Get All Objects by Paging

          // Users can set and there are up to 500 records per page.
          ListObjectsRequest listObjectsRequest("bucketName");
          listObjectsRequest.set_max_keys(500);
          ListObjectsResponse listObjectsResponse;
          int ret = client.list_objects(listObjectsRequest, &listObjectsResponse);
          if (ret != 0) return;
          ...//Response exception handling
          
          bool isTruncated = true;
          while (isTruncated) {
              ListObjectsResponse listObjectsResponse;
              client.list_objects(listObjectsRequest, &listObjectsResponse);
              isTruncated = listObjectsResponse.is_truncated();
              if (listObjectsResponse.next_marker() != "") {
                  listObjectsRequest.set_marker(listObjectsResponse.next_marker());
              }
          }

          Get Results of All Specific Objects by Paging

          // Users can set up to 500 records per page and get them after a specific object.
          ListObjectsRequest listObjectsRequest("bucketName");
          listObjectsRequest.set_max_keys(500);
          listObjectsRequest.set_marker("object");
          ListObjectsResponse listObjectsResponse;
          int ret = client.list_objects(listObjectsRequest, &listObjectsResponse);
          if (ret != 0) return;
          
          bool isTruncated = true;
          while (isTruncated) {
              ListObjectsResponse listObjectsResponse;
              client.list_objects(listObjectsRequest, &listObjectsResponse);
              isTruncated = listObjectsResponse.is_truncated();
              if (listObjectsResponse.next_marker() != "") {
                  listObjectsRequest.set_marker(listObjectsResponse.next_marker());
              }
          }

          Get Results of All Objects with Specified Prefix by Paging

          // Users can set the paging to get objects with specified prefix. There are up to 500 records per page.
          ListObjectsRequest listObjectsRequest("bucketName");
          listObjectsRequest.set_max_keys(500);
          listObjectsRequest.set_prefix("object");
          ListObjectsResponse listObjectsResponse;
          int ret = client.list_objects(listObjectsRequest, &listObjectsResponse);
          if (ret != 0) return;
          
          bool isTruncated = true;
          while (isTruncated) {
              ListObjectsResponse listObjectsResponse;
              client.list_objects(listObjectsRequest, &listObjectsResponse);
              isTruncated = listObjectsResponse.is_truncated();
              if (listObjectsResponse.next_marker() != "") {
                  listObjectsRequest.set_marker(listObjectsResponse.next_marker());
              }
          }

          The parameters that can be called in theListObjectsResponse parsing class returned by the list_object method are:

          Parameters Description
          name Bucket name
          prefix Match the object from the beginning of the prefix to the first delimiter and return it as a set of elements.
          marker Starting point of this query
          max_keys Maximum number of requests returned
          is_truncated It indicates whether all query results have been returned. False: All results have been returned this time. True: Not all results have been returned this time.
          contents Container of an object returned
          +key Object Name
          +last_modified Last modification time of this object
          +etag HTTP protocol entity tag of objects
          +storage_class Storage form of objects
          +size Size of the object content (byte)
          +owner_id User ID of the Bucket Owner
          +owner_display_name Bucket Owner name

          Simulate folder feature

          There is no concept of the folder in the storage results of BOS. All elements are stored as objects, but the BOS users often need to manage files as folders when they use the data. Therefore, BOS provides the ability to create simulation folders. In essence, it creates an object with size 0. This object can be uploaded and downloaded, but the console displays the object ending with "/" as a folder.

          Users can simulate the folder function through the combination of delimiter and prefix parameters. The combined effect of delimiter and prefix is shown as follows:

          If you set the prefix as a folder name, the return value can list the files starting with this prefix, that is, all recursive files and subfolders (directories) in this folder. The file name is displayed in the contents. If you set the delimiter as "/", the return value only lists the files and subfolders (directories) in this folder, the sub-file names (directories) in this folder are returned in the CommonPrefixes section, and the recursive files and folders in the subfolders are not displayed.

          The following are several application modes:

          List All Files in Bucket

          When users want to get all files in the bucket, they can see Get All Objects by Paging.

          List All Files in Directory Recursively

          You can get all files in a directory by setting the Prefix parameter:

          // Construct ListObjectsRequest.
          ListObjectsRequest listObjectsRequest("bucketName");
          // List all files in the fun directory recursively.
          listObjectsRequest.set_prefix("fun/");
          ListObjectsResponse listObjectsResponse;
          int ret = client.list_objects(listObjectsRequest, &listObjectsResponse);
          if (ret != 0) return;
          ...//Response exception handling
          std::cout << "ObjectKeys:" << std::endl;
          // Go throught all objects
          for(const ObjectSummary& objectSummary :listObjectsResponse.contents()) {
              std::cout << objectSummary.key << std::endl;
          }

          Output:

          ObjectKeys:
          fun/
          fun/movie/001.avi
          fun/movie/007.avi
          fun/test.jpg

          View Files and Sub-directory under Directory.

          It can list the files and subdirectories in the directory through the combination of Prefix and Delimiter :

          // Construct ListObjectsRequest.
          ListObjectsRequest listObjectsRequest("bucketName");
          
          // "/" is the delimiter of folders.
          listObjectsRequest.set_delimiter("/");
          
          // List all files and folders in the fun directory.
          listObjectsRequest.set_prefix("fun/");
          
          ListObjectsResponse listObjectsResponse;
          int ret = client.list_objects(listObjectsRequest, &listObjectsResponse);
          if (ret != 0) return;   
          // Go through objects
          std::cout << "\nObjectKeys:" << std::endl;
          for(const ObjectSummary& objectSummary :listObjectsResponse.contents()) {
              std::cout << objectSummary.key << std::endl;
          }
              
          // Go through CommonPrefix
          std::cout << "CommonPrefixs:" << std::endl;
          for (const std::string& commonPrefix : listObjectsResponse.common_prefixes()) {
              std::cout << commonPrefix << std::endl;
          }

          Output:

          ObjectKeys:
          fun/
          fun/test.jpg
          
          CommonPrefixs:
          fun/movie/

          In the returned result, the file in the fun directory is given in the list contents of ObjectSummary . The common_prefixes list shows all the subfolders in the fun directory. It can be seen that the files fun/movie/001.avi and fun/movie/007.avi are not listed for they belong to the movie directory in the fun folder.

          Enumerate storage attribute of objects in bucket

          If users want to view the storage class attribute of all objects in the specified bucket after uploading, it can be implemented through the following codes:

          public void listObjectsStorageClass(){
              ListObjectsRequest listObjectsRequest("bucketName");
              ListObjectsResponse listObjectsResponse;
              int ret = client.list_objects(listObjectsRequest, &listObjectsResponse);
              std::vector<ObjectSummary> objectList = listObjectResponse.contents();
              for(int i = 0; i<objectList.size(); i++) {
                  std::cout << objectList[i].storage_class() << std::endl;
              }
          }

          Concurrent Interface

          The concurrent interface is implemented by multiple I/O channels based on the curl, the single thread is still used (caller’s thread), and no additional threads are created, but the throughput of requests is improved through the non-blocking I/O. The basic interface is as follows:

          struct BceRequestContext {
              BceRequest *request;
              BceResponse *response;
              bool is_own;
          };
          int send_request(int n, BceRequestContext ctx[], int max_parallel = 0);

          The request and response members of context are used to place the pointer of the request object. When is_own is true, the request and response pointers are actively released during the destruction of context. n in the send_request method is the number of concurrent requests, and max_parallel is the maximum number of concurrencies. It is the options setting by default. When the number of requests exceeds the maximum number of concurrent requests, the remaining requests are queued until a request is completed. It judges whether all requests are successfully executed through the return value of the method. Furthermore, it can go through the CTX array to judge the results of each response one by one.

          Delete File

          Delete a single file

          The following code is used to delete an object:

          int deleteObject(Client& client,const std::string bucketName, const std::string objectKey) {
          
              // Delete an object
              return client.delete_object(bucketName, objectKey);           // Specify the bucket name of object to delete and the object name.
          }

          Delete Multiple Files

          Delete multiple files by the command deleteMultiObject through the concurrent interface to improve the throughput of requests, whose codes are as follows:

          int deleteMultiObject(Client& client, const& std::string bucketName, const std::vector<std::string>& objects) { 
              //Delete objects in batch
              std::vector<BceRequestContext> ctx(objects.size());
              int i = 0;
              for (const std::string& objectKey : objects) {
                  // Construct the context information.
                  ctx[i].request = new DeleteObjectRequest(bucketName, objectKey);
                  ctx[i].response = new DeleteObjectResponse;
                  ctx[i].is_own = true;// Destruct the request and response automatically.
                  ++i;
              }
              //Concurrent request
              int ret = client.send_request((int)ctx.size(), &ctx.front());
              if (ret != 0) return ret;
              //Handle the response
              for (size_t i = 0; i < ctx.size(); ++i) {
                  DeleteObjectRequest* request = (DeleteObjectRequest*)ctx[i].request;
                  DeleteObjectResponse* response = (DeleteObjectResponse*)ctx[i].response;
                  if (response->is_fail()) {// Some requests fail to delete. Handle this situation.
                      std::cout << "delete objectKey=" << request->object_name() << " failed"
                                      << " reason: " << response->error().message();
                      ret = RET_SERVICE_ERROR;
                  }
              }
              return ret;
          }

          View Whether File Exists

          Users can check whether a file exists through the following operations:

          HeadObjectRequest request(bucketName, objectKey);
          HeadObjectResponse response;
          int ret = client.head_object(request, &response);
          if (response.status_code() == 404) {
              std::cout << "ObjectKey not exist!" << std::endl;
          }    

          Get and Update Object Metadata

          The object metadata is the attribute description of files uploaded to BOS by users, including HTTP standard attribute (HTTP headers) and user meta (user-defined metadata).

          Get file metadata information

          See Get ObjectMetadata.

          Modify file meta information

          BOS modifies the metadata of an object by copying this object. That is, when you copy the object, you set the destination bucket as the source bucket and the destination object as the source object and set the new metadata to modify the metadata by copying itself. If no new metadata is set, an error occurs.

          void setObjectMeta(Client& client, const std::string& bucketName, const std::string& objectKey, ObjectMetaData* newObjectMetadata) {
                  
              CopyObjectRequest request(bucketName, objectKey, bucketName, objectKey);
          
              // Set a new ObjectMetadata.
              request.set_meta(newObjectMetadata);
          
              //Copy the object
              CopyObjectResponse copyObjectResponse;
              int ret = client.copy_object(request, &copyObjectResponse);
              if (ret != 0) {
                  return;
              }
              //Print the result
              std::cout << "ETag: " << copyObjectResponse.etag() << " LastModified: " << copyObjectResponse.last_modified() << std::endl;
          }

          Copy Files

          Copy a file

          Users can copy an object through the copy_object function, whose codes are as follows:

          void copyObject(Client& client, std::string destBucketName, std::string destKey, const std::string& srcBucketName, const std::string& srcKey) {
              
              //Copy an object
              int ret = client.copy_object(srcBucketName, srcKey, destBucketName, destKey, "STANDARD");// The target object is standard storage.
          }

          The CopyObjectResponse object contains the ETag and modification time of a new object.

          Users can also copy objects through CopyObjectRequest , whose codes are shown as follows:

          //Initialize a client
          Client client = ...;
              
          // Create a CopyObjectRequest object.
          CopyObjectRequest copyObjectRequest(destBucketName, destKey, srcBucketName, srcKey);
              
          // Set a metadata.
          ObjectMetaData* userMetadata = copyObjectRequest.mutable_meta();
          userMetadata->set_user_meta("userMetaKey", "userMetaValue");
              
          // Copy an object
          CopyObjectResponse copyObjectResponse;
          int ret = client.copy_object(copyObjectRequest, &copyObjectResponse);
              
          std::cout << "ETag: " + copyObjectResponse.etag() <<
           " LastModified: " << copyObjectResponse.last_modified());

          Synchronous Copy Feature

          At present, the CopyObject interface of BOS is implemented in the synchronization mode. In the synchronization mode, BOS returns “Copied Successfully” only when the copy is completed actually. The synchronous copy can help users judge the copy status more accurately. However, the user-perceived copy time becomes longer, and the copy time is directly proportional to the file size.

          The synchronous copy mode is more in line with industry practice, and it improves compatibility with other platforms. The synchronous copy mode also simplifies the business logic of the BOS server and improves service efficiency.

          Multipart upload copy

          In addition to copying files through CopyObject API, BOS also provides another copy mode, i.e., Multipart Upload Copy. Users can use the Multipart Upload Copy in the following application scenarios (but Only limited to this), for example:

          • Support the resumable copy.
          • Copy the file larger than 5 GB.
          • The network environment is poor, and it is often disconnected from the BOS service.

          The following introduces the three-step copy step by step.

          The three-step copy includes three steps: init, copy multipart, and complete. The operations of init and complete are the same as those of the multipart upload.

          For easy understanding, the following provides the complete codes for the three-step copy:

          //Step 1:  init
          InitMultiUploadRequest initMultiUploadRequest("targetBucketName","targetObjectName");
          InitMultiUploadResponse initMultiUploadResponse;
          client.init_multipart_upload(initMultiUploadRequest, &initMultiUploadResponse);
          //Get the Object size
          HeadObjectRequest request("targetBucketName", "targetObjectName");
          HeadObjectResponse response;
          client.head_object(request, &response);
          if (response.is_fail()) {
              return RET_SERVICE_ERROR;
          }
          long left_size = response.meta().content_length();
          
          //Step 2: Multipart copy
          long skipBytes = 0;
          int partNumber = 1;
          std::vector<part_t> partETags;
          
          while (left_size > 0) {
              long partSize = 1024 * 1024 * 1L;
              if (left_size < partSize) {
                  partSize = left_size;
              }
              CopyPartRequest copyPartRequest;
              copyPartRequest.set_upload_id(initMultiUploadResponse.upload_id());
              copyPartRequest.set_bucket_name("targetBucketName");
              copyPartRequest.set_object_name("targetObjectName");
              copyPartRequest.set_source_bucket_name("sourceBucketName");
              copyPartRequest.set_source_object_name("sourceObjectName");
              copyPartRequest.set_range(skipBytes, skipBytes + partSize - 1);
              copyPartRequest.set_part_number(partNumber);
          
              CopyPartResponse copyPartResponse; 
              client.copy_part(copyPartRequest, &copyPartResponse);
              // Save the returned partNumber and ETag to the vector.
              part_t partInfo;
              partInfo.etag = copyPartResponse.etag(); 
              partInfo.part_number = partNumber;
              partETags.push_back(partI);
              left_size -= partSize;
              skipBytes += partSize;
              partNumber+=1;
          }
          
          // Step 3: Complete.
          CompleteMultipartUploadRequest completeMultipartUploadRequest("targetBucketName", "targetObjectName", initMultiUploadResponse.upload_id());
          
          // Add the part information, i.e., part merging order.
          for (part_t partInfo : partEtags) {
              completeMultipartUploadRequest.add_part(partInfo.part_number, partInfo.etag);
          }
              
          // Complete MultiPartUpload.
          CompleteMultipartUploadResponse completeMultipartUploadResponse;
          int ret = client.complete_multipart_upload(completeMultipartUploadRequest, &completeMultipartUploadResponse);
              
          // Print the ETag of objects.
          std::cout << completeMultipartUploadResponse.etag() << std::endl;

          Notes:

          1. The offset parameter is expressed in bytes and is the starting offset position of the block.
          2. The size parameter is in bytes, which defines the size of each part. Except for the last part, the size of other parts must be greater than 5 MB.

          Use Convenient Interface

          parallel_copy uses the concurrent API to improve the request throughput and implement the multipart upload copy.

          client.parallel_copy(srcBucketName, sourceObjectName, destBucketName, destObjectName);

          Archive storage

          Upload

          std::string filename = "test_archive";
          FileInputStream file(filename);
          PutObjectRequest request(bucket, "test_archive", &file);
          request.mutable_meta()->set_storage_class("ARCHIVE");
          PutObjectResponse result;
          client.put_object(request, &result);

          Restoration

          RestoreObjectRequest request(bucket, ""test_archive"");
          RestoreObjectResponse response;
          client.restore_object(request, &response);
          print_common_response(response);
          Previous
          Bucket Management
          Next
          Exception Handling