Upload files

Updated at：2025-11-03

Upload files

In BOS, the basic data unit for user operations is an object. An object consists of a key, metadata, and data. The key is the object’s name, the metadata provides a user-defined description as a set of name-value pairs, and the data is the content of the object.

The BOS C++ SDK provides a rich set of file upload APIs, and files can be uploaded in the following ways:

Simple upload
Append upload
Multipart upload
Resumable upload

Simple upload

In simple upload scenarios, BOS supports uploading objects in the form of specified files, data streams, file descriptors, and strings. Please refer to the following code:

                C++
                
            

                int PutObjectDemo(Client& client,const std::string& bucketName, const std::string objectKey){
 // Obtain file data stream
    FileInputStream inputStream("/path/to/test.zip"); // bcesdk/util/util.h
	int ret = 0;
 // Upload object with file name as parameter
    ret = client.upload_file(bucketName, objectKey, "/path/to/test.zip");
 // Upload an object in the form of a data stream
    ret = client.upload_file(bucketName, objectKey, inputStream);
 // Upload object in file descriptor form
 fd_t fd = open("/path/to/test.zip", O_RDWR, 0666);//On Linux, fd_t is defined in common.h
    ret = client.upload_file(bucketName, objectKey, fd);
 // Upload object in string form
    std::string data = "this is data";
	ret = client.put_object(bucketName, objectKey, data);
    return ret;
}
            

Objects are uploaded to BOS as files. The put_object and upload_file functions support uploading objects of up to 5 GB in size. For files larger than 5 GB, please use upload_super_file. The reference implementation is as follows:

                C++
                
                int PutLargeObjectDemo(Client& client,const std::string& bucketName, const std::string objectKey){
    std::string fileName = "/path/to/test.zip"
 return client.upload_super_file(bucketName, objectKey, fileName);//The third parameter can also be in fd_t format
}

Set file meta information

Object metadata refers to the attributes of files provided by users when uploading to BOS. It is mainly divided into two categories: standard HTTP attribute settings (HTTP headers) and user-defined metadata.

Set Http header of object

The BOS C++ SDK interacts with the background HTTP API, allowing users to customize the HTTP headers of objects during uploads. Common HTTP headers are described as follows:

Name	Description	Default value
Content-MD5	File data verification: After setting, BOS will enable file content MD5 verification, compare the MD5 you provide with the MD5 of the file, and throw an error if they are inconsistent	None
Content-Type	File MIME: This defines the file type and web page encoding, determining how the browser reads the file. If unspecified, BOS generates it based on the file's extension. If the file lacks an extension, a default value will be applied.	application/octet-stream
Content-Disposition	Indicate how the MIME user agent displays the attached file, whether to open or download it, and the file name	None
Content-Length	The length of the uploaded file. If it exceeds the length of the stream/file, it will be truncated; if it is insufficient, it will be the actual value	Stream/file duration
Expires	Cache expiration time	None
Cache-Control	Specify the caching behavior of the web page when the object is downloaded	None

Reference code is as follows:

                C++
                
            

                ...
 //Initialize meta
ObjectMetaData meta;
 // Set ContentType
meta.set_content_type("application/json");
 //Set cache-control
meta.set_cache_control("no-cache");
 //Set x-bce-storage-class
meta.set_storage_class("STANDARD");
ret = client.upload_file(bucketName, objectKey, content, meta);
...
            

User-defined meta information

BOS supports user-defined metadata for describing objects. Example usage is shown in the following code:

                C++
                
                // Set the value of custom metadata name to my-data
meta.set_user_meta("name", "my-data");
    
 // Upload Object
ret = client.upload_file(bucketName, objectKey, file_name, meta);

Prompt:

In the above code, the user defines a metadata with the name name and the value my-data

When users download this object, this metadata can also be obtained.

An object may have multiple similar parameters, but the total size of all user meta must not exceed 2KB.

Set the copy attribute of the object

BOS provides a CopyObject API for copying an existing object to a different object. During the copying process, it evaluates the source object’s ETag or modification status to determine whether to proceed. Detailed parameter descriptions are as follows:

Name	Types	Description	Whether required
x-bce-copy-source-if-match	std::string	If the ETag value of the source object matches the ETag value provided by the user, the copy operation is performed; otherwise, it fails.	No
x-bce-copy-source-if-none-match	std::string	If the ETag value of the source object does not match the ETag value provided by the user, the copy operation is performed; otherwise, it fails.	No
x-bce-copy-source-if-unmodified-since	std::string	If the source object has not been modified since x-bce-copy-source-if-unmodified-since, the copy operation will proceed; otherwise, it will fail.	No
x-bce-copy-source-if-modified-since	std::string	If the source object has been modified since x-bce-copy-source-if-modified-since, the copy operation will proceed; otherwise, it will fail.	No

The corresponding example code:

                Cpp
                
            

                // Initialize BosClient
Client client = ...;
 // Create CopyObjectRequest object
CopyObjectRequest copyObjectRequest(destBucketName, destKey, srcBucketName, srcKey);
CopyObjectResponse copyObjectResponse；
 // Set new Metadata
StringMap& userMetadata = *(meta.mutable_user_meta());//StringMap == map<string, string>
userMetadata.clear();
userMetadata["<user-meta-key>"] = "<user-meta-value>";
 copyObjectRequest.set_meta(&meta, false);//If the second parameter (is_own) is true, the meta will be deleted when copyObjectRequest is destructed
//copy-source-if-match
copyObjectRequest.set_if_match("111111111183bf192b57a4afc76fa632");
//copy-source-if-none-match
copyObjectRequest.set_if_none_match("111111111183bf192b57a4afc76fa632");
        
 std::string gmtDate = TimeUtil::now_gmttime();//Current time in GMT format
//copy-source-if-modified-since
copyObjectRequest.set_if_modified_since(gmtDate);
//copy-source-if-unmodified-since
copyObjectRequest.set_if_unmodified_since(gmtDate);
 // Copy Object
client.copy_object(copyObjectRequest， copyObjectResponse；);
std::cout << "ETag: " << copyObjectResponse.etag() << " LastModified: " <<      copyObjectResponse.last_modified() << std::endl;
            

Set storage class when uploading an object

BOS supports standard storage, infrequent access storage, and cold storage. Uploading an object and storing it as an infrequent access storage class is achieved by specifying the StorageClass. The parameters corresponding to the four storage classes are as follows:

Storage class	Parameters
Standard storage	STANDARD
Infrequent access storage	STANDARD_IA
Cold storage	COLD
Archive storage	ARCHIVE

Taking infrequent access storage as an example, the code is as follows:

                Cpp
                
            

                void print_common_response(BceResponse &result) {
    printf("status:%d\n", result.status_code());
    if (result.is_ok()) {
        printf("request-id:%s\n", result.request_id().c_str());
        printf("debug-id:%s\n", result.debug_id().c_str());
    }
    if (result.is_fail()) {
        printf("error-message:%s\n", result.error().message().c_str());
    }
}
int putObjectStorageClass(){
    std::string filename = "file.txt";  
    FileInputStream file(filename);
    PutObjectRequest request(bucket, object, &file);
    request.mutable_meta()->set_storage_class("STANDARD_IA");
    PutObjectResponse result;
    client.put_object(request, &result);
    print_common_response(result);  
    printf("etag: %s\n", result.etag().c_str());
}
            

Using upload progress bar

                C++
                
            

                // Upload progress callback function
 //Note: There must be no time-consuming/blocking operations in this callback function, as it will affect data upload performance.
 //increment: The amount of data uploaded in this time
 //transfered: The total amount of data uploaded
 //total: The amount of data to be uploaded
 //userData: User-defined data, such as object bucket + key, etc.
void progress_callback(int64_t increment, int64_t transfered, int64_t total, void* user_data) {
    std::cout << "progress_callback[" << user_data << "] => " <<
    increment <<" ," << transfered << "," << total << std::endl;
}
 //File to be uploaded
std::string filename = "/tmp/put_file_test";  
FileInputStream file(filename);
PutObjectRequest req(BUCKET, "transfer_progress_t1", &file);
PutObjectResponse rsp;
   
 //Set data related to upload progress
 //The TransferProgress structure is in the header file: "bcesdk/common/common.h"
TransferProgress progress;
progress.transfer_progress_cb = progress_callback;
req.set_progress(progress);
 //Upload data from the filename file
int ret = client()->put_object(req, &rsp);
if (ret) {
    LOGF(WARN, "client err: %d", ret);
}
if (rsp.is_fail()) {
    LOGF(WARN,
        "put_object: [status_code = %d], [message = %s], [requestid = %s]",
         rsp.status_code(),
         rsp.error().message().c_str(),
         rsp.error().request_id().c_str());
}
            

Append upload

Objects created using the simple upload method described above are all of a standard type and do not support append writes. This limitation can be inconvenient in scenarios where frequent data overwriting occurs, such as log files, video surveillance, and live video streaming.

To address this, Baidu AI Cloud Object Storage (BOS) specifically supports the AppendObject method, which allows files to be uploaded in an append-write fashion. Objects created through the AppendObject operation are categorized as Appendable Objects, enabling data to be appended to them. The size limit for AppendObject files is 0–5 GB.

Example code for uploading via AppendObject is as follows:

                Cpp
                
            

                int AppendObjectDemo(Client& client,const std::string& bucketName, const std::string& objectKey) {
 // Obtain data stream
    FileInputStream inputStream("/path/to/test.zip");
 // Upload an object in the form of a data stream
    AppendObjectRequest appendObjectFromInputStreamRequest(bucketName, objectKey, &inputStream);
    AppendObjectResponse appendObjectFromInputStreamResponse;
    int ret = client.append_object(appendObjectFromInputStreamRequest, &appendObjectFromInputStreamResponse);
   
 // Upload object in string form
    std::string data = "this is data";
    AppendObjectRequest appendObjecFromStringtRequest(bucketName, objectKey, data);
    AppendObjectResponse appendObjectFromStringResponse;
    
    ret = client.append_object(appendObjecFromStringtRequest, &appendObjectFromStringResponse);
 // Print ETag
    std::cout << appendObjectFromInputStreamResponse.etag() << std::endl;
 //Print NextAppendOffset
    std::cout << appendObjectFromInputStreamResponse.next_append_offset() << std::endl;
 //Example of append upload, need to add the position of the next append write in the request
    long long nextAppendOffset = appendObjectFromInputStreamResponse.next_append_offset();
    AppendObjectRequest appendObjectFromStringRequest(bucketName,objectKey,data);
    appendObjectFromStringRequest.set_offset(nextAppendOffset);
    AppendObjectResponse appendObjectFromStringResponse;
    ret = client.append_object(appendObjectFromStringRequest, &appendObjectFromStringResponse);
    return ret;
}
            

Multipart upload

In addition to simple and append uploads, BOS also offers another upload method: Multipart Upload. Users can utilize the Multipart Upload mode in various scenarios, including (but not limited to) the following:

When resumable uploads are required.
When uploading files larger than 5GB.
When the connection to the BOS server is frequently interrupted due to unstable network conditions.
Enable streaming file uploads.
The file size cannot be determined before uploading.

The following will introduce the implementation of Multipart Upload step by step. Suppose there is a file with the local path /path/to/file.zip. Since the file is large, it will be transmitted to BOS in parts.

Initialize multipart upload

Use initiateMultipartUpload method to initialize a multipart upload event:

                Cpp
                
            

                // Initiate Multipart Upload
InitMultiUploadRequest initMultiUploadRequest(bucketName, objectKey);
InitMultiUploadResponse initMultiUploadResponse;
int ret = client.init_multipart_upload(initMultiUploadRequest, &initMultiUploadResponse);
 //Exception handling
...    
 // Print UploadId
std::cout << "UploadId: " << initMultiUploadResponse.upload_id() << std::endl;
            

The return result of initMultiUploadResponse contains UploadId, which is the unique identifier for distinguishing multipart upload events, and we will use it in subsequent operations.

Initialization for uploading infrequent access storage class objects

Initialize a multipart upload event for infrequent access storage:

                Cpp
                
            

                void putMultiUploadStorageClass(){
    ObjectMetaData meta;
    meta.set_storage_class("STANDARD_IA");
    InitMultiUploadRequest initMultiUploadRequest(bucketName, objectKey);
    InitMultiUploadResponse initMultiUploadResponse;
    initMultiUploadRequest.set_meta(&meta);
    client.init_multipart_upload(initMultiUploadRequest, &initMultiUploadResponse);
}
            

Initialization for uploading a cold storage class object

Initialize a multipart upload event for infrequent access storage:

                Cpp
                
            

                void putMultiUploadStorageClass(){
    ObjectMetaData meta;
    meta.set_storage_class("COLD)");
    InitMultiUploadRequest initMultiUploadRequest(bucketName, objectKey);
    InitMultiUploadResponse initMultiUploadResponse;
    initMultiUploadRequest.set_meta(&meta);
    client.init_multipart_upload(initMultiUploadRequest, &initMultiUploadResponse);
}
            

Upload parts

The file is then uploaded in multiple parts.

                C++
                
            

                // Set each part to 5MB
 //[Note] Except for the last part, the size of other parts must be >= 100 KB
long partSize = 1024 * 1024 * 5L;
 //Note: When the data to be uploaded in parts is string/in-memory data, the constructor of UploadPartRequest is as follows:
// UploadPartRequest(const std::string &bucket_name, const std::string &object_name,  const std::string &data, int part_number, const std::string &upload_id)     
 //The data field is std::string; do not pass a C-style char* string, as it will cause an error in calculating the data size.
 //Files to be uploaded in parts
std::string partFileName = "/path/to/file.zip";
FileInputStream file(partFileName);
 // Calculate the count of parts
int partCount = static_cast<int>(file.get_size() / partSize);
if (file.get_size() % partSize != 0){
    partCount++;
}
int64_t size = file.size();
int64_t off = 0;
std::vector<part_t> partEtags;
for (int i = 0; off < file.size(); ++i) {
    if (off + partSize > size) {
        partSize = size - off;
    }
    FileInputStream partFile(file.fd(), off, partSize);
    UploadPartRequest uploadPartRequest(bucketName, objectName, partFile, i + 1, initMultiUploadResponse.upload_id());
    UploadPartResponse uploadPartResponse;
    int ret = client.upload_part(uploadPartRequest, &uploadPartResponse);
 //Verify the return value
 // Save the returned PartETag to the List.
    part_t partInfo;
    partInfo.part_number = i+1;
    partInfo.etag = uploadPartResponse.etag();
    partEtags.push_back(partInfo);
    
    off += partSize;
}
            

The core of the above code is to call the upload_part method to upload each part concurrently, but the following points should be noted:

upload_part requires that the size of each part, except the last one, must be greater than or equal to 100 KB. However, the Upload Part API does not immediately verify the size of the uploaded part; the verification is only performed when Complete Multipart Upload is called. If the block size in the upload_part process does not meet the expectation, the complete_multipart_upload API will report an error.
To ensure no errors during network transmission, it is recommended to use the Content-MD5 value returned by BOS for each part after upload_part to verify the correctness of the uploaded part data. When all part data is combined into one Object, it no longer contains the MD5 value.
The part number must be within the range of 1 to 10,000. If this limit is exceeded, BOS will return an InvalidArgument error code.
For each uploaded part, the stream must be positioned at the beginning of the respective part.
After each Part upload, the return result of BOS will include a ETag object, which is a combination of the uploaded block's ETag and block number (PartNumber). It will be used in subsequent steps to complete the multipart upload, so it must be saved. Generally speaking, these ETag objects will be saved in a vector.

Complete multipart upload

Complete the multipart upload as shown in the following code:

                Cpp
                
            

                CompleteMultipartUploadRequest completeMultipartUploadRequest(bucketName, objectKey, initMultiUploadResponse.upload_id());
 //Add part information, i.e., the order of part merging
for (part_t partInfo : partEtags) {
    completeMultipartUploadRequest.add_part(partInfo.part_number, partInfo.etag);
}
    
 // Complete multipart upload
CompleteMultipartUploadResponse completeMultipartUploadResponse;
int ret = client.complete_multipart_upload(completeMultipartUploadRequest, &completeMultipartUploadResponse);
    
 // Print Object's ETag
std::cout << completeMultipartUploadResponse.etag() << std::endl;
            

The partETags in the above code is the list of part_t saved in the second step. After BOS receives the part list submitted by the user, it will verify the validity of each data Part one by one. Once all data parts are validated, BOS will assemble the data parts into a complete Object.

Cancel multipart upload event

Users can cancel multipart uploads by using the abortMultipartUpload method.

                Cpp
                
                AbortMultipartUploadRequest abortMultipartUploadRequest(bucketName, objectKey, uploadId);
AbortMultipartUploadResponse abortMultipartUploadResponse;
 // Cancel multipart upload
int ret = client.abort_multipart_upload(abortMultipartUploadRequest, &abortMultipartUploadResponse);

Retrieve unfinished multipart upload event

Users can obtain the unfinished multipart upload events in the bucket by the list_multipart_uploads method.

                Cpp
                
            

                ListMultipartUploadsRequest listMultipartUploadsRequest(bucketName);
ListMultipartUploadsResponse listMultipartUploadsResponse;
 // Retrieve all upload events within the bucket
int ret = client.list_multipart_uploads(listMultipartUploadsRequest, &
listMultipartUploadsResponse);
if (ret != 0) {
    return ret;
}    
 // Traverse all upload events
for (const MultipartUploadSummary& multipartUpload : listMultipartUploadsResponse.uploads()) {
    std::cout << "Key: " << multipartUpload.key <<
        " UploadId: " << multipartUpload.upload_id << std::endl;
}
            

Note:

By default, if the number of multipart upload events in a bucket exceeds 1,000, only 1,000 objects will be returned, and the is_truncated value in the return result will be True, and next_marker will be returned as the starting point for the next reading.

To return more multipart upload events, you can use the set_marker function to set marker for batch reading.

Get all uploaded part information

Users can obtain all uploaded parts in an upload event by the listParts method.

                Cpp
                
            

                ListPartsRequest listPartsRequest(bucketName, objectKey, uploadId);
    
 // Retrieve all uploaded part information
ListPartsResponse listPartsResponse;
int ret = client.list_parts(listPartsRequest, &listPartsResponse);
if (ret != 0) {
    return ret;
}    
 // Traverse all parts
for (consr PartSummary& part : listPartsResponse.parts()) {
    std::cout << "PartNumber: " << part.part_number << " ETag: " << part.etag;
}
            

If you need to view the storage class of an object, use the following code:

                Cpp
                
            

                public void listPartsStorageClass(){
    ListPartsRequest listPartsRequest(bucketName, objectKey, uploadId);
    
 // Retrieve all uploaded part information
    ListPartsResponse listPartsResponse;
    int ret = client.list_parts(listPartsRequest, &listPartsResponse);
    if (ret != 0) {
        return ret;
    }    
    std::string storageClass = listPartsResponse.storage_class();
}
            

Resumable upload

When users upload large files to BOS, if the network is unstable or the program crashes, the entire upload will fail, and the parts that have been uploaded before the failure will also be invalid. Users have to start over. This not only wastes resources but also often fails to complete the upload after multiple retries in an unstable network environment. Based on the above scenarios, BOS provides the capability of resumable upload:

In a generally stable network, it is recommended to use the three-step upload method, dividing the object into 5 MB blocks, refer to [Multipart Upload](#Multipart upload).
If your network condition is very poor, it is recommended to use the append_object method for resumable upload, appending small data (256 KB) each time, refer to [Append Upload](#Append upload).

Tips

Resumable upload is an encapsulation and enhancement of multipart upload, implemented using multipart upload;

For large files or poor network environments, it is recommended to use multipart upload;

Download file

Archive storage