Multipart upload of objects
In addition to using the putObject() method for file uploads to BOS, BOS also supports another upload mode called Multipart Upload. This mode can be used in scenarios such as:
- When resumable uploads are required.
- When uploading files larger than 5GB.
- When the connection to the BOS server is frequently interrupted due to unstable network conditions.
- Enable streaming file uploads.
- The file size cannot be determined before uploading.
Complete Multipart Upload in parts
Suppose there is a file with the local path /path/to/file.zip. Since the file is large, multipart upload is used to transmit it to BOS. Basic workflow:
- Start a multipart upload.
- Upload individual parts.
- Finalize the multipart upload.
Initialize Multipart Upload
Use initiateMultipartUpload method to initialize a multipart upload event:
Example code:
1BOSInitiateMultipartUploadRequest* initMPRequest = [[BOSInitiateMultipartUploadRequest alloc] init];
2initMPRequest.bucket = @"<bucketname>";
3initMPRequest.key = @"<objectname>";
4initMPRequest.contentType = @"<content type>";
5__block BOSInitiateMultipartUploadResponse* initMPResponse = nil;
6BCETask* task = [client initiateMultipartUpload:initMPRequest];
7task.then(^(BCEOutput* output) {
8 if (output.response) {
9 initMPResponse = (BOSInitiateMultipartUploadResponse*)output.response;
10 NSLog(@"initiate multipart upload success!");
11 }
12 if (output.error) {
13 NSLog(@"initiate multipart upload failure");
14 }
15});
16[task waitUtilFinished];
17NSString* uploadID = initMPResponse.uploadId;
Note:
The return result of initiateMultipartUploadcontainsUploadId, which is the unique identifier for distinguishing multipart upload events, and we will use it in subsequent operations.
Upload parts
Upload the file in segments.
Example code
1// Calculate the count of parts
2NSString* file = @"/path/to/file.zip";
3NSDictionary<NSString*, id>* attr = [[NSFileManager defaultManager] attributesOfItemAtPath:file error:nil];
4uint64_t fileSize = attr.fileSize;
5uint64_t partSize = 1024 * 1024 * 5L;
6uint64_t partCount = fileSize / partSize;
7if (fileSize % partSize != 0) {
8 ++partCount;
9}
10NSMutableArray<BOSPart*>* parts = [NSMutableArray array];
11NSFileHandle* handle = [NSFileHandle fileHandleForReadingAtPath:@"/path/to/file.zip"];
12for (uint64_t i = 0; i < partCount; ++i) {
13 // seek
14 uint64_t skip = partSize * i;
15 [handle seekToFileOffset:skip];
16 uint64_t size = (partSize < fileSize - skip) ? partSize : fileSize - skip;
17 // data
18 NSData* data = [handle readDataOfLength:size];
19 // request
20 BOSUploadPartRequest* uploadPartRequest = [[BOSUploadPartRequest alloc] init];
21 uploadPartRequest.bucket = @"<bucketname>";
22 uploadPartRequest.key = @"<objectname>";
23 uploadPartRequest.objectData.data = data;
24 uploadPartRequest.partNumber = i + 1;
25 uploadPartRequest.uploadId = uploadID;
26 __block BOSUploadPartResponse* uploadPartResponse = nil;
27 task = [client uploadPart:uploadPartRequest];
28 task.then(^(BCEOutput* output) {
29 if (output.response) {
30 uploadPartResponse = (BOSUploadPartResponse*)output.response;
31 BOSPart* part = [[BOSPart alloc] init];
32 part.partNumber = i + 1;
33 part.eTag = uploadPartResponse.eTag;
34 [parts addObject:part];
35 }
36 });
37 [task waitUtilFinished];
38}
Note: The core of the above code is to call the
UploadPartmethod to upload each part concurrently, but the following points should be noted:
- The
uploadPartmethod mandates that the size of each part, except for the last one, must be at least 5 MB. However, this size requirement is only validated when the multipart upload is finalized, not during theUpload Partoperation itself.- To ensure no errors during network transmission, it is recommended to use the Content-MD5 value returned by BOS for each part after
uploadPartto verify the correctness of the uploaded part data. When all part data is combined into one Object, it no longer contains the MD5 value.- The part number must be within the range of 1 to 10,000. If this limit is exceeded, BOS will return an InvalidArgument error code.
- For each uploaded part, the stream must be positioned at the beginning of the respective part.
- After each Part upload, the return result of BOS will include a
BOSPartobject, which is a combination of the uploaded block's ETag and block number (PartNumber). It will be used in subsequent steps to complete the multipart upload, so it must be saved. Generally speaking, theseBOSPartobjects will be saved in an array.
Complete multipart upload
Example code
1BOSCompleteMultipartUploadRequest* compMultipartRequest = [[BOSCompleteMultipartUploadRequest alloc] init];
2compMultipartRequest.bucket = @"<bucketname>";
3compMultipartRequest.key = @"<objectname>";
4compMultipartRequest.uploadId = uploadID;
5compMultipartRequest.parts = parts;
6__block BOSCompleteMultipartUploadResponse* complResponse = nil;
7task = [client completeMultipartUpload:compMultipartRequest];
8task.then(^(BCEOutput* output) {
9 if (output.response) {
10 complResponse = (BOSCompleteMultipartUploadResponse*)output.response;
11 NSLog(@"complte multiparts success!");
12 }
13 if (output.error) {
14 NSLog(@"complte multiparts failure %@", output.error);
15 }
16});
17[task waitUtilFinished];
Note: The
partsin the above code is the list of parts saved in the second step. After BOS receives the list of parts submitted by the user, it will verify the validity of each data Part one by one. Once all data parts are validated, BOS will assemble the data parts into a complete Object.
Complete example
1#import <BaiduBCEBasic/BaiduBCEBasic.h>
2#import <BaiduBCEBOS/BaiduBCEBOS.h>
3void example(void) {
4// Initialize
5BCECredentials* credentials = [[BCECredentials alloc] init];
6credentials.accessKey = @"<access key>";
7credentials.secretKey = @"<secret key>";
8BOSClientConfiguration* configuration = [[BOSClientConfiguration alloc] init];
9configuration.credentials = credentials;
10BOSClient* client = [[BOSClient alloc] initWithConfiguration:configuration];
11// Initialize multipart upload
12BOSInitiateMultipartUploadRequest* initMPRequest = [[BOSInitiateMultipartUploadRequest alloc] init];
13initMPRequest.bucket = @"<bucketname>";
14initMPRequest.key = @"<objectname>";
15initMPRequest.contentType = @"<content type>";
16 __block BOSInitiateMultipartUploadResponse* initMPResponse = nil;
17 BCETask* task = [client initiateMultipartUpload:initMPRequest];
18 task.then(^(BCEOutput* output) {
19 if (output.response) {
20 initMPResponse = (BOSInitiateMultipartUploadResponse*)output.response;
21 NSLog(@"initiate multipart upload success!");
22 }
23 if (output.error) {
24 NSLog(@"initiate multipart upload failure");
25 }
26 });
27 [task waitUtilFinished];
28 NSString* uploadID = initMPResponse.uploadId;
29// Calculate the count of parts
30 NSString* file = @"/path/to/file.zip";
31 NSDictionary<NSString*, id>* attr = [[NSFileManager defaultManager] attributesOfItemAtPath:file error:nil];
32 uint64_t fileSize = attr.fileSize;
33 uint64_t partSize = 1024 * 1024 * 5L;
34 uint64_t partCount = fileSize / partSize;
35 if (fileSize % partSize != 0) {
36 ++partCount;
37 }
38 NSMutableArray<BOSPart*>* parts = [NSMutableArray array];
39 NSFileHandle* handle = [NSFileHandle fileHandleForReadingAtPath:@"/path/to/file.zip"];
40 for (uint64_t i = 0; i < partCount; ++i) {
41 // seek
42 uint64_t skip = partSize * i;
43 [handle seekToFileOffset:skip];
44 uint64_t size = (partSize < fileSize - skip) ? partSize : fileSize - skip;
45 // data
46 NSData* data = [handle readDataOfLength:size];
47 // request
48 BOSUploadPartRequest* uploadPartRequest = [[BOSUploadPartRequest alloc] init];
49 uploadPartRequest.bucket = @"<bucketname>";
50 uploadPartRequest.key = @"<objectname>";
51 uploadPartRequest.objectData.data = data;
52 uploadPartRequest.partNumber = i + 1;
53 uploadPartRequest.uploadId = uploadID;
54 __block BOSUploadPartResponse* uploadPartResponse = nil;
55 task = [client uploadPart:uploadPartRequest];
56 task.then(^(BCEOutput* output) {
57 if (output.response) {
58 uploadPartResponse = (BOSUploadPartResponse*)output.response;
59 BOSPart* part = [[BOSPart alloc] init];
60 part.partNumber = i + 1;
61 part.eTag = uploadPartResponse.eTag;
62 [parts addObject:part];
63 }
64 });
65 [task waitUtilFinished];
66 }
67 BOSCompleteMultipartUploadRequest* compMultipartRequest = [[BOSCompleteMultipartUploadRequest alloc] init];
68 compMultipartRequest.bucket = @"<bucketname>";
69 compMultipartRequest.key = @"<objectname>";
70 compMultipartRequest.uploadId = uploadID;
71 compMultipartRequest.parts = parts;
72 __block BOSCompleteMultipartUploadResponse* complResponse = nil;
73 task = [client completeMultipartUpload:compMultipartRequest];
74 task.then(^(BCEOutput* output) {
75 if (output.response) {
76 complResponse = (BOSCompleteMultipartUploadResponse*)output.response;
77 NSLog(@"complte multiparts success!");
78 }
79 if (output.error) {
80 NSLog(@"complte multiparts failure %@", output.error);
81 }
82 });
83 [task waitUtilFinished];
84}
Cancel multipart upload
Users can cancel multipart uploads by using the abortMultipartUpload method.
Example code:
1BOSAbortMultipartUploadRequest* abortRequest = [[BOSAbortMultipartUploadRequest alloc] init];
2abortRequest.bucket = @"bucket";
3abortRequest.key = @"<objectname>";
4abortRequest.uploadId = uploadID;
5__block BOSAbortMultipartUploadResponse* abortResponse = nil;
6task = [client abortMultipartUpload:abortRequest];
7task.then(^(BCEOutput* output) {
8 if (output.response) {
9 abortResponse = (BOSAbortMultipartUploadResponse*)output.response;
10 NSLog(@"abort multiparts success!");
11 }
12 if (output.error) {
13 NSLog(@"abort multiparts failure %@", output.error);
14 }
15});
16[task waitUtilFinished];
Retrieve unfinished multipart uploads
Users can obtain the unfinished multipart upload events in the bucket by the listMultipartUploads method.
Basic workflow
- Create an instance of the BOSListMultipartUploadsRequest class, and pass in the parameter
<BucketName>. - Instantiate the BOSClient class and call its listMultipartUploads method.
- The listMultipartUploads method provides details about all ongoing multipart uploads.
Example code
1BOSListMultipartUploadsRequest* listMultipartRequest = [[BOSListMultipartUploadsRequest alloc] init];
2listMultipartRequest.bucket = @"<bucketname>";
3__block BOSListMultipartUploadsResponse* listMultipartResponse = nil;
4task = [client listMultipartUploads:listMultipartRequest];
5task.then(^(BCEOutput* output) {
6 if (output.response) {
7 listMultipartResponse = (BOSListMultipartUploadsResponse*)output.response;
8 NSLog(@"list multipart success");
9 }
10 if (output.error) {
11 NSLog(@"list multipart failure %@", output.error);
12 }
13});
14[task waitUtilFinished];
Note:
- By default, if there are more than 1,000 multipart upload events in a bucket, only 1,000 entries will be returned. The IsTruncated field in the response will be True, and the nextKeyMarker will indicate the starting point for subsequent data.
- To fetch additional multipart upload events, use the keyMarker parameter to read data in batches.
Complete example
1#import <BaiduBCEBasic/BaiduBCEBasic.h>
2#import <BaiduBCEBOS/BaiduBCEBOS.h>
3void example(void) {
4 // Initialize
5 BCECredentials* credentials = [[BCECredentials alloc] init];
6 credentials.accessKey = @"<access key>";
7 credentials.secretKey = @"<secret key>";
8 BOSClientConfiguration* configuration = [[BOSClientConfiguration alloc] init];
9 configuration.credentials = credentials;
10 BOSClient* client = [[BOSClient alloc] initWithConfiguration:configuration];
11 BOSListMultipartUploadsRequest* listMultipartRequest = [[BOSListMultipartUploadsRequest alloc] init];
12 listMultipartRequest.bucket = @"<bucketname>";
13 __block BOSListMultipartUploadsResponse* listMultipartResponse = nil;
14 BCETask* task = [client listMultipartUploads:listMultipartRequest];
15 task.then(^(BCEOutput* output) {
16 if (output.response) {
17 listMultipartResponse = (BOSListMultipartUploadsResponse*)output.response;
18 NSLog(@"list multipart success");
19 }
20 if (output.error) {
21 NSLog(@"list multipart failure %@", output.error);
22 }
23 });
24 [task waitUtilFinished];
25 for (BOSMultipartUpload* upload in listMultipartResponse.uploads) {
26 NSLog(@"upload id : %@", upload.uploadId);
27 }
28}
Get all uploaded part information
Users can obtain all uploaded parts in an upload event by the listParts method.
Basic workflow
- Create an instance of the BOSListPartsRequest class, and pass in the parameters
<BucketName>,<ObjectKey>and<UploadId> - Instantiate the BOSClient class and call its listParts method.
- The listParts method provides information about all the parts that have been uploaded.
Example code
1BOSListPartsRequest* listPartsRequest = [[BOSListPartsRequest alloc] init];
2listPartsRequest.bucket = @"<bucketname>";
3listPartsRequest.key = @"<objectname>";
4listPartsRequest.uploadId = @"<upload id>";;
5__block BOSListPartsResponse* listPartsResponse = nil;
6BCETask* task = [client listParts:listPartsRequest];
7task.then(^(BCEOutput* output) {
8 if (output.response) {
9 listPartsResponse = (BOSListPartsResponse*)output.response;
10 NSLog(@"list parts success!");
11 }
12 if (output.error) {
13 NSLog(@"list part failure %@", output.error);
14 }
15});
16[task waitUtilFinished];
17for (BOSPart* part in listPartsResponse.parts) {
18 NSLog(@"part etag %@", part.eTag);
19}
Note:
- By default, if the number of multipart upload events in a bucket surpasses 1,000, only 1,000 records will be returned. In such cases, the IsTruncated value in the response will be True, and the NextPartNumberMarker will indicate the starting point for the next query.
- To retrieve more information about uploaded parts, use the PartNumberMarker parameter to fetch data in batches.
Complete example
1#import <BaiduBCEBasic/BaiduBCEBasic.h>
2#import <BaiduBCEBOS/BaiduBCEBOS.h>
3void example(void) {
4 // Initialize
5 BCECredentials* credentials = [[BCECredentials alloc] init];
6 credentials.accessKey = @"<access key>";
7 credentials.secretKey = @"<secret key>";
8 BOSClientConfiguration* configuration = [[BOSClientConfiguration alloc] init];
9 configuration.credentials = credentials;
10 BOSClient* client = [[BOSClient alloc] initWithConfiguration:configuration];
11 BOSListPartsRequest* listPartsRequest = [[BOSListPartsRequest alloc] init];
12 listPartsRequest.bucket = @"<bucketname>";
13 listPartsRequest.key = @"<objectname>";
14 listPartsRequest.uploadId = @"<upload id>";;
15 __block BOSListPartsResponse* listPartsResponse = nil;
16 BCETask* task = [client listParts:listPartsRequest];
17 task.then(^(BCEOutput* output) {
18 if (output.response) {
19 listPartsResponse = (BOSListPartsResponse*)output.response;
20 NSLog(@"list parts success!");
21 }
22 if (output.error) {
23 NSLog(@"list part failure %@", output.error);
24 }
25 });
26 [task waitUtilFinished];
27 for (BOSPart* part in listPartsResponse.parts) {
28 NSLog(@"part etag %@", part.eTag);
29 }
30}
