Multipart upload of objects
Updated at：2025-11-03
Multipart Upload Scenarios
In addition to using the putObject() method for file uploads to BOS, BOS also supports another upload mode called Multipart Upload. This mode can be used in scenarios such as:
- When resumable uploads are required.
- When uploading files larger than 5GB.
- When the connection to the BOS server is frequently interrupted due to unstable network conditions.
- Enable streaming file uploads.
- The file size cannot be determined before uploading.
Multipart Upload Process
Suppose there is a file with the local path /path/to/file.zip. Since the file is large, multipart upload is used to transmit it to BOS. Basic workflow:
- Start a multipart upload.
- Upload individual parts.
- Finalize the multipart upload.
Initialize Multipart Upload
Use initiateMultipartUpload method to initialize a multipart upload event:
Plain Text
1let bucketName = "test-harmony-bucket"
2let objectName = "test-multi-upload-object"
3let args = new InitiateMultipartUploadArgs();
4 // Set the uploaded object to standard storage class
5args.storageClass = "STANDARD";
6// initialization
7let initMultiUploadResult: InitiateMultipartUploadResult;
8try {
9 // Set the contentType parameter to undefined as a placeholder and use the default value
10 initMultiUploadResult = await bosClient.initiateMultipartUpload(bucketName, objectName, undefined, args);
11 logger.info(`init multi upload success, info: ${JSON.stringify(initMultiUploadResult)}`)
12} catch(bosResponse) {
13 logger.error(`errCode: ${bosResponse.error.code}`)
14 logger.error(`requestId: ${bosResponse.error.requestId}`)
15 logger.error(`errMessage: ${bosResponse.error.message}`)
16 logger.error(`statusCode: ${bosResponse.statusCode}`)
17}Note:
The return result of initiateMultipartUploadcontainsUploadId, which is the unique identifier for distinguishing multipart upload events, and we will use it in subsequent operations.
Upload parts
Plain Text
1let stat = fs.lstatSync("/path/to/file.zip");
2let file = fs.openSync(filePath, fs.OpenMode.READ_ONLY);
3 if (stat.size < MAX_SINGLE_OBJECT_SIZE) { //The maximum file size of BOS is 48.8 TB
4 return Promise.reject(fillBceError(`The file size exceeds ${MAX_SINGLE_OBJECT_SIZE}`));
5}
6 // If the file is smaller than the size of one part, there is no need to use multipart upload
7if (stat.size < MIN_MULTIPART_SIZE || this._clientImpl.clientOptions.multiPartSize < MIN_MULTIPART_SIZE) {
8 return Promise.reject(fillBceError(`multipart size should not be less than ${MIN_MULTIPART_SIZE}`));
9}
10 // Calculate the size of each part
11let partSize = (this._clientImpl.clientOptions.multiPartSize + MULTIPART_ALIGN - 1) / MULTIPART_ALIGN * MULTIPART_ALIGN;
12let partNum = Math.floor((stat.size + partSize - 1) / partSize);
13 // If the number of parts exceeds 10000, the size of each part will be recalculated, and the default part size will no longer be used
14if (partNum > MAX_PART_NUMBER) {
15 partSize = (stat.size + MAX_PART_NUMBER - 1) / MAX_PART_NUMBER;
16 partSize = (partSize + MULTIPART_ALIGN - 1) / MULTIPART_ALIGN * MULTIPART_ALIGN;
17 partNum = Math.floor((stat.size + partSize - 1) / partSize);
18}
19logger.debug(`start to upload super file, total parts: ${partNum}, part size: ${partSize}`);
20let args = new InitiateMultipartUploadArgs();
21if (storageClass && isValidStorageClass(storageClass)) {
22 args.storageClass = storageClass;
23}
24 // Use the uploadId returned by the result of initializing multipart upload
25let uploadId = initMultiUploadResult.uploadId as string;
26// group task by partNum
27let completeArgs = new CompleteMultipartUploadArgs();
28completeArgs.partInfo = new PartInfo();
29completeArgs.partInfo.parts = [];
30let partIndex = 1;
31 // Upload each part in sequence
32while (partIndex <= partNum) {
33 logger.debug(`upload part: ${partIndex}`);
34 let uploadSize = partSize;
35 let offset = (partIndex - 1) * partSize;
36 if (uploadSize > stat.size - offset) {
37 uploadSize = stat.size - offset;
38 }
39 let data = new ArrayBuffer(uploadSize);
40 fs.readSync(file.fd, data, {offset: offset, length: uploadSize});
41 try {
42 let etag = await bosClient.uploadPart(bucketName, objectName, uploadId, partIndex, data);
43 // After each part is uploaded successfully, it is necessary to record the eTag information and partNumber information
44 let uploadInfo = new UploadInfoType();
45 uploadInfo.partNumber = partIndex;
46 uploadInfo.eTag = etag;
47 completeArgs.partInfo.parts.push(uploadInfo);
48 } catch (bosResponse) { // If there is a failure during the upload process
49 logger.error(`upload error, info: ${JSON.stringify(bosResponse.error)}`)
50 await bosClient.abortMultipartUpload(bucketName, objectName, uploadId); //Abort this multipart upload
51 fs.closeSync(file);
52 return Promise.reject(bosResponse);
53 }
54 partIndex++;
55}
56 fs.closeSync(file.fd); // Close the local fileNote:
- The UploadPart method requires the size of each part to either be an integer multiple of 1MB or greater than 5MB. However, the Upload Part interface does not validate the part size immediately; this check happens only during the finalization of the multipart upload.
- To ensure data integrity during network transmission, it is advised to verify each uploaded part's data using the Content-MD5 value provided by BOS after UploadPart. Once all parts are combined into a single object, the MD5 value is no longer included.
- The part number must be within the range of 1 to 10,000. If this limit is exceeded, BOS will return an InvalidArgument error code.
- After uploading a part, BOS returns a PartETag object, which consists of the uploaded block's ETag and its part number (PartNumber). These objects are critical for completing the multipart upload and must be saved. Typically, PartETag objects are stored in a List.
Complete multipart upload
Plain Text
1let completeMultiUploadResult:CompleteMultipartUploadResult;
2try {
3 completeMultiUploadResult = await bosClient.completeMultipartUpload(bucketName, objectName, uploadId, completeArgs);
4} catch(bosResponse) {
5 logger.error(`complete multipart fail, info: ${JSON.stringify(bosResponse.error)}`)
6 await bosClient.abortMultipartUpload(bucketName, objectName, uploadId);
7}Complete example
This part has been encapsulated into the BosClient.uploadSuperFile method, and users can call it directly:
Plain Text
1import { logger, Credential, BosClient, ClientOptions } from "bos"
2import { CompleteMultipartUploadResult } from "bos/src/main/ets/bos/api/DataType"
3
4 let credential = new Credential(AccessKeyID, SecretAccessKey, Token); //Temporary AK/SK and Token returned by STS
5let clientOptions = new ClientOptions();
6 clientOptions.endpoint = "bj.bcebos.com"; //Pass in the domain name of the region where the bucket is located
7 let bosClient = new BosClient(credential, clientOptions); // Create BosClient
8
9let bucketName = "test-harmony-bucket"
10let objectName = "multi_upload_object.txt"
11let path = "/path/to/file.zip"
12
13let result: CompleteMultipartUploadResult;
14
15try {
16 result = await bosClient.uploadSuperFile(bucketName, objectName, path);
17 logger.info(`upload super file success, result : ${JSON.stringify(result)}`);
18} catch (bosResponse) {
19 logger.error(`errCode: ${bosResponse.error.code}`)
20 logger.error(`requestId: ${bosResponse.error.requestId}`)
21 logger.error(`errMessage: ${bosResponse.error.message}`)
22 logger.error(`statusCode: ${bosResponse.statusCode}`)
23}Cancel multipart upload
Users can cancel multipart uploads by using the abortMultipartUpload method.
Plain Text
1try {
2 await bosClient.abortMultipartUpload(bucketName, objectName, uploadId);
3} catch(bosResponse) {
4 logger.error(`errCode: ${bosResponse.error.code}`)
5 logger.error(`requestId: ${bosResponse.error.requestId}`)
6 logger.error(`errMessage: ${bosResponse.error.message}`)
7 logger.error(`statusCode: ${bosResponse.statusCode}`)
8}Retrieve unfinished multipart uploads
Users can obtain the unfinished multipart upload events in the bucket by the listMultipartUploads method.
Example code
Plain Text
1try {
2let listMultipartUploadsResult = await bosClient.listMultipartUploads(bucketName);
3 for (let multiUpload of listMultipartUploadsResult.uploads as ListMultipartUploadsType[]) {
4 logger.info(`key: ${multiUpload.key}, uploadId: ${multiUpload.uploadId}`);
5 }
6} catch(bosResponse) {
7 logger.error(`errCode: ${bosResponse.error.code}`)
8 logger.error(`requestId: ${bosResponse.error.requestId}`)
9 logger.error(`errMessage: ${bosResponse.error.message}`)
10 logger.error(`statusCode: ${bosResponse.statusCode}`)
11}Note:
- By default, if the number of multipart upload events in a bucket surpasses 1,000, only 1,000 records will be returned. In such cases, the IsTruncated value in the response will be True, and the NextKeyMarker will indicate the starting point for the next query.
- To retrieve additional multipart upload events, you can use the keyMarker parameter in the ListMultipartUploadsResult instance for batch processing.
Get all uploaded part information
Users can obtain all uploaded parts in an upload event by the listParts method.
Example code
Plain Text
1try {
2 let listPartsResult = await bosClient.listParts(bucketName, objectName, "e8583fb592ec699d7ecca085fc46fdd8");
3 for (let part of listPartsResult.parts as ListPartType[]) {
4 logger.info(`partNumber: ${part.partNumber}, eTag: ${part.eTag}`);
5 }
6} catch(bosResponse) {
7 logger.error(`errCode: ${bosResponse.error.code}`)
8 logger.error(`requestId: ${bosResponse.error.requestId}`)
9 logger.error(`errMessage: ${bosResponse.error.message}`)
10 logger.error(`statusCode: ${bosResponse.statusCode}`)
11}Note:
- By default, if the number of multipart upload events in a bucket surpasses 1,000, only 1,000 records will be returned. In such cases, the IsTruncated value in the response will be True, and the NextPartNumberMarker will indicate the starting point for the next query.
- To access more information about uploaded chunks, you can use the partNumberMarker parameter in the ListPartsResult for batch processing.