File management
Upload files
In BOS, the basic data unit for user operations is an object. An object consists of a key, metadata, and data. The key is the object’s name, the metadata provides a user-defined description as a set of name-value pairs, and the data is the content of the object.
The BOS GO SDK provides a rich set of file upload APIs, and files can be uploaded in the following ways:
- Simple upload
- Append upload
- Fetch and upload
- Multipart upload
- Automatic three-step upload
Simple upload
In simple upload scenarios, BOS supports uploading objects in the form of specified files, data streams, binary strings, and strings. Please refer to the following code:
1// import "github.com/baidubce/bce-sdk-go/bce"
2 // Upload from local file
3etag, err := bosClient.PutObjectFromFile(bucketName, objectName, fileName, nil)
4 // Upload from string
5str := "test put object"
6etag, err := bosClient.PutObjectFromString(bucketName, objectName, str, nil)
7 // Upload from byte array
8byteArr := []byte("test put object")
9etag, err := bosClient.PutObjectFromBytes(bucketName, objectName, byteArr, nil)
10 // Upload from data stream
11bodyStream, err := bce.NewBodyFromFile(fileName)
12etag, err := bosClient.PutObject(bucketName, objectName, bodyStream, nil)
13 // Use the basic API and provide necessary parameters to upload from the data stream
14bodyStream, err := bce.NewBodyFromFile(fileName)
15etag, err := bosClient.BasicPutObject(bucketName, objectName, bodyStream)Files are uploaded to BOS as objects. The simple upload API supports uploading files no larger than 5 GB. Once the upload is successfully processed, BOS returns the ETag of the object in the Header as the file's identifier.
Set file meta information
Object metadata refers to the attributes of files provided by users when uploading to BOS. It is mainly divided into two categories: standard HTTP attribute settings (HTTP headers) and user-defined metadata.
Set Http header of object
The BOS GO SDK essentially calls the backend HTTP API, allowing users to customize the HTTP headers of objects when uploading files. The descriptions of commonly used HTTP headers are as follows:
| Name | Description | Default value |
|---|---|---|
| Content-MD5 | File data verification: After setting, BOS will enable file content MD5 verification, compare the MD5 you provide with the MD5 of the file, and throw an error if they are inconsistent | There are |
| Content-Type | File MIME Type: Specifies the file type and webpage encoding, determining the form and encoding used by browsers to read the file. If not explicitly set, BOS automatically generates it based on the file's extension. If the file has no extension, a default value is used. | application/octet-stream |
| Content-Disposition | Indicate how the MIME user agent displays the attached file, whether to open or download it, and the file name | None |
| Content-Length | The length of the uploaded file. If it exceeds the length of the stream/file, it will be truncated; if it is insufficient, it will be the actual value | Length of the stream/file |
| Expires | Cache expiration time | None |
| Cache-Control | Specify the caching behavior of the web page when the object is downloaded | None |
| x-bce-content-crc32c | crc32c of the uploaded data: The BOS server calculates the crc32c value and compares it to the client-calculated value. If they mismatch, the upload fails. The value is saved in metadata, and the server response returns the crc32c value. | Empty string |
| x-bce-content-crc32c-flag | When set to true, the BOS server calculates crc32c, saves it in metadata, and the server response returns the crc32c value. | false |
Reference code is as follows:
1// import "github.com/baidubce/bce-sdk-go/services/bos/api"
2args := new(api.PutObjectArgs)
3 // Set the MIME type of the uploaded content
4args.ContentType = "text/javascript"
5 // Set the length of the uploaded content
6args.ContentLength = 1024
7 // Set the cache expiration time
8args.Expires = "Mon, 19 Mar 2018 11:55:32 GMT"
9 // Set the cache behavior
10args.CacheControl = "max-age=3600"
11 // Set the crc32c value of the uploaded data
12args.ContentCrc32c = "the crc32c of Your Data"
13 // Set whether the BOS server needs to calculate crc32c for this upload
14args.ContentCrc32cFlag = true
15etag, err := bosClient.PutObject(bucketName, objectName, bodyStream, args)Note: When uploading an object, the SDK automatically sets ContentLength and ContentMD5 to ensure data integrity. If users manually specify ContentLength, it must be a value between 0 and the actual object size. Values outside this range, including negative numbers or larger than the object size, will result in an error.
User-defined meta information
BOS supports user-defined metadata for describing objects. Example usage is shown in the following code:
1// import "github.com/baidubce/bce-sdk-go/services/bos/api"
2args := new(api.PutObjectArgs)
3 // Set user-defined metadata
4args.UserMeta = map[string]string{
5 "name1": "my-metadata1",
6 "name2": "my-metadata2",
7}
8etag, err := bosClient.PutObject(bucketName, objectName, bodyStream, args)Prompt:
- In the above code, the user has customized metadata with names “name1” and “name2”, and values “my-metadata1” and “my-metadata2” respectively.
- When users download this object, this metadata can also be obtained.
- An object may have multiple similar parameters, but the total size of all user meta must not exceed 2KB.
Set storage class when uploading an object
BOS supports standard storage, infrequent access storage, and cold storage. Uploading an object and storing it as an infrequent access storage class is achieved by specifying the StorageClass. The parameters corresponding to the three storage classes are as follows:
| Storage class | Parameters |
|---|---|
| Standard storage | STANDARD |
| Infrequent access storage | STANDARD_IA |
| Cold storage | COLD |
Taking infrequent access storage as an example, the code is as follows:
1// import "github.com/baidubce/bce-sdk-go/services/bos/api"
2args := new(api.PutObjectArgs)
3args.StorageClass = api.STORAGE_CLASS_STANDARD_IA
4etag, err := bosClient.PutObject(bucketName, objectName, bodyStream, args)Append upload
Objects created using the simple upload method described above are all of a standard type and do not support append writes. This limitation can be inconvenient in scenarios where frequent data overwriting occurs, such as log files, video surveillance, and live video streaming.
To address upload challenges, Baidu AI Cloud Object Storage (BOS) supports AppendObject, enabling files to be uploaded in an append-write mode. Objects created with the AppendObject method are marked as Appendable Object types, allowing data to be appended. The size limit for AppendObject operations is 0~5GB. In cases of poor network conditions, it is recommended to use AppendObject by appending small amounts of data at a time (e.g., 256 KB).
Example code for uploading via AppendObject is as follows:
1// import "github.com/baidubce/bce-sdk-go/services/bos/api"
2args := new(api.AppendObjectArgs)
3 // 1. Upload with the original API, set to infrequent access storage, and set the offset position for appending
4args.StorageClass = api.STORAGE_CLASS_STANDARD_IA
5 // An offset of 0 or not setting it means uploading a new file, which supports append upload. When resuming append upload, the offset parameter needs to be set to the file size after the last upload is completed
6args.Offset = 1024
7res, err := bosClient.AppendObject(bucketName, objectName, bodyStream, args)
8 // 2. Encapsulated simple API, only supports setting offset
9res, err := bosClient.SimpleAppendObject(bucketName, objectName, bodyStream, offset)
10 // 3. Encapsulated upload-from-string API, only supports setting offset
11res, err := bosClient.SimpleAppendObjectFromString(bucketName, objectName, "abc", offset)
12 // 4. Encapsulated API for uploading files from the given file name, only supports setting offset
13res, err := bosClient.SimpleAppendObjectFromFile(bucketName, objectName, "<path-to-local-file>", offset)
14 fmt.Println(res.ETag) // Print ETag
15 fmt.Println(res.ContentMD5) // Print ContentMD5
16 fmt.Println(res.NextAppendOffset) // Print NextAppendOffsetSet file meta information
File metadata (object metadata) refers to the attributes of a file when it is uploaded to BOS.
Set Http header of object
The BOS GO SDK essentially calls the backend HTTP API, allowing users to customize the HTTP headers of objects when uploading files. The descriptions of commonly used HTTP headers are as follows:
| Name | Description | Default value |
|---|---|---|
| x-bce-content-crc32c | crc32c of the uploaded data: The BOS server calculates the crc32c value and compares it to the client-calculated value. If they mismatch, the upload fails. The value is saved in metadata, and the server response returns the crc32c value. | Empty string |
| x-bce-content-crc32c-flag | When set to true, the BOS server calculates crc32c, saves it in metadata, and the server response returns the crc32c value. | false |
Reference code is as follows:
1// import "github.com/baidubce/bce-sdk-go/services/bos/api"
2args := new(api.AppendObjectArgs)
3 // Set the crc32c value of the uploaded data
4args.ContentCrc32c = "the crc32c of this append data"
5 // Set whether the BOS server needs to calculate crc32c for this upload
6args.ContentCrc32cFlag = true
7etag, err := bosClient.AppendObject(bucketName, objectName, bodyStream, args, nil)Fetch and upload
BOS enables users to automatically retrieve relevant content from a given URL and save it as an object with a specified name in the designated bucket.
1// import "github.com/baidubce/bce-sdk-go/services/bos/api"
2args := new(api.FetchObjectArgs)
3 // 1. Original API for fetching, set to asynchronous fetch mode
4args.FetchMode = api.FETCH_MODE_ASYNC
5res, err := bosClient.FetchObject(bucketName, objectName, url, args)
6 // 2. Basic fetch API, default to synchronous fetch mode
7res, err := bosClient.BasicFetchObject(bucketName, objectName, url)
8 // 3. Easy-to-use API, directly specify optional parameters
9res, err := bosClient.SimpleFetchObject(bucketName, objectName, url,
10 api.FETCH_MODE_ASYNC, api.STORAGE_CLASS_STANDARD_IA)
11 fmt.Printf("res: %v", res) // Print the fetch resultMultipart upload
Apart from uploading files to BOS via simple upload and append upload methods, BOS also offers another upload method: Multipart Upload. This mode is suitable for several scenarios (but not limited to these):
- When resumable uploads are required.
- When uploading files larger than 5GB.
- When the connection to the BOS server is frequently interrupted due to unstable network conditions.
- Enable streaming file uploads.
- The file size cannot be determined before uploading.
The BOS GO SDK provides control parameters for multipart operations:
- MultipartSize: The size of each part, default is 10 MB, minimum not less than 5 MB
- MaxParallel: The number of concurrencies for chunk operations, default is 10
The following sample code sets the part size to 20 MB and the concurrency number to 100:
1// import "github.com/baidubce/bce-sdk-go/services/bos"
2client := bos.NewClient(<your-ak>, <your-sk>, <endpoint>)
3client.MultipartSize = 20 * (1 << 20)
4client.MaxParallel = 100In addition to the parameters mentioned above, each part number is aligned to 1 MB, with a maximum of 10,000 parts. If the parts are too small and exceed this limit, the part size will be automatically adjusted.
The following will introduce the implementation of Multipart Upload step by step. Suppose there is a file with the local path /path/to/file.zip. Since the file is large, it will be transmitted to BOS in parts.
Initialize multipart upload
Use BasicInitiateMultipartUpload method to initialize a multipart upload event:
1res, err := bosClient.BasicInitiateMultipartUpload(bucketName, objectKey)
2 fmt.Println(res.UploadId) // Print the UploadId got after initializing the multipart uploadThe return result contains UploadId, which is the unique identifier for distinguishing multipart upload events, and we will use it in subsequent operations.
Initialization for uploading infrequent access storage class objects
The InitiateMultipartUpload API provided by the BOS GO SDK can set other parameters related to multipart upload. The following code initializes a multipart upload event for infrequent access storage:
1// import "github.com/baidubce/bce-sdk-go/services/bos/api"
2args := new(api.InitiateMultipartUploadArgs)
3args.StorageClass = api.STORAGE_CLASS_STANDARD_IA
4res, err := bosClient.InitiateMultipartUpload(bucketName, objectKey, contentType, args)
5 fmt.Println(res.UploadId) // Print the UploadId got after initializing the multipart uploadInitialization for uploading a cold storage class object
Initialize a multipart upload event for infrequent access storage:
1// import "github.com/baidubce/bce-sdk-go/services/bos/api"
2args := new(api.InitiateMultipartUploadArgs)
3args.StorageClass = api.STORAGE_CLASS_COLD
4res, err := bosClient.InitiateMultipartUpload(bucketName, objectKey, contentType, args)
5 fmt.Println(res.UploadId) // Print the UploadId got after initializing the multipart uploadUpload parts
The file is then uploaded in multiple parts.
1// import "github.com/baidubce/bce-sdk-go/bce"
2// import "github.com/baidubce/bce-sdk-go/services/bos"
3// import "github.com/baidubce/bce-sdk-go/services/bos/api"
4file, _ := os.Open("/path/to/file.zip")
5 // The part size is aligned with MULTIPART_ALIGN = 1 MB
6partSize := (bosClient.MultipartSize +
7 bos.MULTIPART_ALIGN - 1) / bos.MULTIPART_ALIGN * bos.MULTIPART_ALIGN
8 // Get the file size and calculate the number of parts. The maximum number of parts is MAX_PART_NUMBER = 10,000
9fileInfo, _ := file.Stat()
10fileSize := fileInfo.Size()
11partNum := (fileSize + partSize - 1) / partSize
12 if partNum > bos.MAX_PART_NUMBER { // If the maximum number of parts is exceeded, the part size needs to be adjusted
13 partSize = (fileSize + bos.MAX_PART_NUMBER + 1) / bos.MAX_PART_NUMBER
14 partSize = (partSize + bos.MULTIPART_ALIGN - 1) / bos.MULTIPART_ALIGN * bos.MULTIPART_ALIGN
15 partNum = (fileSize + partSize - 1) / partSize
16}
17 // Create a list to save the ETag and PartNumber information after each part is uploaded
18partEtags := make([]api.UploadInfoType, 0)
19 // Upload parts one by one
20for i := int64(1); i <= partNum; i++ {
21 // Calculate the offset and the upload size of this time
22 uploadSize := partSize
23 offset := partSize * (i - 1)
24 left := fileSize - offset
25 if left < partSize {
26 uploadSize = left
27 }
28 // Create a file stream with specified offset and size
29 partBody, _ := bce.NewBodyFromSectionFile(file, offset, uploadSize)
30 // Upload the current part
31 etag, err := bosClient.BasicUploadPart(bucketName, objectKey, uploadId, int(i), partBody)
32 // Save the serial number and ETag returned after the current part is uploaded successfully
33 partEtags = append(partEtags, api.UploadInfoType{int(i), etag})
34}The core of the above code is to call the BasicUploadPart method to upload each part, but the following points should be noted:
- The BasicUploadPart method requires that each part, except the last one, must have a size of at least 5 MB. However, the API does not immediately validate the size of the parts; validation only occurs during the completion of the multipart upload.
- The part number must be within the range of 1 to 10,000. If this limit is exceeded, BOS will return an InvalidArgument error code.
- After each Part upload, the return result of BOS will include a
PartETagobject, which is a combination of the uploaded block's ETag and block number (PartNumber). It will be used in subsequent steps to complete the multipart upload, so it must be saved. Generally speaking, thesePartETagobjects will be saved in a List.
Upload parts with parameters
When uploading parts, you can use UploadPart to set the file meta information (object meta). File meta information is the attribute description of the file when the user uploads the file to BOS. Users can customize the Http header of the object when uploading files to set the file metadata. The description of commonly used HTTP headers is as follows:
| Name | Description | Default value |
|---|---|---|
| x-bce-content-crc32c | crc32c of the uploaded data: The BOS server calculates the crc32c value and compares it to the client-calculated value. If they mismatch, the upload fails. The value is saved in metadata, and the server response returns the crc32c value. | Empty string |
| x-bce-content-crc32c-flag | When set to true, the BOS server calculates crc32c, saves it in metadata, and the server response returns the crc32c value. | false |
Reference code is as follows:
1// import "github.com/baidubce/bce-sdk-go/bce"
2// import "github.com/baidubce/bce-sdk-go/services/bos"
3// import "github.com/baidubce/bce-sdk-go/services/bos/api"
4file, _ := os.Open("/path/to/file.zip")
5 // The part size is aligned with MULTIPART_ALIGN = 1 MB
6partSize := (bosClient.MultipartSize +
7 bos.MULTIPART_ALIGN - 1) / bos.MULTIPART_ALIGN * bos.MULTIPART_ALIGN
8 // Get the file size and calculate the number of parts. The maximum number of parts is MAX_PART_NUMBER = 10,000
9fileInfo, _ := file.Stat()
10fileSize := fileInfo.Size()
11partNum := (fileSize + partSize - 1) / partSize
12 if partNum > bos.MAX_PART_NUMBER { // If the maximum number of parts is exceeded, the part size needs to be adjusted
13 partSize = (fileSize + bos.MAX_PART_NUMBER + 1) / bos.MAX_PART_NUMBER
14 partSize = (partSize + bos.MULTIPART_ALIGN - 1) / bos.MULTIPART_ALIGN * bos.MULTIPART_ALIGN
15 partNum = (fileSize + partSize - 1) / partSize
16}
17 // Create a list to save the ETag and PartNumber information after each part is uploaded
18partEtags := make([]api.UploadInfoType, 0)
19 // Upload parts one by one
20for i := int64(1); i <= partNum; i++ {
21 // Calculate the offset and the upload size of this time
22 uploadSize := partSize
23 offset := partSize * (i - 1)
24 left := fileSize - offset
25 if left < partSize {
26 uploadSize = left
27 }
28 // Create a file stream with specified offset and size
29 partBody, _ := bce.NewBodyFromSectionFile(file, offset, uploadSize)
30 args := new(api.UploadPartArgs)
31
32 // Set the crc32c value of the uploaded data
33 args.ContentCrc32c = "the crc32c of this part"
34 // Set whether the BOS server needs to calculate crc32c for this upload
35 args.ContentCrc32cFlag = true
36 // Upload the current part
37 etag, err := bosClient.UploadPart(bucketName, objectKey, uploadId, int(i), partBody)
38 // Save the serial number and ETag returned after the current part is uploaded successfully
39 partEtags = append(partEtags, api.UploadInfoType{int(i), etag})
40}Complete multipart upload
Complete the multipart upload as shown in the following code:
1// import "github.com/baidubce/bce-sdk-go/services/bos/api"
2completeArgs := api.CompleteMultipartUploadArgs{Parts: partEtags}
3res, _ := bosClient.CompleteMultipartUploadFromStruct(
4 bucketName, objectKey, uploadId,&completeArgs)
5 // Output the content of the result object
6fmt.Println(res.Location)
7fmt.Println(res.Bucket)
8fmt.Println(res.Key)
9fmt.Println(res.ETag)The partETags in the above code is the list of partETag saved in the second step. After BOS receives the part list submitted by the user, it will verify the validity of each data Part one by one. Once all data parts are validated, BOS will assemble the data parts into a complete Object.
Cancel multipart upload
Users can cancel multipart uploads by using the abortMultipartUpload method.
1bosClient.AbortMultipartUpload(bucketName, objectKey, uploadId)Retrieve unfinished multipart upload
Users can obtain the unfinished multipart upload events in the bucket by the ListMultipartUploads method.
1// List all unfinished chunk information under the given bucket
2res, err := bosClient.BasicListMultipartUploads(bucketName)
3 // Output the status information of the return result
4fmt.Println(res.Bucket)
5fmt.Println(res.Delimiter)
6fmt.Println(res.Prefix)
7fmt.Println(res.IsTruncated)
8fmt.Println(res.KeyMarker)
9fmt.Println(res.NextKeyMarker)
10fmt.Println(res.MaxUploads)
11 // Traverse all unfinished chunk information lists
12for _, multipartUpload := range res.Uploads {
13 fmt.Println("Key:", multipartUpload.Key, ", UploadId:", multipartUpload.UploadId)
14}Note: 1. By default, if the number of multipart upload events in a bucket exceeds 1,000, only 1,000 objects will be returned, and the IsTruncated value in the return result will be True, and NextKeyMarker will be returned as the starting point for the next reading. 2. To return more multipart upload events, you can use the KeyMarker parameter to read in batches.
Get all uploaded part information
Users can obtain all uploaded parts in an upload event by the ListParts method.
1// Use the basic API to list the currently successfully uploaded parts
2res, err := bosClient.BasicListParts(bucketName, objectKey, uploadId)
3 // Use the original API to provide parameters and list up to 100 currently successfully uploaded parts
4args := new(api.ListPartsArgs)
5args.MaxParts = 100
6res, err := bosClient.ListParts(bucketName, objectKey, uploadId, args)
7 // Print the returned status result
8fmt.Println(res.Bucket)
9fmt.Println(res.Key)
10fmt.Println(res.UploadId)
11fmt.Println(res.Initiated)
12fmt.Println(res.StorageClass)
13fmt.Println(res.PartNumberMarker)
14fmt.Println(res.NextPartNumberMarker)
15fmt.Println(res.MaxParts)
16fmt.Println(res.IsTruncated)
17 // Print part information
18for _, part := range res.Parts {
19 fmt.Println("PartNumber:", part.PartNumber, ", Size:", part.Size,
20 ", ETag:", part.ETag, ", LastModified:", part.LastModified)
21}Note: 1. By default, if the number of multipart upload events in a bucket exceeds 1,000, only 1,000 objects will be returned, and the IsTruncated value in the return result will be True, and NextPartNumberMarker will be returned as the starting point for the next reading. 2. To return more multipart upload events, you can use the PartNumberMarker parameter to read in batches.
The above example is implemented sequentially using APIs without concurrent execution. If you need to speed up, users need to implement concurrent uploads. For the convenience of users, BOS client has specially encapsulated the concurrent API for multipart upload UploadSuperFile:
- API:
UploadSuperFile(bucketName, objectName, fileName, storageClass string) error -
Parameters:
- bucketName: The name of the bucket where the object is uploaded
- objectName: The name of the uploaded object
- fileName: Local file name
- storageClass: The storage class of the uploaded object, default is standard storage
-
Response value:
- error: Errors during upload, empty if successful
Users only need to provide bucket, object, filename to perform concurrent multipart uploads, and can also specify the storageClass of the uploaded object.
Automatic three-step upload
For objects larger than 5 GB, the encapsulated three-step upload is provided, please refer to the following code:
1// import "github.com/baidubce/bce-sdk-go/bce"
2 // Upload from local file
3res, err := bosClient.ParallelUpload(bucketName, objectName, fileName, "", nil)Returns the response and error information associated with the request. The arguments provided are the same as those for a simple upload.
Download file
The BOS GO SDK provides a variety of file download APIs, allowing users to download files from BOS in the following ways:
- Simple streaming download
- Download to a local file
- Range download
Simple streaming download
Users can read the object into a stream using the following code:
1// Directly retrieve an object by providing the Bucket and Object
2res, err := bosClient.BasicGetObject(bucketName, objectName)
3 // Retrieve ObjectMeta
4meta := res.ObjectMeta
5 // Retrieve the read stream (io.ReadCloser) of the Object
6stream := res.Body
7 // Ensure to disable the Object read stream
8defer stream.Close()
9 // Call the Read method of the stream object to process the Object
10...Note: 1. The return result object of the above API contains various information of the object, including the bucket where the object is located, the name of the object, metadata, and a read stream. 2. The metadata of the object can be obtained through the ObjectMeta field of the result object, which includes the ETag defined when the Object was uploaded, Http Header, and custom metadata. 3. The read stream of the returned Object can be obtained through the Body field of the result object. By operating the read stream, the content of the Object can be read into a file or memory, or other operations can be performed.
Download to a local file
Users can directly download an object to the specified file using the following code:
1err := bosClient.BasicGetObjectToFile(bucketName, objectName, "path-to-local-file")Range download
To achieve more functions, you can specify the download range and return header to achieve more refined acquisition of the object. If the specified download range is 0-100, it will return data from byte 0 to byte 100, including byte 100, with a total of 101 bytes of data, i.e., [0, 100].
1// Specify the range start position and return header
2responseHeaders := map[string]string{"ContentType": "image/gif"}
3rangeStart := 1024
4rangeEnd := 2048
5res, err := bosClient.GetObject(bucketName, objectName, responseHeaders, rangeStart, rangeEnd)
6 // Only specify the start position
7res, err := bosClient.GetObject(bucketName, objectName, responseHeaders, rangeStart)
8 // Do not specify range
9res, err := bosClient.GetObject(bucketName, objectName, responseHeaders)
10 // Do not specify the optional return header
11res, err := bosClient.GetObject(bucketName, objectName, nil)Based on the range download API, users can implement segmented download and resumable download of files. For the convenience of users, the BOS GO SDK encapsulates the concurrent download API DownloadSuperFile:
- API:
DownloadSuperFile(bucketName, objectName, fileName string) error -
Parameters:
- bucketName: The name of the bucket where the downloaded object is located
- objectName: The name of the downloaded object
- fileName: The name of the local file where the object is saved
-
Response value:
- error: Errors during download, empty if successful
This API uses concurrency control parameters to perform concurrent range downloads, directly downloading to the file specified by the user.
Other methods
Get the storage class of the object
The storage class attributes of an object are divided into STANDARD (standard storage), STANDARD_IA (infrequent access storage), and COLD (cold storage). This can be achieved through the following code:
1res, err := bosClient.GetObjectMeta(bucketName, objectName)
2fmt.Println(res.StorageClass)Get only object metadata
The GetObjectMeta method retrieves only the ObjectMetadata and not the object content itself. Here is an example:
1res, err := bosClient.GetObjectMeta(bucketName, objectName)
2fmt.Printf("Metadata: %+v\n", res)Get file download URL
Users can obtain the URL of a specified object through the following code:
1// 1. Original API, which can set bucket, object name, expiration time, request method, request header and request parameters
2url := bosClient.GeneratePresignedUrl(bucketName, objectName,
3 expirationInSeconds, method, headers, params)
4 // 2. Basic API, default to GET method, only need to set expiration time
5url := bosClient.BasicGeneratePresignedUrl(bucketName, objectName, expirationInSeconds)Description:
expirationInSecondsis the specified URL validity period, calculated from the current time. It is an optional parameter, and the system default value is 1,800 seconds if not configured. To set a permanent non-expiration time, theExpirationInSecondsparameter can be set to -1. You cannot set it to any other negative value.- If the file to be obtained is expected to be publicly readable, the corresponding URL link can be quickly concatenated and obtained through simple rules:
http://{$bucketName}.{$region}.bcebos.com/{$objectName}.
List files in the storage space
The BOS GO SDK allows users to list objects in the following two ways:
- Simple listing
- Advanced listing via parameters
Additionally, users can simulate folder structures while listing files.
Simple listing
When users need to quickly and easily list specific files, they can use the ListObjects method, which returns a ListObjectsResult object containing the results of the request. All descriptive information about the object can be retrieved from the Contents field of the ListObjectsResult object.
1listObjectResult, err := bosClient.ListObjects(bucketName, nil)
2 // Print the status result of the current ListObjects request
3fmt.Println("Name:", listObjectResult.Name)
4fmt.Println("Prefix:", listObjectResult.Prefix)
5fmt.Println("Delimiter:", listObjectResult.Delimiter)
6fmt.Println("Marker:", listObjectResult.Marker)
7fmt.Println("NextMarker:", listObjectResult.NextMarker)
8fmt.Println("MaxKeys:", listObjectResult.MaxKeys)
9fmt.Println("IsTruncated:", listObjectResult.IsTruncated)
10 // Print the specific result of the Contents field
11for _, obj := range listObjectResult.Contents {
12 fmt.Println("Key:", obj.Key, ", ETag:", obj.ETag, ", Size:", obj.Size,
13 ", LastModified:", obj.LastModified, ", StorageClass:", obj.StorageClass)
14}Note: 1. By default, if the number of objects in the bucket is greater than 1,000, only 1,000 objects will be returned, and the IsTruncated value in the return result will be True, and NextMarker will be returned as the starting point for the next reading. 2. To increase the number of returned objects, the marker parameter can be used for reading in batches.
Advanced listing via parameters
Beyond the simple listing previously mentioned, users can execute more customized query functions by configuring parameters in ListObjectsArgs. The available parameters for ListObjectsArgs include the following:
| Parameters | Function |
|---|---|
| Prefix | Restrict the returned object keys to those starting with a specific prefix |
| Delimiter | Delimiter: a character used to group object names. It groups all objects containing the specified prefix and appearing for the first time. Object names located between the delimiter characters are treated as group elements: |
| Marker | Set the result to start returning from the first object in alphabetical order after the marker |
| MaxKeys | Limit the maximum number of returned objects. If not set, the default is 1,000, and the value of max-keys cannot be greater than 1,000 |
Note: 1. If there is an object named with the prefix, when querying only with prefix, all returned keys will still include the object named with the Prefix. For details, see [Recursively List All Files in a Directory](#Simulate folder function). 2. If there is an object named with the prefix, when querying with the combination of prefix and delimiter, there will be Null in all returned keys, and the key names do not contain the prefix. For details, see [View Files and Subdirectories in a Directory](#Simulate folder function).
The following are several examples to illustrate the method of listing via parameters:
1// import "github.com/baidubce/bce-sdk-go/services/bos/api"
2args := new(api.ListObjectsArgs)
3 //Specify the maximum response parameter as 500
4args.MaxKeys = 500
5 // Specify to meet a specific prefix
6args.Prefix = "my-prefix/"
7 // Specify a delimiter to implement folder-like functions
8args.Delimiter = "/"
9 // Set the sorting result after a specific object
10args.Marker = "bucket/object-0"
11listObjectResult, err := bosClient.ListObjects(bucketName, args)Simulate folder function
In BOS storage, the concept of folders does not exist. All data elements are stored as objects. However, for data management purposes, BOS users often simulate folder structures. This is achieved by creating an object with a size of 0, which can be uploaded and downloaded. These objects are displayed as folders in the console if their names end with “/”.
Users can simulate folder structures by combining the delimiter and prefix parameters. The combined behavior of delimiter and prefix is as follows:
If prefix is set to a folder name, files starting with the prefix can be listed, including all files and subfolders (directory) recursively listed under the folder. File names are displayed in contents. If delimiter is set to /, the return values only list the files and subfolders (directory) under the folder. The names of subfolders (directory) under the folder are returned in the CommonPrefixes. The files and folders recursively listed under the subfolders are not displayed.
The following are several application methods:
List all files in the bucket
When users need to get all files under the bucket, they can refer to the following code:
1// import "github.com/baidubce/bce-sdk-go/services/bos/api"
2args := new(api.ListObjectsArgs)
3args.Delimiter = "/"
4listObjectResult, err := bosClient.ListObjects(bucketName, args)Recursively list all files in a directory
You can obtain all files in the a directory by setting the Prefix parameter:
1// import "github.com/baidubce/bce-sdk-go/services/bos/api"
2args := new(api.ListObjectsArgs)
3args.Prefix = "fun/"
4args.MaxKeys = 1000
5listObjectResult, err := bosClient.ListObjects(bucketName, args)
6fmt.Println("Objects:")
7for _, obj := range listObjectResult.Contents {
8 fmt.Println(obj.Key)
9}Output:
1Objects:
2fun/
3fun/movie/001.avi
4fun/movie/007.avi
5fun/test.jpgView files and subdirectories in a directory
With the combination of Prefix and Delimiter, it can list files and subdirectories under the directory:
1// import "github.com/baidubce/bce-sdk-go/services/bos/api"
2args := new(api.ListObjectsArgs)
3args.Delimiter = "/"
4args.Prefix = "fun/"
5listObjectResult, err := bosClient.ListObjects(bucketName, args)
6 // Traverse all Objects (current directory and direct sub-files)
7fmt.Println("Objects:")
8for _, obj := range listObjectResult.Contents {
9 fmt.Println(obj.Key)
10}
11 // Traverse all CommonPrefix (subdirectory)
12fmt.Println("CommonPrefixs:")
13for _, obj := range listObjectResult.CommonPrefixes {
14 fmt.Println(obj.Prefix)
15}Output:
1Objects:
2fun/
3fun/test.jpg
4CommonPrefixs:
5fun/movie/In the returned results, the list in ObjectSummaries shows the files under the fun directory. The list in CommonPrefixs shows all subfolders under the fun directory. It can be seen that the two files fun/movie/001.avi and fun/movie/007.avi are not listed because they belong to the movie directory under the fun folder.
List storage properties of objects in a bucket
After users complete the upload, if they need to view the storage class properties of all objects in a specified bucket, it can be achieved through the following code:
1listObjectResult, err := bosClient.ListObjects(bucketName, args)
2for _, obj := range listObjectResult.Contents {
3 fmt.Println("Key:", obj.Key)
4 fmt.Println("LastModified:", obj.LastModified)
5 fmt.Println("ETag:", obj.ETag)
6 fmt.Println("Size:", obj.Size)
7 fmt.Println("StorageClass:", obj.StorageClass)
8 fmt.Println("Owner:", obj.Owner.Id, obj.Owner.DisplayName)
9}Permission control
Set access permissions for objects
Currently, BOS supports two methods for setting ACLs. The first method is to use Canned ACL. During PutObjectAcl, the object access permission is set via the header "x-bce-acl" or "x-bce-grant-permission". Currently configurable permissions include private and public-read. The two types of headers cannot appear in the same request simultaneously. The second method is to upload an ACL file. For details, refer to [Setting Object Permission Control](BOS/API Reference/Object-Related Interface/Basic Operations/PutObject.md)..
Set Canned ACL
Canned ACL is a predefined access permission, allowing users to set it for specific objects, supporting three interfaces:
1// 1. Use x-bce-acl Header to set
2 err := bosClient.PutObjectAclFromCanned(bucketName, objectName, cannedAcl) //The value of cannedAcl can be: private, public-read
3 // 2. Use x-bce-grant-{permission} Header to set
4err1 := bosClient.PutObjectAclGrantRead(bucketName, objectName, userId)
5err2 := bosClient.PutObjectAclGrantFullControl(bucketName, objectName, userId)
6 // userId is the authorized user, supports variable parameters, pass in multiple user IDsSet custom ACL
Users can refer to the following code to set bucket's custom access permissions, supporting four different parameters:
1// import "github.com/baidubce/bce-sdk-go/bce"
2// import "github.com/baidubce/bce-sdk-go/services/bos/api"
3 // 1. Directly upload ACL file stream
4aclBodyStream := bce.NewBodyFromFile("<path-to-acl-file>")
5err := bosClient.PutObjectAcl(bucketName, objectName, aclBodyStream)
6 // 2. Directly use ACL json string
7 // id: user id, permission: user permission
8aclString := `{
9 "accessControlList":[
10 {
11 "grantee":[{
12 "id":"e13b12d0131b4c8bae959df4969387b8"
13 }],
14 "permission":["FULL_CONTROL"]
15 }
16 ]
17}`
18err := bosClient.PutObjectAclFromString(bucketName, objectName, aclString)
19 // 3. Use ACL file
20err := bosClient.PutObjectAclFromFile(bucketName, objectName, "<acl-file-name>")
21 // 4. Set using ACL struct object
22grantUser1 := api.GranteeType{Id: "<user-id-1>"}
23grantUser2 := api.GranteeType{Id: "<user-id-2>"}
24grant1 := api.GrantType{
25 Grantee: []api.GranteeType{grantUser1},
26 Permission: []string{"FULL_CONTROL"},
27}
28grant2 := api.GrantType{
29 Grantee: []api.GranteeType{grantUser2},
30 Permission: []string{"READ"},
31}
32grantArr := make([]api.GrantType, 0)
33grantArr = append(grantArr, grant1)
34grantArr = append(grantArr, grant2)
35args := &api.PutObjectAclArgs{AccessControlList: grantArr}
36err := bosClient.PutObjectAclFromStruct(bucketName, objectName, args)Obtain access permissions for objects
The following code retrieves an object's access permission:
1result, err := bosClient.GetObjectAcl(bucketName, objectName)The fields of the returned result object contain detailed information about access permissions, with specific definitions as follows:
1type GetObjectAclResult struct {
2 AccessControlList []struct{
3 Grantee []struct{
4 Id string
5 }
6 Permission []string
7 }
8}Delete access permissions for objects
For objects with set access permissions, this interface can be called to delete:
1err := bosClient.DeleteObjectAcl(bucketName, objectName)Delete file
Delete single file
The following code can be referenced to delete an object:
1// Specify the name of the object to be deleted and the bucket where it is located
2err := bosClient.DeleteObject(bucketName, objectName)Delete multiple files
Users can also delete multiple files under the same bucket with one call, with the following parameters:
| Parameter name | Description | Parent node |
|---|---|---|
| objects | A container holding object information to be deleted, containing one or more object elements | - |
| +key | Name of object to be deleted | objects |
An example is as follows:
1// import "github.com/baidubce/bce-sdk-go/services/bos/api"
2// import "github.com/baidubce/bce-sdk-go/bce"
3objectList := `{
4 "objects":[
5 {"key": "aaa"},
6 {"key": "bbb"}
7 ]
8}`
9 // 1. Original API, providing list stream of multiple objects
10objectListStream, err := bce.NewBodyFromString(objectList)
11res, err := bosClient.DeleteMultipleObjects(bucketName, objectListStream)
12 // 2. Provide json string for deletion
13res, err := bosClient.DeleteMultipleObjectsFromString(bucketName, objectList)
14 // 3. Provide List object for deleting objects
15deleteObjectList := make([]api.DeleteObjectArgs, 0)
16deleteObjectList = append(deleteObjectList, api.DeleteObjectArgs{"aaa"})
17deleteObjectList = append(deleteObjectList, api.DeleteObjectArgs{"bbb"})
18multiDeleteObj := &api.DeleteMultipleObjectsArgs{deleteObjectList}
19res, err := bosClient.DeleteMultipleObjectsFromStruct(bucketName, multiDeleteObj)
20 // 4. Directly provide the list of names of objects to be deleted
21deleteObjects := []string{"aaa", "bbb"}
22res, err := bosClient.DeleteMultipleObjectsFromKeyList(bucketName, deleteObjects)Description:
When deleting multiple objects at one time, the returned result contains a list of names of objects that failed to be deleted. When some objects are deleted successfully,
rescontains a list of names that failed to be deleted. When some objects are deleted successfully,errisnilandresis notnil. To determine that all are deleted successfully:errisnilandresis&{[]}.
Check if a file exists
Users can check if a file exists through the following operations:
1// import "github.com/baidubce/bce-sdk-go/bce"
2_, err := bosClient.GetObjectMeta(bucketName, objectName)
3if realErr, ok := err.(*bce.BceServiceError); ok {
4 if realErr.StatusCode == 404 {
5 fmt.Println("object not exists")
6 }
7}
8fmt.Println("object exists")Get and update file meta information
File metadata, or object metadata, describes the attributes of files uploaded to BOS. It includes two types: HTTP standard attributes (HTTP headers) and user-defined metadata.
Get file meta information
The GetObjectMeta method retrieves only the ObjectMetadata and not the object content itself. Here is an example:
1res, err := bosClient.GetObjectMeta(bucketName, objectName)
2fmt.Printf("Metadata: %+v\n", res)Modify file meta information
BOS allows users to modify the metadata of an object by copying it. To do this, set the destination bucket and object to match the source bucket and object while specifying new metadata. Metadata modification is achieved through this copy operation. If no new metadata is specified, an error will occur. In such cases, the copy mode must be set to "replace" (default is "copy"). The following is an example:
1// import "github.com/baidubce/bce-sdk-go/bce"
2args := new(api.CopyObjectArgs)
3 // Must set the copy mode to “replace”. The default “copy” cannot modify metadata
4args.MetadataDirective="replace"
5 // Set the Metadata parameter value, please refer to the official website for specific fields
6args.LastModified = "Wed, 29 Nov 2017 13:18:08 GMT"
7args.ContentType = "text/json"
8 // Use the CopyObject API to modify Metadata, the source object and the destination object are the same
9res, err := bosClient.CopyObject(bucketName, objectName, bucketName, objectName, args)Copy files
Copy a file
Users can copy an object using the CopyObject method, as shown in the following code:
1// 1. Original API, which can set copy parameters
2res, err := bosClient.CopyObject(bucketName, objectName, srcBucketName, srcObjectName, nil)
3 // 2. Ignore copy parameters and use default
4res, err := bosClient.BasicCopyObject(bucketName, objectName, srcBucketName, srcObjectName)
5fmt.Println("ETag:", res.ETag, "LastModified:", res.LastModified)The result object returned by the above API contains the ETag of the newly created object and its LastModified timestamp.
Copy object by setting copy parameters
Example code:
1// import "github.com/baidubce/bce-sdk-go/services/bos/api"
2args := new(api.CopyObjectArgs)
3 // Set user-defined metadata
4args.UserMeta = map[string]string{"<user-meta-key>": "<user-meta-value>"}
5res, err := bosClient.CopyObject(bucketName, objectName, srcBucketName, srcObjectName, args)
6fmt.Println("ETag:", res.ETag, "LastModified:", res.LastModified)During the copying process, users can inspect the ETag or modification status of the source object to determine whether the copy operation is necessary. Detailed parameter explanations are as follows:
| Name | Types | Description | Whether required |
|---|---|---|---|
| x-bce-copy-source-if-match | String | If the ETag value of the source object matches the ETag value provided by the user, the copy operation is performed; otherwise, it fails. | No |
| x-bce-copy-source-if-none-match | String | If the ETag value of the source object does not match the ETag value provided by the user, the copy operation is performed; otherwise, it fails. | No |
| x-bce-copy-source-if-unmodified-since | String | If the source object has not been modified since x-bce-copy-source-if-unmodified-since, the copy operation will proceed; otherwise, it will fail. | No |
| x-bce-copy-source-if-modified-since | String | If the source object has been modified since x-bce-copy-source-if-modified-since, the copy operation will proceed; otherwise, it will fail. | No |
| x-bce-acl | String | Headers supported by CannedACL enable users to set object permissions, allowing either private or public-read access. | No |
| x-bce-content-crc32c | String | crc32c of the uploaded data: The BOS server calculates the crc32c value and compares it to the client-calculated value. If they mismatch, the upload fails. The value is saved in metadata, and the server response returns the crc32c value. | No |
| x-bce-content-crc32c-flag | Bool | When set to true, the BOS server calculates crc32c, saves it in metadata, and the server response returns the crc32c value. | No |
The corresponding example code:
1// import "github.com/baidubce/bce-sdk-go/services/bos/api"
2args := new(api.CopyObjectArgs)
3 // Set user-defined metadata
4args.UserMeta = map[string]string{"<user-meta-key>": "<user-meta-value>"}
5 // Set copy-source-if-match
6args.IfMatch = "111111111183bf192b57a4afc76fa632"
7 // Set copy-source-if-none-match
8args.IfNoneMatch = "111111111183bf192b57a4afc76fa632"
9 // Set copy-source-if-modified-since
10args.IfModifiedSince = "Fri, 16 Mar 2018 17:07:21 GMT"
11 // Set copy-source-if-unmodified-since
12args.IfUnmodifiedSince = "Fri, 16 Mar 2018 17:07:21 GMT"
13 // Set x-bce-acl
14args.CannedAcl = "private"
15 // Set the crc32c value of the uploaded data
16args.ContentCrc32c = "the crc32c of Your Data"
17 // Set whether the BOS server needs to calculate crc32c for this upload
18args.ContentCrc32cFlag = true
19res, err := bosClient.CopyObject(bucketName, objectName, srcBucketName, srcObjectName, args)
20fmt.Println("ETag:", res.ETag, "LastModified:", res.LastModified)Multipart copy
In addition to copying files using the CopyObject API, BOS also provides an alternative copy method: ParallelCopy. This mode can be used in various scenarios (but is not limited to these):
- When a resumable copy process is required.
- For copying files larger than 5GB.
- When the connection to the BOS server frequently disconnects due to poor network conditions.
Example code:
1// Automatic three-step copy, which can set copy parameters
2res, err := bosClient.ParallelCopy(srcBucketName, srcObjectName, bucketName, objectName, nil, nil)
3fmt.Println("ETag:", res.ETag)Synchronous copy function
Currently, BOS’s CopyObject API operates in a synchronous manner. This means BOS waits until the copy operation is fully completed before returning a success status. While synchronous copying provides a more accurate status for users, it also takes longer and is proportional to the file size.
The synchronous copy approach aligns better with industry standards and improves compatibility with other platforms. It also simplifies the server’s business logic and enhances service efficiency.
Archive type
Archive upload
Set StorageClass to api.STORAGE_CLASS_ARCHIVE when uploading
1// import "github.com/baidubce/bce-sdk-go/services/bos/api"
2args := api.PutObjectArgs{
3 StorageClass:api.STORAGE_CLASS_ARCHIVE,
4}
5resCommon, err := bosClient.PutObjectFromFile(bucketName, objectName, filename, &args)Restore
Use the newly encapsulated api RestoreObject
1// import "github.com/baidubce/bce-sdk-go/services/bos/api"
2 // restoreDays indicates the duration of the restored state, a numeric value in days, greater than 0 but less than 30. The default restoring duration is 7 days, with a maximum of 30 days.
3 // RESTORE_TIER_STANDARD = "Standard" //Standard restoration of objects - default
4 // RESTORE_TIER_EXPEDITED = "Expedited" //Expedited restoration of objects
5err := bosClient.RestoreObject(bucketName, objectName, restoreDays, api.RESTORE_TIER_STANDARD)Select file
The SelectObject API supports executing SQL statements on the content of objects in specified formats (CSV/JSON) in BOS, and returns the file content required by users after screening, analyzing, and filtering the object content through SQL, a structured query language. Example code:
Select CSV
1// import "encoding/base64"
2 // CSV file selection parameters
3csvObject := "test.csv"
4 originSQL := "select * from BosObject" // Note that BosObject here is a fixed writing, representing csvObject
5base64SQL := base64.StdEncoding.EncodeToString([]byte(originSQL))
6csvArgs := &api.SelectObjectArgs{
7 SelectType: "csv",
8 SelectRequest: &api.SelectObjectRequest{
9 Expression: base64SQL,
10 ExpressionType: "SQL",
11 InputSerialization: &api.SelectObjectInput{
12 CompressionType: "NONE",
13 CsvParams: map[string]string{
14 "fileHeaderInfo": "IGNORE",
15 "recordDelimiter": "Cg==",
16 "fieldDelimiter": "LA==",
17 "quoteCharacter": "Ig==",
18 "commentCharacter": "Iw==",
19 },
20 },
21 OutputSerialization: &api.SelectObjectOutput{
22 OutputHeader: false,
23 CsvParams: map[string]string{
24 "quoteFields": "ALWAYS",
25 "recordDelimiter": "Cg==",
26 "fieldDelimiter": "LA==",
27 "quoteCharacter": "Ig==",
28 },
29 },
30 RequestProgress: &api.SelectObjectProgress{
31 Enabled: true,
32 },
33 },
34}
35csvRes, err := bosClient.SelectObject(bucketName, csvObject, csvArgs)
36if err != nil {
37 fmt.Println(err)
38 return
39}Select JSON
1// import "encoding/base64"
2 // JSON file selection parameters
3jsonObject := "test.json"
4 originSQL := "select * from BosObject" // Note that BosObject here is a fixed writing, representing jsonObject
5base64SQL := base64.StdEncoding.EncodeToString([]byte(originSQL))
6jsonArgs := &api.SelectObjectArgs{
7 SelectType: "json",
8 SelectRequest: &api.SelectObjectRequest{
9 Expression: base64SQL,
10 ExpressionType: "SQL",
11 InputSerialization: &api.SelectObjectInput{
12 CompressionType: "NONE",
13 JsonParams: map[string]string{
14 "type": "LINES",
15 },
16 },
17 OutputSerialization: &api.SelectObjectOutput{
18 JsonParams: map[string]string{
19 "recordDelimiter": "Cg==",
20 },
21 },
22 RequestProgress: &api.SelectObjectProgress{
23 Enabled: true,
24 },
25 },
26}
27jsonRes, err := bosClient.SelectObject(bucketName, jsonObject, jsonArgs)
28if err != nil {
29 fmt.Println(err)
30 return
31}Parse result
The response returned by the SelectObject API adopts a fixed structure encoding method. For specific parsing methods, please refer to [SelectObject code example](BOS/API Reference/Object-Related Interface/Select scanning/SelectObject.md)
Operations related to object tagging
- Example of using PutObjectTag
1args := &api.PutObjectTagArgs{
2 ObjectTags: []api.ObjectTags{
3 {
4 []api.ObjectTag{
5 {
6 Key: "key1",
7 Value: "value1",
8 },
9 {
10 Key: "key2",
11 Value: "value2",
12 },
13 {
14 Key: "key3",
15 Value: "value3",
16 },
17 },
18 },
19 },
20}
21err = bosClient.PutObjectTag(bucketName, objectName, args)- Example of using GetObjectTag
1res, err := bosClient.GetObjectTag(bucketName, objectName)- Example of using DeleteObjectTag
1err := bosClient.DeleteObjectTag(bucketName, objectName)