Using BOS Service via BOS-Util

Command format

The general format of the tool is as follows:

1    java -jar bosutil.jar -endpoint {endpoint} -ak {access_key} -sk {secret_key}
2        -cmd {command} [-bucketName {bucket_name}] [-other_options {option_value}...]

The fields with "{}"in the above format are required parameters, and the corresponding parameter values should be enclosed in double quotes, such as: -endpoint "http://bj.bcebos.com". The fields using "[]" are optional parameters, specifically determined by the cmd parameter. For the "-cmd" option, only one of the following supported commands can be selected for execution:

Note: The following examples are all in Linux/Unix environment. If used on Windows, you need to pay attention to:

• Use ^ for command line
• Use of ": On Windows, parameters must not be enclosed with ', but with ". If a parameter includes ", it must be escaped using ".

Public options

The following table outlines the public options available for the command line. These four options are mandatory for the command format, while other options depend on the specified cmd.

LIST_OBJECTS command

Function

List all objects under the specified bucket recursively and export the list to a file.

Syntax

1java -jar bosutil.jar -endpoint {endpoint} -ak {access_key} -sk {secret_key}
2    -cmd LIST_OBJECTS -bucketName {bucket_name}

Option

Mandatory parameters include ak, sk, endpoint, cmd, and the bucket name parameter bucketName corresponding to the bucket being listed.

If specified, other additional parameters will be ignored.

Detailed description

This command will list all objects under the specified bucket into a local file, which is saved in the same directory as the Java runtime, with the filename composed of the given bucketName and "_object_list." If The given bucketName is "mybucekt"and the current Java runtime directory is"/home/user1", a file "/home/user1/mybucket_object_list” will be generated, and each line of this file contains the information of an object, including the bucket name, object full name (including prefix), object size, last modification time of object and ETag of obejct. The columns of the output file is separated by \t(tab). The format of the output file is as follows:

1BucketName\tObjectKey\tSize\tLastModified\tEtag\n
2... ... ... ...

Usage example

1java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
2    -cmd LIST_OBJECTS -bucketName "bucket"

SET_CONTENT_TYPE_BY_FILE_EXTENSION command

Function

This command configures the contentType information for the specified object.

Syntax

1    java -jar bosutil.jar -endpoint {endpoint} -ak {access_key} -sk {secret_key}
2        -cmd SET_CONTENT_TYPE_BY_FILE_EXTENSION
3        [-objectList {objectListFilePath}] |
4        [-bucketName {bucket_name}
5        -objectName {object_name}
6        -prefix {prefix}
7        -marker {marker}
8        -contentType {content-type}
9        -defaultContentType {default content-type}

Option

In addition to the required AK, SK, endpoint, and cmd parameters, other parameters are listed in the table below:

Detailed description

This command sets the content-type information for the objects specified in Option under specified endpoint based on the AK/SK information.

The tool includes a built-in dictionary of common suffixes and content-types. For the object whose content-type needs to be set, the Util tool will determine the content-type based on the user-defined contentType parameter, the built-in dictionary, and the defaultContentType parameter, following these specific rules:

1. First, identify the object suffix and compare it with the contentType parameter provided in the command line. If an exact match is found, use the specified contentType parameter. For example, if an object has the suffix ".js" and the contentType parameter includes an entry ".js:text/javascript", it will be treated as an exact match.
2. If the contentType parameter is not precisely specified in the command line, check if there is a fuzzy specification such as ".*:text/plain" in the contentType parameter and use it if available. If no fuzzy specification exists or the contentType parameter is not provided, refer to the built-in dictionary to retrieve the content-type associated with the object suffix. If a match is found, the content-type from the built-in dictionary will be applied.
3. If no match is located in the built-in dictionary, the value specified by the defaultContentType parameter in the command line will be used. If the defaultContentType parameter is not set, the process will terminate without setting the content-type for the object, and the object details will be printed on the command line.

This command offers two methods for specifying the object whose content-type needs to be set.

• -objectList method: It provides an object list file with the columns separated by \t(tab), and the format of each line is "BucketName\tObjectName" (where \t represents a tab character).
• -bucketName -objectName -prefix -marker method: This method uses bucketName, prefix, marker, and objectName for matching. The bucketName parameter is mandatory to specify the bucket, while other parameters support exact specification with objectName or fuzzy matching using prefix and marker. For marker, users only need to provide the prefix during the initial execution. If a failure occurs midway, the execution does not need to restart. Users can locate the file last_marker+${bucket_name}.txt in the current directory, extract the last line (formatted as object name + count of completed tasks), and re-execute the command with this object name as the marker value to continue the unfinished batch operation.

Usage example

1java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
2    -cmd SET_CONTENT_TYPE_BY_FILE_EXTENSION \
3    -defaultContentType "text/plain" \
4    -contentType ".js:text/javascript;.flv:video/flv" \
5    -objectList "./a/filelist"

1java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
2    -cmd SET_CONTENT_TYPE_BY_FILE_EXTENSION \
3    -defaultContentType "text/plain" \
4    -contentType ".js:text/javascript;.flv:video/flv" \
5    -bucketName "abc" \
6    -objectName "test-object"

SET_OBJECTS_ACL

Function

This command sets the ACL information for the specified object

Syntax

1java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key"
2    -cmd SET_OBJECTS_ACL
3    -bucketName {bucket_name}
4    [-objectName {object_name}] |
5    [-prefix {prefix}]
6    [-marker {marker}]
7    [-canned {canned acl}] |
8    [-aclConfigFile {acl_config_file}]
9    [-threadNumber {concurrent number}]

Option

In addition to the required AK, SK, endpoint, and cmd parameters, other parameters are listed in the table below:

Detailed description

This command configures the ACL information for the specified object within the given endpoint using AK/SK credentials.

This tool supports two methods for setting ACL information for an object:

1. Use the canned parameter to specify object access permission, supporting PRIVATE and PUBLIC-READ;
2. Provide an ACL file in JSON format through the aclConfigFile parameter, as follows:

1{
2    "accessControlList": [
3        {
4            "grantee": [
5                {
6                    "id": "e13b12d0131b4c8bae959df4969387b8"
7                }
8            ],
9            "permission": [
10                "READ"
11            ]
12        }
13    ]
14}

This command specifies the object whose ACL is to be set as follows:

• -bucketName -objectName -prefix -marker method: This method uses bucketName, prefix, marker, and objectName for matching. The bucketName parameter is mandatory to specify the bucket, while other parameters support exact specification with objectName or fuzzy matching using prefix and marker. For marker, users only need to provide the prefix during the initial execution. If a failure occurs midway, the execution does not need to restart. Users can locate the file last_marker+${bucket_name}.txt in the current directory, extract the last line (formatted as object name + count of completed tasks), and re-execute the command with this object name as the marker value to continue the unfinished batch operation.

Usage example

1java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
2    -cmd SET_OBJECTS_ACL \
3    -bucketName "bucket1" \
4    -objectNmae "prefix1/object1" \
5    -canned "PRIVATE"

1java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
2    -cmd SET_OBJECTS_ACL \
3    -bucketName "bucket1" \
4    -prefix "prefix" \
5    -marker "" \
6    -canned "PRIVATE"

1java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
2    -cmd SET_OBJECTS_ACL \
3    -bucketName "bucket1" \
4    -prefix "prefix" \
5    -marker "" \
6    -aclConfigFile "example.txt"

TEST_CONTENT_TYPE command

Function

The Util tool has an integrated dictionary of commonly used suffixes and content-types. You can test the predefined content-types in this dictionary using the TEST_CONTENT_TYPE command.

Syntax

1java -jar bosutil.jar \
2    -cmd TEST_CONTENT_TYPE \
3    -extension {extension-value}

Option

The required parameters include cmd and extension parameters.

If specified, other additional parameters will be ignored.

Detailed description

This command is designed solely to retrieve the content-type information for a given suffix. It does not interact with the BOS client, nor does it require AK/SK credentials or endpoint information. Users only need to provide the suffix parameter they wish to test, and the corresponding content-type information will be returned to the command line based on predefined internal settings. For unsupported suffixes, an appropriate unsupported message will be returned.

Usage example

1java -jar bosutil.jar -cmd TEST_CONTENT_TYPE -extension ".mp4"

GET_OBJECTS_META command

Function

Retrieve the metadata of objects based on the user-specified object acquisition method.

Syntax

1java -jar bosutil.jar -endpoint {endpoint} -ak {access_key} -sk {secret_key}
2    -cmd GET_OBJECTS_META
3    [-objectList {objectListFilePath}] |
4    [-bucketName {bucket_name}
5    -objectName {object_name}
6    -prefix {prefix}
7    -marker {marker}

Option

In addition to the required AK, SK, endpoint, and cmd parameters, other parameters are listed in the table below:

Detailed description

According to AK/SK information, under the specified endpoint, the command obtains objects and the metadata information of these objects according to the two methods described in Option. The metadata information of each object is output as a line to the Command line.

The following two methods are supported to obtain object, so as to obtain metadata information:

• -objectList method: It provides an object list file with the columns separated by \t(tab) and \t(tab), and the format of each line is "BucketName\ObjectName" (where \t represents a tab character).
• -bucketName -objectName -prefix -marker method: This method uses bucketName, prefix, marker, and objectName for matching. The bucketName parameter is mandatory to specify the bucket, while other parameters support exact specification with objectName or fuzzy matching using prefix and marker. For marker, users only need to provide the prefix during the initial execution. If a failure occurs midway, the execution does not need to restart. Users can locate the file last_marker+${bucket_name}.txt in the current directory, extract the last line (formatted as object name + count of completed tasks), and re-execute the command with this object name as the marker value to continue the unfinished batch operation.

Users can choose either of the two methods described above. If both methods are specified, the first method utilizing the objectList parameter will be prioritized, while parameters from the second method will be disregarded. Any additional parameters that are provided will also be ignored.

Usage example

1java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
2    -cmd GET_OBJECTS_META \
3    -objectList "./a/filelist"

1java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
2    -cmd GET_OBJECTS_META \
3    -bucketName "abc" \
4    -objectName "test-object"

SET_OBJECTS_META command

Function

Set the metadata for objects based on the object acquisition method defined by the user.

Syntax

1java -jar bosutil.jar -endpoint {endpoint} -ak {access_key} -sk {secret_key}
2    -cmd SET_OBJECTS_META
3    -bucketName {bucket_name}
4    -objectName {object_name}
5    -prefix {prefix}
6    -marker {marker}
7    -meta {metadata_string}
8    -replace
9    -threadNumber {concurrent_number}

Option

In addition to the required AK, SK, endpoint, and cmd parameters, other parameters are listed in the table below:

The required parameters meta and replace are described below, and other additional parameters will be ignored if specified:

• Meta: Specify metadata in the format "key:value". Multiple pieces of metadata should be concatenated using "|" as a delimiter and provided as a single string parameter.

  • The meta value content supports variable expressions, such as cacheControl:|__user_meta__path:aa/bb|contentDisposition:attachment;filename=
    "${object_basename}", where ${object_basename} represents the object basename. When processing the object meta in batch, a variable expression can be used to set a specific meta field for all objects, achieving different meta values.
  • In meta, users can customize user meta. The key for user meta must start with __user_meta__
• Replace: If this parameter is specified, it indicates replacing the object's existing metadata. If not provided, the metadata will not be replaced (this is the default behavior). In the absence of replacement, the original metadata will be merged with the new metadata. In cases where new metadata conflicts with existing metadata, the new metadata will take precedence.

Detailed description

According to AK/SK information, under the specified endpoint, the command sets the metadata information of the objects obtained according to the object acquisition method described in the Option and the meta and replace parameters. Success or error messages will be output to the command line.

Use the following methods to obtain the objects for which metadata needs to be set:

-bucketName -objectName -prefix -marker method: This method uses bucketName, prefix, marker, and objectName for matching. The bucketName parameter is mandatory to specify the bucket, while other parameters support exact specification with objectName or fuzzy matching using prefix and marker. For marker, users only need to provide the prefix during the initial execution. If a failure occurs midway, the execution does not need to restart. Users can locate the file last_marker+${bucket_name}.txt in the current directory, extract the last line (formatted as object name + count of completed tasks), and re-execute the command with this object name as the marker value to continue the unfinished batch operation.

Usage example

• linux/macOS

  Shell Copy

  1 java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
  2      -cmd SET_OBJECTS_META \
  3      -objectName "test-bueckt" \
  4      -prefix "test-" \
  5      -meta 'cacheControl:|\__user_meta__path:aa/bb|contentDisposition:attachment;filename="${object_basename}"' \

1-   Windows: Note the escaping of double quotes `"`
2```shell
3java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" ^
4 -cmd SET_OBJECTS_META ^
5 -bucketName "test-bueckt" ^
6 -prefix "test-" ^
7 -meta "contentType:text/javascript|contentEncoding:gb18030|contentDisposition:attachment; filename=\"${object_basename}\"|__user_meta__path:a/b" ^
8 -replace

DELETE_OBJECTS command

Function

Perform batch deletion of objects based on the object acquisition method specified by the user.

Note: This operation is irreversible. Please proceed with caution.

Syntax java -jar bosutil.jar -endpoint {endpoint} -ak {access_key} -sk {secret_key} -cmd DELETE_OBJECTS [-objectList {objectListFilePath}] | [-bucketName {bucket_name} -objectName {object_name} -prefix {prefix} -marker {marker} -include {include} -preview {preview}

Option

In addition to the required AK, SK, endpoint, and cmd parameters, other parameters are listed in the table below:

Detailed description

According to AK/SK information, this command will delete the objects obtained by the method specified in Option in batch under the specified endpoint. If there is an error in deleting the object, the error information will be printed to the command line. If an object is uploaded during the execution of this command, and the object prefix matches the prefix being deleted, it cannot be guaranteed that the newly uploaded object will be deleted.

When using the include regex match to delete an object, ensure that the regular expression aligns with your expectations before executing the command. You can use the preview parameter to view the list of objects slated for deletion in advance.

If objectList is specified, only the files listed in this file will be deleted. The columns in the file are separated by \t(tab) , , with the content format as

1${bucketName}\t${objectName}

Usage example

1java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
2    -cmd DELETE_OBJECTS \
3    -objectList "./a/filelist"

1java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
2    -cmd DELETE_OBJECTS \
3    -bucketName "test-bucket" \
4    -prefix "test-"

1java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
2    -cmd DELETE_OBJECTS \
3    -bucketName "test-bucket" \
4    -prefix "test-"
5    -include "test-.*jpeg"
6    -preview "true"

RENAME_OBJECTS command

Function

Execute batch renaming operations using the object renaming information file specified by the user.

Syntax

1java -jar bosutil.jar -endpoint {endpoint} -ak {access_key} -sk {secret_key}
2    -cmd RENAME_OBJECTS
3    -objectList {objectListFilePath}

Option

In addition to the required AK, SK, endpoint, and cmd parameters, other parameters are listed in the table below:

Detailed description

This command performs a batch operation on each piece of rename information provided by the file specified in the Option under the specified endpoint according to the AK/SK information. The operation of this command is divided into two steps. First, the copy operation is performed, and then the delete operation is performed. Therefore, if there is an error in any step, the recovery operation will be performed for rollback, and the error information will be output.

This command will automatically read the file and perform a batch rename operation on each line specifying the source object and the new name. The progress information of the processed file is saved in a file with "last_marker_" as the prefix, followed by the basename of the objectList filename, and ".txt" as the extension (e.g., last_marker_objectlist.txt), which records the line count of objectList and the actual count of processed objects. Other additional parameters will be ignored if specified.

Usage example

1java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
2    -cmd RENAME_OBJECTS \
3    -objectList "./rename-file-list"

CHANGE_PREFIX command

Function

Modify the prefix within the bucket based on the user-defined prefix.

Syntax

1java -jar bosutil.jar -endpoint {endpoint} -ak {access_key} -sk {secret_key}
2    -cmd CHANGE_PREFIX
3    -bucketName {bucket_name}
4    -prefix {original_preifx}
5    -newPrefix {new_prefix}

Option

In addition to the required AK, SK, endpoint, and cmd parameters, other parameters are listed in the table below:

If specified, other additional parameters will be ignored.

Detailed description

This command simulates directory renaming operations similar to folder management. Using AK/SK credentials, it identifies all objects under the specified prefix in the bucket at the given endpoint, assigns new names under a different prefix for each object sequentially, and processes the renaming in batches. Each prefix renaming operation is divided into two steps: first, a copy operation is performed; second, a delete operation is executed. If an error occurs at any step, a recovery mechanism will be triggered for rollback, and the error will be logged. During execution, if objects are uploaded to the same prefix simultaneously, it cannot guarantee that the newly uploaded objects will be processed.

Usage example

1java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
2    -cmd CHANGE_PREFIX \
3    -bucketName "mybucket" \
4    -prefix "test" \
5    -newPrefix "demo"

COMPUTE_SIZE command

Function

Compute the total size of all objects under the prefix specified by the user in the bucket.

Syntax

1java -jar bosutil.jar -endpoint {endpoint} -ak {access_key} -sk {secret_key}
2    -cmd COMPUTE_SIZE
3    -bucketName {bucket_name}
4    [-prefix {preifx}]
5    [-friendly {friendly}]
6    [-level {level}]
7    [-threadNumber {concurrent_number}]

Option

In addition to the required AK, SK, endpoint, and cmd parameters, other parameters are listed in the table below:

If specified, other additional parameters will be ignored.

Detailed description

This command simulates directory operations for calculating folder sizes. Using AK/SK credentials, it identifies all objects under the specified prefix in the bucket at the given endpoint, retrieves the size of each object sequentially, performs cumulative batch calculations, and outputs the total size of the prefix. This is analogous to calculating the total size of an entire directory. During execution, simultaneous object uploads to the same prefix may result in some newly uploaded objects not being included in the calculation. Concurrency of calculations can be enhanced by adjusting the level and threadNumber settings to optimize runtime. Note that this command does not account for the space occupied by fragmented objects, whereas space usage displayed in the console includes these fragments.

Usage example

1java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
2    -cmd COMPUTE_SIZE \
3    -bucketName "mybucket" \
4    -prefix "test"

1java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_security_key" \
2    -cmd COMPUTE_SIZE \
3    -bucketName "mybucket" \
4    -level "2" \
5    -threadNumber "20"

TEST_SPEED command

Function

Test the download speed.

Syntax

1java -jar bosutil.jar -endpoint {endpoint} -ak {access_key} -sk {secret_key}
2    -cmd TEST_SPEED
3    [-testSize 10485760]

Option

In addition to the required AK, SK, endpoint, and cmd parameters, other parameters are listed in the table below:

Description

If specified, other additional parameters will be ignored.