Command Format

The general format of this tool is as follows:

    java -jar bosutil.jar -endpoint {endpoint} -ak {access_key} -sk {secret_key}
        -cmd {command} [-bucketName {bucket_name}] [-other_options {option_value}...]

The fields marked with "{}" in the above format are required parameters, and the corresponding parameter value should be enclosed by double quotation marks, such as -endpoint "http://bj.bcebos.com". The fields marked with "[]" are optional parameters, which are determined by the cmd parameter. For the "-cmd" option, only one of the following supported commands can be selected for execution:

Description: The following example is used in the Linux/Unix environment. If you use it in Windows environment, you should note that:

• Use the symbol ^ at the command line break.
• Use of the ": In the Windows environment, the parameters cannot be enclosed by the mark ' but the mark ". If the parameter contains “"”, it needs to be escaped using the mark “"”.

Public Options

The table below lists the public options supported by the command line. These four options are the required options given in the command format. Other options are set according to the specified cmd.

LIST_OBJECTS Command

Feature

It recursively lists all objects information in the bucket given by the user to the file.

Syntax

java -jar bosutil.jar -endpoint {endpoint} -ak {access_key} -sk {secret_key}
    -cmd LIST_OBJECTS -bucketName {bucket_name}

Option

Required AK, SK, endpoint, and cmd parameters, as well as name parameter bucketName of the bucket listed.

Other additional parameters are ignored if they are specified.

Detailed Description

This command lists all objects in the specified bucket to a local file. This file is saved in the same directory of the Java runtime. It uses the combination of given bucketName and "_object_list" as the file name. If the given bucketName is "mybucket" and current Java run directory is "/home/user1", it generates a file "/home/user1/mybucket_object_list". Each line of this file is the information of an object, including bucket name, full name of the object (including prefix), object size, last modification time of the object, and eTag of the object. The columns of output files are separated by \t(tab). The format example of output files is shown as follows:

BucketName\tObjectKey\tSize\tLastModified\tEtag\n
... ... ... ...

Usage Example

java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
    -cmd LIST_OBJECTS -bucketName "bucket"

SET_CONTENT_TYPE_BY_FILE_EXTENSION Command

Feature

This command is used to set the contentType information of the given objects..

Syntax

    java -jar bosutil.jar -endpoint {endpoint} -ak {access_key} -sk {secret_key}
        -cmd SET_CONTENT_TYPE_BY_FILE_EXTENSION
        [-objectList {objectListFilePath}] |
        [-bucketName {bucket_name}
        -objectName {object_name}
        -prefix {prefix}
        -marker {marker}
        -contentType {content-type}
        -defaultContentType {default content-type}

Option

For the parameters except for the required AK, SK, endpoint, and cmd parameters, see the table below:

Detailed Description

This command is used to set the content-type information for the object specified in the Option under the specified endpoint according to the AK/SK information.

This tool is provided with a built-in dictionary of common suffixes and Content-Type. For the object whose Content-Type is to set, the Util tool determines its Content-Type by the user-specified contentType parameter, built-in dictionary, and defaultContentType parameter according to the following rules:

1. Get the suffix of this object first, and match it with the contentType parameter in the command line. If it is specified exactly, the specified contentType parameter is used. If the suffix of an object is ".js" and there is an option ".js:text/javascript" in the contentType parameter, it is specified exactly.
2. If the contentType parameter is not exactly specified in the command line, it judges whether there is the fuzzy specified ".*:text/plain" in contentType. If so, it is used. If it is not specified fuzzy or the contentType parameter is not given, redirect to the built-in dictionary to get the Content-Type corresponding to the suffix of this object. If it is got successfully, the built-in Content-Type is used for setting.
3. If the built-in dictionary does not match, it is set according to the value specified by the defaultContentType parameter in the command line. If the defaultContentType parameter is not set in the command line, it returns directly, does not set this object, and prints this object to the command line.

This command specifies the object of Content-Type to be set in two ways.

• -objectList method: That is, it provides an object list file. Each column is separated by\t(tab), and the format of each row is "BucketName\tObjectName", where \t represents the tab character.
• -bucketName -objectName -prefix -marker method: That is, bucketName, prefix, marker and objectName are used for matching. This way must provide the bucketName parameter to specify the bucket. Other parameters can be exactly specified by the objectName. Furthermore, the prefix and marker can be used for fuzzy matching. Usage of marker: Users only need to enter the prefix for the first execution. If the execution fails halfway, you do not need to re-execute it, but you need to find the file last_marker+${bucket_name}.txt in the current directory, get the last line (with format <object name, number of tasks completed>), take the object name as the value of the marker, and execute the command again to continue the batch operation tasks not completed last time.

Usage Example

java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
    -cmd SET_CONTENT_TYPE_BY_FILE_EXTENSION \
    -defaultContentType "text/plain" \
    -contentType ".js:text/javascript;.flv:video/flv" \
    -objectList "./a/filelist"

java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
    -cmd SET_CONTENT_TYPE_BY_FILE_EXTENSION \
    -defaultContentType "text/plain" \
    -contentType ".js:text/javascript;.flv:video/flv" \
    -bucketName "abc" \
    -objectName "test-object"

SET_OBJECTS_ACL

Feature

This command is used to set the ACL information of given objects.

Syntax

java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key"
    -cmd SET_OBJECTS_ACL
    -bucketName {bucket_name}
    [-objectName {object_name}] |
    [-prefix {prefix}]
    [-marker {marker}]
    [-canned {canned acl}] |
    [-aclConfigFile {acl_config_file}]
    [-threadNumber {concurrent number}]

Option

Except for the required ak, sk, endpoint, and cmd parameters, see other parameters listed in the table below:

Detailed Description

This command is used to set the ACL information for the object specified in the options under the specified endpoint according to the AK/SK information.

This tool sets the ACL information for objects in two ways:

1. Specify the access control of objects through the canned parameter, which can be PRIVATE and PUBLIC-READ.
2. Provide an ACL file through the aclConfigFile parameter. The file content is in the JSON format as follows:

{
    "accessControlList": [
        {
            "grantee": [
                {
                    "id": "e13b12d0131b4c8bae959df4969387b8"
                }
            ],
            "permission": [
                "READ"
            ]
        }
    ]
}

This command specifies the objects of ACL to be set as follows:

• -bucketName -objectName -prefix -marker method: means that bucketName, prefix, marker and objectName are used for matching. This way must provide the bucketName parameter to specify the bucket. Other parameters can be exactly specified by the objectName. Furthermore, the prefix and marker can be used for fuzzy matching. Usage of the marker: Users only need to enter the prefix for the first execution. If the execution fails halfway, you do not need to re-execute, but you need to find the file last_marker+${bucket_name}.txt in the current directory, get the last line (with format <object name, number of tasks completed>), take the object name as the value of the marker, and execute the command again to continue the batch operation tasks not completed last time.

Usage Example

java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
    -cmd SET_OBJECTS_ACL \
    -bucketName "bucket1" \
    -objectNmae "prefix1/object1" \
    -canned "PRIVATE"

java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
    -cmd SET_OBJECTS_ACL \
    -bucketName "bucket1" \
    -prefix "prefix" \
    -marker "" \
    -canned "PRIVATE"

java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
    -cmd SET_OBJECTS_ACL \
    -bucketName "bucket1" \
    -prefix "prefix" \
    -marker "" \
    -aclConfigFile "example.txt"

TEST_CONTENT_TYPE Command

Feature

The Util tool is provided with a built-in dictionary of common suffixes and Content-Type, which can be used to test the Content-Type preset in the dictionary by executing the TEST_CONTENT_TYPE command.

Syntax

java -jar bosutil.jar \
    -cmd TEST_CONTENT_TYPE \
    -extension {extension-value}

Options

The required parameters are cmd and extension parameters.

Other additional parameters are ignored if they are specified.

Detailed Description

This command is only used to get the Content-Type information of specified suffix. Therefore, it needs to neither interact with BOS nor specify the AK/SK and endpoint information but needs to provide the suffix parameters to test, and then return the corresponding information to the command line according to the internally supported Content-Type setting rules, or return the prompt information “Not supported” for the unsupported suffix.

Usage Example

java -jar bosutil.jar -cmd TEST_CONTENT_TYPE -extension ".mp4"

GET_OBJECTS_META Command

Feature

It is used to get the metadata information of these objects according to the user-specified object-getting mode.

Syntax

java -jar bosutil.jar -endpoint {endpoint} -ak {access_key} -sk {secret_key}
    -cmd GET_OBJECTS_META
    [-objectList {objectListFilePath}] |
    [-bucketName {bucket_name}
    -objectName {object_name}
    -prefix {prefix}
    -marker {marker}

Options

For the parameters except for the required AK, SK, endpoint, and cmd parameters, see the table below:

Detailed Description

This command is used to get the objects and their metadata information in two ways described in the Options under the specified endpoint according to the AK/SK information. The metadata information of each object is one line output to the command line display.

The objects can be got in the following two ways to get the metadata information:

• -objectList: means that it provides an object list file. Each column is separated by \t(tab), each column is separated by \t(tab) and the format of each row is "BucketName\ObjectName", where \t represents the tab character?.
• -bucketName -objectName -prefix -marker: means that bucketName, prefix, marker, and objectName are used for matching. This way must provide the bucketName parameter to specify the bucket. Other parameters can be exactly specified by the objectName. Furthermore, the prefix and marker can be used for fuzzy matching. Usage of marker: Users only need to enter the prefix for the first execution. If the execution fails halfway, you do not need to re-execute but need to find the file last_marker+${bucket_name}.txt in the current directory, get the last line (with format <object name, number of tasks completed>), take the object name as the value of marker, and execute the command again to continue the batch operation tasks not completed last time.

Users can select one of the above two ways. If two ways are specified, the first way is selected to read from the objectList parameter, and the parameters specified in the second way is ignored. Other additional parameters are ignored if they are specified.

Usage Example

java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
    -cmd GET_OBJECTS_META \
    -objectList "./a/filelist"

java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
    -cmd GET_OBJECTS_META \
    -bucketName "abc" \
    -objectName "test-object"

SET_OBJECTS_META Command

Feature

It is used to set the metadata information of these objects according to the user-specified object getting mode.

Syntax

java -jar bosutil.jar -endpoint {endpoint} -ak {access_key} -sk {secret_key}
    -cmd SET_OBJECTS_META
    -bucketName {bucket_name}
    -objectName {object_name}
    -prefix {prefix}
    -marker {marker}
    -meta {metadata_string}
    -replace
    -threadNumber {concurrent_number}

OptionS

For the parameters except for the required AK, SK, endpoint, and cmd parameters, see the table below:

The required parameters meta and replace are described below. Other additional parameters are ignored if they are specified.

• meta: The metadata information is given in the form of "key: value", and multiple information is connected with "|" and provided as a string parameter.

  • The meta value content supports the variable expression, e.g., cacheControl:|__user_meta__path:aa/bb|contentDisposition:attachment;filename=
    "${object_basename}", where ${object_basename} indicates the object basename. During the batch processing of meta information of objects, you can set a meta field of all objects through the variable expression to implement different meta values.
  • Users can customize the user meta in the meta, and the key of user meta must start with __user_meta__.
• Replace: If this parameter is provided, it indicates that the original metadata information of objects is replaced. Otherwise, it is not replaced by default. No replacement means to merge the original meta and the new meta. If the contents of the new meta and original meta are repeated, the new meta is used.

Detailed Description

This command is used to get the objects in the object getting mode as described in the Options under the specified endpoint according to the AK/SK information and set the metadata information of these objects according to the meta and replace parameters. The success or error information is output to the command line.

Get the objects of the metadata to set in the following ways:

-bucketName -objectName -prefix -marker method: means that bucketName, prefix, marker, and objectName are used for matching. This way must provide the bucketName parameter to specify the bucket. Other parameters can be exactly specified by the objectName. Furthermore, the prefix and marker can be used for fuzzy matching. Usage of the marker: Users only need to enter the prefix for the first execution. If the execution fails halfway, you do not need to re-execute, but you need to find the file last_marker+${bucket_name}.txt in the current directory, get the last line (with format <object name, number of tasks completed>), take the object name as the value of marker, and execute the command again to continue the batch operation tasks not completed last time.

Usage Example

• linux/macOS

   java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
        -cmd SET_OBJECTS_META \
        -objectName "test-bueckt" \
        -prefix "test-" \
        -meta 'cacheControl:|__user_meta__path:aa/bb|contentDisposition:attachment;filename="${object_basename}"' \

- Windows: Pay attention to the escape of double quotation marks `"`.

```shell
java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" ^
 -cmd SET_OBJECTS_META ^
 -bucketName "test-bueckt" ^
 -prefix "test-" ^
 -meta "contentType:text/javascript|contentEncoding:gb18030|contentDisposition:attachment; filename=\"${object_basename}\"|__user_meta__path:a/b" ^
 -replace

DELETE_OBJECTS command

Feature

Delete the objects got in the user specified object getting mode in batch.

Note: Be careful to perform the operation because this operation is unrecoverable.

Syntax: java -jar bosutil.jar -endpoint {endpoint} -ak {access_key} -sk {secret_key} -cmd DELETE_OBJECTS [-objectList {objectListFilePath}] | [-bucketName {bucket_name} -objectName {object_name} -prefix {prefix} -marker {marker}

Option

For the parameters other than the required AK, SK, endpoint, and cmd parameters, see the table below:

Detailed Description

This command is used to delete the objects got in the specified way as described in the Options under the specified endpoint according to the AK/SK information. If an error object is deleted, the error information is printed to the command line. If an object is uploaded during the execution of this command and the prefix of the object matches the prefix that is being deleted, it cannot ensure that the uploaded object is deleted.

If objectList is specified, only all files listed in this file are deleted, the columns in this file are separated by \t(tab), and the content format is as follows:

${bucketName}\t${objectName}

Usage Example

java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
    -cmd DELETE_OBJECTS \
    -objectList "./a/filelist"

java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
    -cmd DELETE_OBJECTS \
    -bucketName "test-bucket" \
    -prefix "test-"

RENAME_OBJECTS Command

Feature

Rename the objects in batch according to the user-specified object rename information file.

Syntax

java -jar bosutil.jar -endpoint {endpoint} -ak {access_key} -sk {secret_key}
    -cmd RENAME_OBJECTS
    -objectList {objectListFilePath}

Options

For the parameters other than the required AK, SK, endpoint, and cmd parameters, see the table below:

Detailed Description

This command is used to operate each rename information provided in the file specified in the Options under the specified endpoint according to the AK/SK information. The operation of this command is completed in two steps. Execute the copy operation first, and then execute the delete operation. Therefore, if an error occurs in any step, the recovery operation is executed to return and output the error information.

This command automatically reads the file and renames the specified source object and the new name for each line in batch. The progress information of the processed file is saved in a file starting with "last_marker_", followed by the basename of objectlist file name and with ".txt" as the extension (such as last_marker_objectlist.txt), which saves the number of objectList lines and the number of objects processed actually. Other additional parameters are ignored if they are specified.

Usage Sample

java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
    -cmd RENAME_OBJECTS \
    -objectList "./rename-file-list"

CHANGE_PREFIX Command

Feature

This command is to change the prefix according to the prefix in the user-specified bucket.

Syntax

java -jar bosutil.jar -endpoint {endpoint} -ak {access_key} -sk {secret_key}
    -cmd CHANGE_PREFIX
    -bucketName {bucket_name}
    -prefix {original_preifx}
    -newPrefix {new_prefix}

Options

For the parameters other than the required AK, SK, endpoint, and cmd parameters, see the table below:

Other additional parameters are ignored if they are specified.

Detailed Description

This command simulates the folder mode to rename the directory. It finds all objects under the prefix in the bucket under the specified endpoint according to the AK/SK information, creates the names under the prefix for each object in turn, and renames them in batch. Each renaming operation of the prefix is completed in two steps. Execute the copy operation first, and then execute the delete operation. Therefore, if an error occurs in any step, the recovery operation is executed to return and output the error information. If the objects are uploaded to the same prefix during execution, it cannot ensure that the uploaded objects are modified.

Usage Sample

java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
    -cmd CHANGE_PREFIX \
    -bucketName "mybucket" \
    -prefix "test" \
    -newPrefix "demo"

COMPUTE_SIZE Command

Feature

This command is to compute the total size of all objects under the entire prefix according to the prefix in the user-specified bucket.

Syntax

java -jar bosutil.jar -endpoint {endpoint} -ak {access_key} -sk {secret_key}
    -cmd COMPUTE_SIZE
    -bucketName {bucket_name}
    [-prefix {preifx}]
    [-friendly {friendly}]
    [-level {level}]
    [-threadNumber {concurrent_number}]

Options

For the parameters other than the required AK, SK, endpoint, and cmd parameters, see the table below:

Additional parameters are ignored if they are specified.

Detailed Description

This command simulates the folder mode to compute the directory size. It finds all objects under the prefix in the bucket under the specified endpoint according to the AK/SK information, gets the size of each object in turn for batch cumulative computation, and outputs the total size of this prefix, which is similar to the computation of the total size of the entire directory. If the objects are uploaded to the same prefix during the execution, it cannot ensure that the uploaded objects are computed. You can improve the computing concurrency and save running time by setting the appropriate level and threadNumber value. This command does not count the space occupied by the fragment object, while the occupied space size on the console includes the fragment.

Usage Example

java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_secret_key" \
    -cmd COMPUTE_SIZE \
    -bucketName "mybucket" \
    -prefix "test"

java -jar bosutil.jar -endpoint "http://bj.bcebos.com" -ak "your_access_key" -sk "your_security_key" \
    -cmd COMPUTE_SIZE \
    -bucketName "mybucket" \
    -level "2" \
    -threadNumber "20"

TEST_SPEED Command

Feature

This command is to test the download speed.

Syntax

java -jar bosutil.jar -endpoint {endpoint} -ak {access_key} -sk {secret_key}
    -cmd TEST_SPEED
    [-testSize 10485760]

Options

For the parameters except for the required ak, sk, endpoint, and cmd parameters, see the table below:

Description

Additional parameters are ignored if they are specified.