Configuration file

server.properties

The server.properties file is used to configure ports, traffic limits, and count of RPC service threads. Without special requirements, generally, only the listener port No. should be modified, and it defaults to 8090.

1# Port No. listened to in the server process
2serverPort=8090
3# Count of corresponding processing threads of each client request
4fileWorkerNum=50

client.properties

client.properties file (must be UTF-8 encoded): This file is the core configuration file for migration tasks, located in the client/resources directory of the project. This file should be modified to adjust client and server settings.

The file mainly contains the migration task list information and concurrency settings. Some configurations in the file will be reloaded automatically upon modification, such as serverThreadNum (expected number of concurrent threads of single migration request on the server). Others cannot be reloaded, such as taskList (migration task list). Refer to notes in the document for detailed configurations.

See the below for the configuration file examples of different types of migration tasks. Refer to the examples in the table to modify the client.properties file.

Configuration example

Parameter description

• serverThreadNumber: This configuration item sets the number of threads used by the server to process a single request after the request is sent. Under conditions where source location, destination location, and server resources allow, this value is typically set to maximize server bandwidth and need to be adjusted several times.server memory usage = serverThreadNumber ∗ blockSize ∗ maxConnectionsPerServer

• serverList: Migration server list, where each server consists of an IP and port No., with servers separated by ;. Example:

  Java Copy

  1# The serverList option can be left empty. If empty, no migration servers are available, and migration will pause.
  2serverList=192.168.1.1 8090;192.168.1.2 8090

• filesPerRequest: Number of files distributed to the migration server after each request. It is recommended that the completion time of each migration request should exceed 30 seconds after this value is configured. Since each request uses one TCP connection, ensure the previous TCP connection is disabled properly before initiating the next request to avoid excessive TCP fd. When the migration starts, gradually increase this value until each migration request takes more than 30 seconds to complete.

• taskList: This parameter requires the absolute path of task list file. The task list file contains one or more pre-processed tasks, each corresponding to an absolute path of an object_list file, where each line in the object_list file corresponds to a data migration in the minimum granularity. The configuration example is as follows:

  Java Copy

  1# The following indicates the absolute path of the task list file is /home/work/list/task_list
  2.taskList=/home/work/list/task_list

  Content example of task list file:

  Java Copy

  1/home/work/list/object_list1
  2/home/work/list/object_list2
  3# /home/work/list/object_list3

  Plain Text Copy

  1> 1. The details above indicate that the taskList includes three object_list entries, each corresponding to an object_list file. Each file contains multiple lines, where each line represents the data migration at its smallest granularity.  
  2> 2. Lines starting with `#` indicate that this line is commented out and is excluded from migration tasks.

• listDelimiter and listFormat

listDelimiter indicates the delimiter between columns in each line of the object_list file. listFormat indicates the columns that should be included in each line in the object_list file, with column names separated by ;. Field values are case-sensitive. The configuration example is as follows:

1```java
2listDelimiter=" "
3listFormat=bucketName;objectName;dstObjectName
4```
5Content example of **object_list file**:
6
7```java
8bucketA	prefixA/aaa/object1	prefixB/bbb/object1
9bucketA	prefixA/aaa/object2	prefixB/bbb/object2
10# bucketA prefixA/aaa/object3 prefixB/bbb/object3
11bucketA	prefixA/aaa/object4	prefixB/bbb/object4
12```
13    > 1. The content above corresponds with the `listDelimiter`and `listFormat` parameters one by one. Each line consists of 3 columns: bucketName, objectName, and dstObjectName, separated by spaces.
14    > 2. Lines starting with `#` indicate that this line is commented out and is excluded from migration tasks.
15
16Currently supported fields in the file list format include the following:
17
18| field | Meaning | Example |
19-----              | -----              | -----
20| srcAk | Source location ak |
21| srcSk | Source location sk |
22| dstAk | Destination location (BOS) ak |
23| dstSk | Destination location (BOS) sk |
24| srcBucketName | Source location bucket name |
25| bucketName | Source location bucket name |
26| objectName | Source location object name |
27| dstBucketName | Destination location bucket name |
28| dstObjectName | Post-migration name. Name can be modified through this item. |
29| contentType | If it is unavailable at the source location, it is the contentType stored in BOS after migration | text/plain |
30| contentDisposition | If it is unavailable at the source location, it is the contentDisposition stored in BOS after migration | attachment; filename="abc.mp4" |
31| toBosIa | Migration to BOS IA | toBosIa |

srcBucketName/bucketName is of an "or" relationship. Choose either of them, with priority given to srcBucketName.

• migrationStrategy: migration strategy

  1. FORCE_OVERWRITE: Forcibly overwrite files at the destination location, which may damage files at the destination location. It is risky. Use this strategy with caution and only when the need is confirmed
  2. KEEP_DESTITNATION: Skip if a file with the same name exists at the destination location
  3. KEEP_NEWER: If a file with the same name exists at the destination location and is newer than the file at the source location, skip it; otherwise, overwrite it. If no file with the "same name" exists at the destination location (as files may require renaming during migration), copy it directly to the destination location according to the migration rules.

  • When the two migration strategies KEEP_DESTITNATION and KEEP_NEWER are applicable to smooth business transitions, and is used for with dual-writing between the source and destination locations. Their efficiency is comparable, and the strategy should be selected based on specific needs.
  • If an unknown migration strategy is set, different from the three predefined ones, it will automatically default to KEEP_DESTINATION.

• headQps, putQps and bandwidth: QPS and bandwidth rate limit

  1. headQps indicates the maximum QPS limit for getObjectMeta requests during the data source-to-BOS migration or BOS data check, with the maximum of 5,000 times/second;
  2. putQps indicates the QPS upper limit for putObject requests during the data source-to-BOS migration, with the maximum of 1,500 times/second;

  3. Bandwidth specifies the maximum bandwidth limit for putObject requests during migration from the data source to BOS, with an upper limit of 3000 Mbps.

     Java Copy

     1headQps=5000
     2putQps=1500
     3bandwidth=3000