Server Properties

The file server.properties is used to configure ports, flow limitation and thread count of RPC services, and only the listening port number is modified without special requirements, with the default listening port number of 8090.

## The port number listened by server progress 
serverPort=8090 

# Number of processing threads corresponding to each client request 
fileWorkerNum=50 

Client Properties

client.propertiesIt must be encoded with UTF8 It mainly includes the information on migration task list and concurrency, and some configurations in the configuration files are reloaded automatically after being modified, such as serverThreadNum (the number of concurrent threads expected to be used by server for a single migration request); some cannot be reloaded, such as taskList (task list), please refer to the descriptions in the reference files for specific configurations.

Configuration Example

Parameter Description

• serverThreadNumber After the single request set by such configuration is sent to server, server uses multiple threads to process such request. If the conditions of source and destination end as well as server resources permit, the number is set with the standard to take full of the server bandwidth, which needs adjusting several times. Memory occupied by server = serverThreadNumber \x{2217} blockSize \x{2217} maxConnectionsPerServer

• serverList Migrate the server list, each server consists of ip and port number, and ; is used between servers for separation, the example is shown as follows:

  serverList=192.168.1.1 8090;192.168.1.2 8090 
  The option serverList` can be null, no migration server is available if it is null, the migration is suspended. 

• filesPerRequest Number of files distributed to migration server by each request. It is recommended that each migration request is completed in more than 30 seconds after this value is configured. As one request uses one tcp for connection, it is necessary to ensure that the connection of last tcp can be disconnected normally at the beginning of the next request to avoid excessive tcp fd. You can try to increase the value from small to large at the beginning of the migration until the completion time of each migration request is greater than 30 seconds.

• taskList Task list (absolute path). The taskList contains one or more migration task lists (absolute path of the file) that are pre-processed, and the number of lines of each file represents the granularity of the task, the smaller the granularity, the less repetitive the migration work will be as the task progress recovers.

  Example:

  /home/work/list/object_list1.txt 
  /home/work/list/object_list2.txt 
  #/home/work/list/object_list2.txt 

  The contents of above taskList represent the list of three files. When it starts with #, it indicates that this line is commented out, and is not handled as a migration task.

• listDelimiter and listFormat

  The format of file list corresponds to the properties name in the configuration file, and the field value is case-sensitive. The listDelimiterrepresents the delimiter between each line of each column of data in the file list; the listFormat represents which columns shall be included in each line of the file list, and the column is separated with;.

  listDelimiter=" " 
  listFormat=BucketName;ObjectName;dstObjectName 

  Example for the contents of file list

  bucketA 	 prefixA/aaa/object1 	 prefixB/bbb/object1 
  bucketA 	 prefixA/aaa/object2 	 prefixB/bbb/object2 
  #bucketA prefixA/aaa/object3 prefixB/bbb/object3 
  bucketA 	 prefixA/aaa/object4 	 prefixB/bbb/object4 

  When it starts with #, it indicates that this line is commented out, and is not handled as a migration task.

  The fields supported the format of current file list include:

  field	Meaning	Example
  srcAk	AK at source end
  srcSk	SK at source end
  dstAk	AK at destination end (bos)
  dstSk	SK at destination end (bos)
  srcBucketName	bucket name at source end
  BucketName	bucket name at source end
  ObjectName	object name at source end
  dstBucketName	bucket name at destination end
  dstObjectName	The name after migration can be changed by such item
  contentType	If not available at source end, store at contentType of bos after migration	text/plain
  contentDisposition	If not available at source end, store at contentDisposition of bos after migration	attachment; filename="abc.mp4"
  toBosIa	Whether to migrate to bos ia	toBosIa

srcBucketName/BucketName is yes/or relation, taking one or the other, and the priority shall be given to srcBucketName.

• migrationStrategy Migration policies

  1. FORCE_OVERWRITE: mandatory overwriting of the files at the destination end may destroy destination-end files, which is dangerous and shall be used with caution and used on the premise of determining the requirements;
  2. KEEP_DESTITNATION: skip if a file with the same name on the destination end exists
  3. KEEP_NEWER: if a file with the same name exists at the destination end, it is newer than that at the source end, skip the file; otherwise override it; if the file with the "same name" does not exist at the destination end (the file name may be changed during the migration), copy it to the destination end directly according to the migration rules

  • KEEP_DESTITNATION and KEEP_NEWER are two migration policies that are appropriate for smooth business switching, are equally efficient for double-write at source and destination end, and shall be used depending on the specific requirements.
  • The migration policy is set to be an unknown policy other than the three policies, and it is set to be KEEP_DESTINATION by default.

• headQps, putQps and bandwidth: QPS and bandwidth speed limit

  1. HeadQps represents the upper limit of QPS for getobjectMeta requests when migrating from data source to BOS or comparing and checking BOS data, with a maximum of 5000 times/second;
  2. PutQps represents the upper limit of QPS for putObject requests when migrating from data source to BOS, with a maximum of 1500 times/second;

  3. Bandwidth represents the upper limit of bandwidth for putObject requests when migrating from data source to BOS, with a maximum of 3000 MBps.

     headQps=5000 
     putQps=1500 
     bandwidth=3000