You can see the specific migration progress of corresponding file list from status.log, and if the client crashes, you can find corresponding breakpoint to resume the migration from the log, and the progress is resumed according to the grading of the task list.

Example:

client.properties

taskList=/path/to/task_list.txt 

The contents of task_list.txt are shown as follows:

/home/work/list/object_list1.txt 
/home/work/list/object_list2.txt 
#/home/work/list/object_list2.txt 

The contents of object_list1.txt are shown as follows:

bucket1 obj1
bucket1 obj2
bucket1 obj3
#bucket1 obj4 
bucket1 obj5

The contents of object_list2.txt are shown as follows:

bucket1 obj21
#bucket1 obj22 
bucket1 obj23
bucket1 obj24
bucket1 obj25
bucket1 obj26

The contents of status.log are shown as follows:

2015-12-31 17:13:42.013 object_list1.txt 2/-1 
2015-12-31 17:13:45.014 object_list2.txt 3/-1 
2015-12-31 17:13:48.014 object_list2.txt 4/6 
2015-12-31 17:13:51.014 object_list1.txt 5/5 finished 
2015-12-31 17:13:53.014 object_list2.txt 4/6 

The command object_list1.txt 2/-1 indicates that the second line of the list object_list1.txt has been processed, among which -1 means that the file has not been read and we do not know how many lines it has; if it crashes at this time, we do not know which files in the list have been migrated because of concurrency.

To restart the task from a progress, a new list file needs to be create, for example in this example object_list1_continue.txt, object_list2_continue.txt, and then place the list file to resume the progress into the new task_list, with refer to task_list_continue.txt in this example.

The configuration to restart the task and to resume the progress is shown as follows:

In client.properties

taskList=task_list_continue.txt 

Among them, thetask_list_continue.txtare shown as follows

object_list1_continue.txt 
object_list2_continue.txt 

Just use the new configuration to restart the client.

Redo

When all the lists that need to be migrated are processed, the files that failed or have problems during the migration need to be redone (retransmitted). Sort out the list to be retransmitted from the migration log: In terms of redo, the unsent section uses the migration policy for normal migration, but for the failed lists of files, the policy FORCE_OVERWRITE is considered for migration as there may be data error due to migration.