DTS architecture

The DTS data transmission service’s back-end system mainly consists of the transmission module and monitoring module.

1. The transmission module’s features include data extraction, data transformation, and data loading, of which:

   • The data extraction feature is used for exporting the data awaiting migration from the upstream database, for example, exporting source data information like database table structure, exporting of the full snapshot of the data in the table, and exporting of incremental data that are generated by the database in real time.
   • The data transformation feature transforms the data extracted from the upstream database into a format acceptable to the downstream database. When the upstream and downstream database types differ from one another (in other words, in case of data migration between heterogeneous databases), upstream database format needs to be interpreted.
   • The data loading feature is responsible for loading the transformed data into the downstream database. Thanks to a variety of optimized techniques like data merge, batch loading, and parallel loading, DTS can ensure high-efficiency data migration between upstream and downstream databases.
2. The monitoring module is mainly used in monitoring the health status of the transmission module. If a given transmission module is abnormal and fails to run, the monitoring module is responsible for assigning data migration task running on transmission module to other transmission modules in normal running, and the entire process is completed within one second, ensuring the high availability of the DTS task.

DTS migration mode

The DTS (data transmission service) migrates the data from the user’s self-built database to the RDS, and establishes data synchronization or establishes the data synchronization feature between two RDS services. The DTS divides the entire data migration process into three steps, as detailed below:

1. Structure migration: structure migration task is responsible for exporting the upstream database’s data structure information like database table structure, view, trigger, and storage process, which are subjected to format conversion and imported into downstream database (due to different upstream/downstream database types, some structure information might not support the migration).
2. Full migration: When starting, full migration task first records incremental data synchronization point (used in subsequent incremental migration). After recording, it reads full data by batch from the upstream database through snapshot read, and imports the data into the downstream database. The entire process supports breakpoint continuation;
3. Incremental migration: The incremental migration task does not adopt query means but directly extracts incremental data from logs in an upstream database. Thus it has strong timeliness. When starting, incremental migration task extracts increments starting from the synchronization points recorded before full migration, writes them into the downstream database, and finally completes all increments generated after full migration snapshot read. That ensures real-time consistency between the upstream and downstream database. Since the idempotent operation is adopted when incremental data is written into the downstream database (the idempotent operation is premised on constraints by primary key or unique key in the table), the data from incremental migration overlaps with data exported from full migration. However, no data inconsistency like data duplication takes place.