Brief Principle: Incremental data is written to both the old and new storage engines (ES and DB). Then data from the old storage engine (ES) is migrated to the new storage engine (DB). After that finish, data consistency is verified. Finally, read data are switched to the new storage system.
This is my first time performing a data migration, so I would appreciate any reviews, suggestions, or references to mature solutions.
Steps:
- Create Database Tables: Based on the schema of ES, create tables in MongoDB and MySQL respectively.
- Dual Write Incremental Data to ES and DB: Write the real-time incremental data to both ES and DB simultaneously.
- Migrate Historical Data from ES to DB: Write all data from ES to DB using the Logstash tool.
- Data Verification: Traverse all data in ES, ensuring that each data can be found in DB. If the data is inconsistent, modify the data in DB according to the data in ES.
- Switch new Query Engine: Switch the query engine from ES to DB.
- Monitor Data Read/Write: Check if the read/write operations are normal, especially whether the read performance meets the requirements.
- Stop Dual Write and remove ES Cluster: Stop writing incremental data to ES and remove the ES cluster.