I am creating a data migration tool. What it does in its core is very simple take data from one database and write it to another database.

What I want to do is I have 2 production databases which contain data of my vendors. I want to take those data and write it to my local database for development purpose. My vendor list will change overtime it may decrease or increase. I don’t want to change in code level everytime I have to do data migration. I want that the vendorlist and their read data and write data queries to be given in a configuration file and for every vendor in vendorlist I want to run a step that reads the data from production db and write it to my local db. The core logic for all the steps will be the same. So, I was wondering if it was possible to generate the steps instances dynamically by just changing their parameters for each vendor.

I have thought to design my vendorlist in application.yml file like this:

vendors:
    - name: "vendor1"
      datasource: prodDB1
      readQuery: "some sql query to read data"
      writeQuery: "some sql query to write data"
    - name: "vendor2"
      datasource: prodDB2
      readQuery: "some sql query to read data"
      writeQuery: "some sql query to write data"

     

I have all my datasources configuread already. I know this is not the most elegant approach but we can deal with that later. For now please help me out with how to achieve what I want with Spring batch or else sugget some alternate approaches to this problem.