I need some help with a GCP based architecture that I am trying to design.
Here is the scenario:
Some point-of-sale devices are writing to a transactional database. If I want to write this sale data to a cloud data warehouse (say bigquery for OLAP), can I use a database migration service (DMS) with CDC enabled and use this data coming into an object storage (like GCS bucket) to populate the data warehouse instead of a pub/sub + dataflow system?
It is transaction data which means I am tracking the sales at the point of sale device.
In an ideal case,
- we would have the POC devices write to the pub/sub as publishers
- have a dataflow job read from the pub/sub as subscribers
- This dataflow job would do some transformations on the data and write to a data warehouse
I agree that this would be very scalable and flexible in terms of data transformations.
But if the sale is not too much can I just link DMS to the transaction database storing this sales-related data and replicate the changes in an object store every time there is a transaction?
My thought process is that DMS-based architecture might be much easier to implement than the pub/sub architecture and much more economical.
Is this a reasonable architecture? Please share your thoughts and correct me if I am getting anything wrong.