I am looking for some advice to setup an event-driven architecture in Azure. It’s the first time that I am designing this type of architecture so I am not sure about my setup, but these are my requirements:
The data is coming from a Service Bus that produces up to 500.000 messages per application per day. I want to setup real-time reporting on the data and I want to write the messages to a DWH.
I thought about a realtime and batch-processing setup that looks like this:
-
First, I thought about setting up an Azure Function that filters out unwanted data from the Event Bus and subsequently writes the correct messages to an Event Hub and a Blob Storage.
-
After that, I am thinking about setting up Stream Analytics with Event Hub so I can visualize the data with Power BI in realtime.
-
To process the data in Blob Storage, I thought about using databricks or ADF to transform the data and write it to the DWH.
Do you have any advice for me? Should I change something or consider another solution? I am not sure regarding the performance or costs in this setup, so I am grateful for any feedback. I also thought about processing the messages with databricks and store the data in a data lake with bronze, silver and storage layers (https://www.databricks.com/product/data-lake-on-azure) or to use streaming analytics also as an ETL-tool (https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-solution-patterns).