I am doing a data engineering project on IoT where I got stuck into a use case which states that
If we create a table in the warehouse to store the IoT sensor data generated every second by IoT devices, 10 devices generate that data. That table storing this data is updated frequently like every 20 seconds and other than that other tables store the dependent data.
So My doubt is how can we utilize the warehouse table for data analysis ?
Possible Answers:
1. Execute the Join query at every 20 seconds to extract the data from all the tables.
Benefits:
Easy to execute
Cons:
More resource consumption because frequently execution of query .
2. Data Modelling
The purpose of data modeling is to make the query execution fast and optimized.
So the point here rises is what kind of data modeling we can do to make the use case a successful one ?