Handling Incremental Data Loading and SCD Type 2 for joined tables in Delta Live Tables on Databricks
I’m working on a project utilizing Delta Live Tables on Databricks, where I need to create a dimension (Kimball style) with slowly changing dimension type 2. The dimension is the result of a join between several bronze tables incrementally loaded using autoloader. The resulting silver table needs to be streaming or append-only since I need to use it as a source for a streaming table loaded using apply changes with scd type 2. However, I’m facing challenges due to the streaming behavior.