Relative Content

Tag Archive for azure-databricksdatabricks-autoloader

Change Tracking Using Databricks Autoloader and ForEachBatch

I am using Autoloader in Trigger Once mode to load Parquet files from an S3 location. My goal is to implement change data capture by comparing the source and target Delta tables to identify and capture INSERTS, UPDATES, and DELETES. Instead of performing a MERGE, I aim to build an append-only table that logs these changes. Below is my ForEachBatch upsert function. Although it runs, it does not capture any DELETES or UPDATES.