Relative Content

Tag Archive for azure-data-factory

Auditing Pipleines in Azure Data Factory

I recently started a Data Engineering role and I was tasked with finding a way to audit our existing pipelines in Azure Data Factory. The requirements are to get all existing components of the pipelines, when they run, if they ran successfully and if any rows were added/deleted to databases after the pipeline ran.

ADF UserErrorFileNotFound using partitions

We are encountering an issue with the Copy activity in Azure DataFactory. Our data source is Oracle, and we use dynamic range partitioning. The data is then saved as parquet files.

If condition in set variable is throwing an error azure data factory

I am extracting href from a link to get the url for pagination, i am storing an api response in blob as json. Using lookup activity i am reading the json and set a variable for next pagination url. The below logic is working fine when file has a pagination and url is in a link array. But when there is no pagination url present in a file then link is not an array and i am getting this error

Rename and reset a file creating a ZIP in Azure Data Factory

I’m banging my head trying to do the following (I don’t even know if it’s possible): create a ZIP, based on a list of files (using “List of Files” option) and, at the same time, rename a file and store it in a different folder, using Azure Data Factory.

How to run a pipeline on Weekly basis in ADF

I have a requirement to run a pipeline only on weekends that is Saturday and Sunday. I used schedule based trigger and apply start date and end date but there is no option I am able to see through I can pipeline on weekend