We have 2 folders in adls source and destination. source folder has 4 file and destination has 3. How can we implement the Increment load in ADF. What all activities we have to perform to achieve incremental load in ADF.
My approach :
1) use getmetadata activity parallelly for both source and
destination folder and pass child item on field list.
2) store the immediate result on set variable activity parallelly for both source and destination.
3) Use If condition and pass condition if(not(contains(destination_set_varriable,set_varriable) then use copy activity
2
You can follow below procedure to achieve your requirement:
Use two get metadata activities to get files in source and destination with Child items field list, you can find outputs of get metadata as shown below:
Source:
Destination:
Add filter activity on success of metadata activities with source child items as items @activity('source').output.childItems and give condition @not(contains(activity('destination').output.childItems,item())) as shown below:
You will get filter output activity as shown below:
Add copy activity to filter activity, take source with dataset parameter file_name for file name with value @activity('Filter1').output.Value[0].name as source and add sink dataset, debug the pipeline, the file which is not there in destination will copy to destination successfully as shown below: