how to create an archiving pipeline which will have only 1 parameter called “tenure” which will copy the last 3 year data from “Source” folder to a different folder “Archive” and delete them from “Source” folder tenure – 3 soure:: 2020/A.csv …. Archive: 2022/A.csv, 2023/A.csv, 2024/A.csv
I tried above problem by parameterizing filename in copydata activity it is copying empty files with names 2022, 2023, 2024 how to solve this problem
3
As you want to copy last 3 years source folders to target folders, you can make use of for-loop in ADF pipeline.
First create tenure parameter of int type and give a default value of current year 2024 for that in the pipeline parameters section.
Now, take a For-Each activity and give below expression in that.
@range(sub(pipeline().parameters.tenure,2),3)
This will generate an array of [2022,2023,2024] if the parameter value is 2024. The pipeline will iterate through this array and will copy from each source folder 2022 to target folder 2022 in each iteration.
Inside For-Each, take a copy activity. For the copy activity source dataset, give your target container in the dataset. For remaining target location path, use wildcard file path in the copy activity source.
For the sink of the copy activity, you need to parameterize the dataset folder path. Create a string parameter and use that in folder path of the dataset.
In the copy activity sink, give @item() as value for this parameter.
To delete the tenure-3 folder from source, after for-loop, take a delete activity. Create a dataset with source container and parameterize the folder path in the dataset. Give the below expression for that parameter in the pipeline.
@string(sub(pipeline().parameters.tenure,2))
This will give the results like below and will delete tenure-3 folder from source.
2