I’m creating a pipeline where I have ton of files that I have to loop through, log and load to snowflake
steps in pipeline
- get metadata activity
- filter for files
- set variable (filename)
- foreach loop
- copy data step (copy file to BLOB)
- data flow – where I take the file in BLOB and add filename) (please lmk if I can do this in copy data step, pretty sure I can’t)
step 6 is where I’m having the issue. where do I create the parameter, in the data flow step itself. or in the source and sink data sets? I create a parameter in the data flow itself but then I don’t see the parameter in the source and sink datasets to parameterize the filename that the flow writes to