I am building a pipeline in Fabric and I would like to receive an email every time an activity within that dataflow is completed. The problem is that I cannot figure out how to set a variable equal to the name of an activity to pass that through to the email notification. As such, without doing that I would have to create a separate notification activity making the pipeline significantly larger.
Here is how I would imagine the notification would happen. After each one is successful it sets a variable equal to the activity name, and then an email gets sent saying “Dataflow1 completed successfully”, “Dataflow2 completed successfully”, etc.
Is there any way to set a variable equal to the activity name in a pipeline?
In ADF,
Multiple dependencies with the same source are OR’ed together.
Multiple dependencies with different sources are AND’ed together.
my below blog explains it in detail :
https://datasharkx.wordpress.com/2021/08/19/error-logging-and-the-art-of-avoiding-redundant-activities-in-azure-data-factory/
In case if you need to send an email notification the moment the activity is done, you need to have separate flows per activity else based on your above flow it would wait for completion of all activities for the email to be triggered.
Also the names of the activities cannot be derived dynamically as of today.
Currently, there is no dynamic expression for getting the activity names in a pipeline in fabric or normal ADF. So, you can try the below workaround using nested pipelines to achieve your requirement.
First put all of your required activities in a pipeline and call that pipeline from another parent pipeline using an Execute pipeline activity.
In the parent pipeline, take a web activity after the completion of the Execute pipeline activity.
Use the below REST API to get the activity runs details of the child pipeline run using the pipeline run id from the Execute pipeline id.
POST https://api.fabric.microsoft.com/v1/workspaces/<your WS Id>/datapipelines/pipelineruns/<job id>/queryactivityruns
As per the doc,
This returns an array of JSON which will have the information about all activities and status of those activities in the pipeline run.
Using this JSON output from the web activity, you can make a list of activity name and its status and send that information to your mail after the web activity.
You need to get the Access token using methods mentioned in this documentation.
You can go through this similar SO answer to know more about this approach.