I have set up a log router for a Dataflow step with the following
filter: resource.type=”dataflow_step” AND logName=”projects/${PROJECT_ID}/logs/dataflow.googleapis.com%2Fworker” AND (severity=”INFO” OR severity=”ERROR”)
This router is configured to push the data to a Pub/Sub topic, which triggers a Cloud Function to perform an action. My goal is to notify downstream users whether the Dataflow job is successful or not. However, when checking my Cloud Run function logs, I see many messages, including multiple Status=Done messages for a single Dataflow job. I need to notify the downstream users only once per Dataflow job.
How can I achieve this?