I created an Eventarc Trigger called by google.cloud.bigquery.v2.JobService.InsertJob to trigger a cloud run job on BigQuery table changes (insert, update, delete) that could happen any time during the day.
This works pretty well as it triggers a Cloud Run instance (whenever a change occurs on my BQ table) that reads data from the BQ table and automatically creates/updates/deletes Jira issues.
The problem is that it might take more or less time to finish processing data which could lead to issues when two changes happen very quickly one after the other on table. In that case, the second change could trigger a cloud run instance in parallel of the first one that did not finish processing data yet.
So, I’m trying to make it more reliable at scale by forcing jobs to be executed in sequence instead of having issues with parallelization.
Is it possible to still use event-based triggering of the same cloud run endpoint (with eventarc) while queuing the executions ?
I’ve started looking at Cloud Tasks and Workflows but could not manage to build a system that works with these resources.
Thank you in advance for your help.