We’re building a web application through which end users can create and run asynchronous data-intensive search jobs. These search jobs can take anywhere from 1 hour to 1 day to complete.

I’m somewhat new to GCP (and cloud architectures in general) and am trying to best architect a system to handle these asynchronous user tasks. I’ve tentatively settled on using Cloud Run Jobs to handle the data processing task itself, but we will need a basic queueing system to ensure that only so many user requests are handled in parallel (to respect database connection limits, job API rate limits, etc.). I’d like to keep everything centralized to GCP and avoid re-implementing services that GCP can already provide, so I figured that Cloud Tasks could be an easy way to build and manage this queueing system. However, from the Cloud Tasks documentation, it appears that every task created with a generic HTTP target must respond in a maximum of 30 minutes. Frustratingly, it appears that if Cloud Tasks triggers App Engine, the task can be given up to 24 hours to respond. There is no exception or special implementation for Cloud Run Jobs.

With this in mind, will we have to design and build our own queueing system? Or is there a way to finagle Cloud Tasks to work with Cloud Run Job’s 24 hour maximum runtime?