I’m using a GCP cloud function, planning to trigger runs via requests. My function runs a web scraper to build a dataset, which should probably take around 30 minutes. I think I saw GCP options for scheduling jobs, but in my case I need to manually renew and pass in an API key every run and I thought it’d be easier to do this by making HTTP requests to the GCP function endpoint and passing the API key in the request body.

However, the whole job has to run before my cloud function can send back a response, so it’ll always timeout from the requester side. This doesn’t interfere with the ability of my job to run, but it seems to imply that cloud functions aren’t made for running long jobs. Are GCP cloud functions a bad fit for what I’m trying to do? Are there better alternatives?