I am trying to setup an alert in GCP that will trigger when logs are missing for 48hours.
I have a service that runs on a cron job on-prem, but submits logs to GCP when it runs. It only take a minute to run, but only needs to run every other day, potentially once a week. I need to ensure the cron job is running the task. We already have alerting pipelines/notification I want to use within GCP.
I believe the standard absence alert wont work for more than 23.5hrs, and I am hoping that someone out there has a novel workaround for this.
One idea is a GCP function that runs regularly (48hours/week etc), that simply scans the logs in the last window (48hours/week etc) and if it cant finds logs from my service it will spit out it own logs reporting the missing logs.
Then a logging alert could be setup to listen to ‘missing logs’ log from that function, and send an alert if it find one.
Is this stupid? Is there a way I could setup a monitoring metric for this for a long period? I feel it should be easy, but not seeing it.
Any help greatly appreciated.