My company heavily utilizes Terraform to manage our GCP environment. We have has great success in abstracting resources to make managing our environment in a config driven manner.
The one resource we are really having trouble generalizing is resource “google_monitoring_alert_policy” in the realm of creating notifications around resource quotas. We want to have alerts for near limit and limit on individual quotas.
For example below is the MQL needed in the terraform resource to alert on the quota DISKS-TOTAL-GB-per-project-region. When dealing with a large number of quota rersources this becomes a real bear to work with.
Does anyone know if a way to really generalize this and reduce the verbosity or possibly suggest another paradigm to deal with GCP alert policies in Terraform
resource "google_monitoring_alert_policy" "default" {
conditions {
display_name = "Quota usage reached defined threshold"
name = "projects/xxx-xxx-xxx-xxx/alertPolicies/xxxxxxxxxxxx/conditions/xxxxx"
condition_monitoring_query_language {
duration = "60s"
query = "fetch consumer_quota|filter resource.project_id=='xxx-xxx-xxx-xxx'|{metric serviceruntime.googleapis.com/quota/allocation/usage|filter metric.quota_metric=='compute.googleapis.com/disks_total_storage'&&resource.location=='xx-xxxx'|map add [metric.limit_name: 'DISKS-TOTAL-GB-per-project-region']|align next_older(1d)|group_by [resource.project_id,resource.service,metric.quota_metric,metric.limit_name,resource.location],.max;metric serviceruntime.googleapis.com/quota/limit|filter metric.quota_metric=='compute.googleapis.com/disks_total_storage'&&metric.limit_name=='DISKS-TOTAL-GB-per-project-region'&&resource.location=='xx-xxxx'|align next_older(1d)|group_by [resource.project_id,resource.service,metric.quota_metric,metric.limit_name,resource.location],.min}|ratio|every 30s|condition gt(val(), 0.8 '1')"
}
}
}
}