I am creating a Grafana SLO dashboard with alerting using Terraform.
I have also added the ability to send alerts to slack channel in case any alert fires as below
<code>resource "grafana_slo" "my_slo" {
count = var.provision_slo ? 1 : 0
name = "My APIs"
description = "Test"
objectives {
value = 0.98
window = "28d"
}
query {
ratio {
success_metric = " // query here exists"
total_metric = " // Query here exists"
group_by_labels = ["env"]
}
type = "ratio"
}
label {
key = "team_name"
value = "myTeam"
}
label {
key = "service_name"
value = "myService"
}
destination_datasource {
uid = "The UID is also present"
}
alerting {
fastburn {
annotation {
key = "name"
value = "SLO Burn Rate Very High"
}
annotation {
key = "description"
value = "Error budget is burning too fast"
}
label {
key = "grafana_slo_severity"
value = "critical"
}
label {
key = "grafana_slo_window"
value = "28d"
}
}
slowburn {
annotation {
key = "name"
value = "SLO Burn Rate High"
}
annotation {
key = "description"
value = "Error budget is burning too fast"
}
label {
key = "grafana_slo_severity"
value = "warning"
}
label {
key = "grafana_slo_window"
value = "28d"
}
}
}
}
resource "grafana_contact_point" "slack_alerts" {
name = "Slack Alerts"
slack {
url = "Webhook url on my channel"
recipient = "#MY-alerts-channel"
username = "Grafana SLO Alerts"
title = "Alert: {{ range .Alerts }}{{ .Annotations.name }}{{ end }} for {{ .CommonLabels.service_name }}"
text = "{{ range .Alerts }}Severity: {{ .Labels.grafana_slo_severity }}nDescription: {{ .Annotations.description }}n{{ end }}"
}
}
resource "grafana_notification_policy" "slo_policy" {
group_by = ["..."]
contact_point = grafana_contact_point.slack_alerts.name
group_wait = "30s"
group_interval = "5m"
repeat_interval = "4h"
policy {
matcher {
label = "grafana_slo_severity"
match = "="
value = "critical"
}
contact_point = grafana_contact_point.slack_alerts.name
}
policy {
matcher {
label = "grafana_slo_severity"
match = "="
value = "warning"
}
contact_point = grafana_contact_point.slack_alerts.name
}
}
</code>
<code>resource "grafana_slo" "my_slo" {
count = var.provision_slo ? 1 : 0
name = "My APIs"
description = "Test"
objectives {
value = 0.98
window = "28d"
}
query {
ratio {
success_metric = " // query here exists"
total_metric = " // Query here exists"
group_by_labels = ["env"]
}
type = "ratio"
}
label {
key = "team_name"
value = "myTeam"
}
label {
key = "service_name"
value = "myService"
}
destination_datasource {
uid = "The UID is also present"
}
alerting {
fastburn {
annotation {
key = "name"
value = "SLO Burn Rate Very High"
}
annotation {
key = "description"
value = "Error budget is burning too fast"
}
label {
key = "grafana_slo_severity"
value = "critical"
}
label {
key = "grafana_slo_window"
value = "28d"
}
}
slowburn {
annotation {
key = "name"
value = "SLO Burn Rate High"
}
annotation {
key = "description"
value = "Error budget is burning too fast"
}
label {
key = "grafana_slo_severity"
value = "warning"
}
label {
key = "grafana_slo_window"
value = "28d"
}
}
}
}
resource "grafana_contact_point" "slack_alerts" {
name = "Slack Alerts"
slack {
url = "Webhook url on my channel"
recipient = "#MY-alerts-channel"
username = "Grafana SLO Alerts"
title = "Alert: {{ range .Alerts }}{{ .Annotations.name }}{{ end }} for {{ .CommonLabels.service_name }}"
text = "{{ range .Alerts }}Severity: {{ .Labels.grafana_slo_severity }}nDescription: {{ .Annotations.description }}n{{ end }}"
}
}
resource "grafana_notification_policy" "slo_policy" {
group_by = ["..."]
contact_point = grafana_contact_point.slack_alerts.name
group_wait = "30s"
group_interval = "5m"
repeat_interval = "4h"
policy {
matcher {
label = "grafana_slo_severity"
match = "="
value = "critical"
}
contact_point = grafana_contact_point.slack_alerts.name
}
policy {
matcher {
label = "grafana_slo_severity"
match = "="
value = "warning"
}
contact_point = grafana_contact_point.slack_alerts.name
}
}
</code>
resource "grafana_slo" "my_slo" {
count = var.provision_slo ? 1 : 0
name = "My APIs"
description = "Test"
objectives {
value = 0.98
window = "28d"
}
query {
ratio {
success_metric = " // query here exists"
total_metric = " // Query here exists"
group_by_labels = ["env"]
}
type = "ratio"
}
label {
key = "team_name"
value = "myTeam"
}
label {
key = "service_name"
value = "myService"
}
destination_datasource {
uid = "The UID is also present"
}
alerting {
fastburn {
annotation {
key = "name"
value = "SLO Burn Rate Very High"
}
annotation {
key = "description"
value = "Error budget is burning too fast"
}
label {
key = "grafana_slo_severity"
value = "critical"
}
label {
key = "grafana_slo_window"
value = "28d"
}
}
slowburn {
annotation {
key = "name"
value = "SLO Burn Rate High"
}
annotation {
key = "description"
value = "Error budget is burning too fast"
}
label {
key = "grafana_slo_severity"
value = "warning"
}
label {
key = "grafana_slo_window"
value = "28d"
}
}
}
}
resource "grafana_contact_point" "slack_alerts" {
name = "Slack Alerts"
slack {
url = "Webhook url on my channel"
recipient = "#MY-alerts-channel"
username = "Grafana SLO Alerts"
title = "Alert: {{ range .Alerts }}{{ .Annotations.name }}{{ end }} for {{ .CommonLabels.service_name }}"
text = "{{ range .Alerts }}Severity: {{ .Labels.grafana_slo_severity }}nDescription: {{ .Annotations.description }}n{{ end }}"
}
}
resource "grafana_notification_policy" "slo_policy" {
group_by = ["..."]
contact_point = grafana_contact_point.slack_alerts.name
group_wait = "30s"
group_interval = "5m"
repeat_interval = "4h"
policy {
matcher {
label = "grafana_slo_severity"
match = "="
value = "critical"
}
contact_point = grafana_contact_point.slack_alerts.name
}
policy {
matcher {
label = "grafana_slo_severity"
match = "="
value = "warning"
}
contact_point = grafana_contact_point.slack_alerts.name
}
}
The Dashboard gets generated perfectly and I can even see alerts in firing
state. But they are not reaching the slack channel. I manually tested it from the contact point on the grafana UI and I received the message but nothing automatically.
Also,in the Notification policies tab on the UI, I can see :
I would appreciate any sort of help regarding this. What is going wrong here ?
Thanks!