we are starting to use openshift alertmanager so that it sends us a report based on our config to slack channel. Right now we are using this config to send us a report
receivers:
- name: Default
slack_configs:
- channel: ocp-test-alerts-default
send_resolved: true
api_url: >-
<our-slack-webhook>
title: >-
{{- if eq .Status "firing" }}:rotating_light:{{- else if eq .Status "resolved" }}:white_check_mark:{{- end }} [{{ .Status | toUpper }}]: {{ .Alerts | len }} {{ .CommonLabels.alertname }}
text: |-
{{ range .Alerts }}
• :memo:*Summary:* {{ .Annotations.summary }}
• :mag:*Description:* {{ .Annotations.description }}
• :books:*Namespace | Pod:* {{ .Labels.namespace }} / {{ if .Labels.poddisruptionbudget }}{{ .Labels.poddisruptionbudget }}{{ else }}<>{{ end }}
• :link:*Runbook:* {{ if .Annotations.runbook_url }}<{{ .Annotations.runbook_url }}|URL>{{ else }}No URL{{ end }}
• :clock1:*Start Time:* {{ .StartsAt.Format "02.01.2006 15:04:05" }}
🔹🔹🔹🔹🔹🔹🔹
{{ end }}
• :globe_with_meridians:*External URL:* <{{ .ExternalURL }}|URL>
• :label:*CommonLabels:*
{{ range $key, $value := .CommonLabels }}
- {{$key}}: {{$value}}
{{ end }}
- name: 'null'
- name: Watchdog
route:
group_by:
- namespace
group_interval: 1m
group_wait: 30s
receiver: Default
repeat_interval: 2m
routes:
- matchers:
- alertname = Watchdog
receiver: Watchdog
- matchers:
- alertname = InfoInhibitor
receiver: 'null'
- receiver: Critical
matchers:
- severity = critical
And the message is something like that
🚨 [FIRING]: 2 UnmanagedRoutes
• 📋Summary: Route owned by an Ingress no longer managed
• 🔍Description: This alert fires when there is a Route owned by an unmanaged Ingress.
• :books:Namespace | Pod: corporate-banking-test / <>
• :link:Runbook: No URL
• ⏰Start Time: 08.12.2024 17:43:42
enter code here
🔹🔹🔹🔹🔹🔹🔹
enter code here
• 📋Summary: Route owned by an Ingress no longer managed
• 🔍Description: This alert fires when there is a Route owned by an unmanaged Ingress.
• :books:Namespace | Pod: corporate-banking-test / <>
• :link:Runbook: No URL
• ⏰Start Time: 08.12.2024 17:43:42
enter code here
🔹🔹🔹🔹🔹🔹🔹
enter code here
• 🌐External URL: URL
• 🏷️CommonLabels:
- alertname: UnmanagedRoutes
- container: route-controller-manager
- endpoint: https
- instance: 10.100.0.94:8443
- job: route-controller-manager
- namespace: corporate-banking-test
- openshift_io_alert_source: platform
- pod: route-controller-manager-64786f6984-j86hj
- prometheus: openshift-monitoring/k8s
- service: route-controller-manager
- severity: warning:globe_with_meridians:External URL: URL
And my question is – is there any better to handle these reports? Like for example sometimes its error that not containing pod, basically what I am asking where can I find a best-practice tips to handle this alerts to slack config