Alerting
Alertmanager (part of the kube-prometheus-stack) routes Prometheus alerts to notification channels. This page covers configuring basic alerting for the platform.
What this page covers
- Alertmanager architecture
- Configuring a notification receiver (Discord, Slack, email)
- Writing a basic PrometheusRule
- Useful pre-built alert rules for the platform
Alertmanager receivers
Configure Alertmanager to send alerts to a Discord or Slack webhook:
# alertmanager.yaml (values for kube-prometheus-stack)
alertmanager:
config:
receivers:
- name: discord
discord_configs:
- webhook_url: https://discord.com/api/webhooks/...
route:
receiver: discord
Useful alert rules
The kube-prometheus-stack ships many default rules. Additional rules to add for the platform:
| Alert | Condition |
|---|---|
| Node memory pressure | node_memory_MemAvailable_bytes < 500MB |
| Pod CrashLooping | kube_pod_container_status_restarts_total > 5 in 10m |
| Disk filling | node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.1 |
| Certificate expiring | certmanager_certificate_expiration_timestamp_seconds - time() < 7 days |