feat: add postgres/gitea/blackbox alerts and more node alerts
Some checks failed
terraform-dev / validate (push) Successful in 6s
terraform-dev / plan (push) Failing after 11s
terraform-dev / apply (push) Has been skipped

This commit is contained in:
Alexandr
2026-04-03 11:34:08 +03:00
parent 03dfa99400
commit 5af763ebb1
18 changed files with 393 additions and 1 deletions

View File

@ -0,0 +1,19 @@
name: "DEV ADIBROV - High Disk IO"
expression: |
rate(node_disk_io_time_seconds_total{job="node_exporter", device!~"dm-.*"}[5m]) * 100
threshold: 90
for: "10m"
condition_type: "gt"
need_reduce: true
reducer_type: "max"
no_data_state: "OK"
exec_err_state: "Error"
labels:
service: "system"
severity: "warning"
team: "infra"
summary: |
Высокая нагрузка на диск: {{ printf "%.0f" $values.B.Value }}% IO на {{ $labels.instance }} ({{ $labels.device }})
description: |
Устройство {{ $labels.device }} на {{ $labels.instance }} занято на {{ printf "%.0f" $values.B.Value }}% в течение 10 минут.
Высокая нагрузка на IO может замедлять все сервисы на хосте.

View File

@ -0,0 +1,22 @@
name: "DEV ADIBROV - Disk Space Low"
expression: |
(
1 - node_filesystem_avail_bytes{job="node_exporter", fstype=~"ext.|xfs|zfs", mountpoint!~"^(/sys|/proc|/dev|/run|/boot).*"}
/ node_filesystem_size_bytes{job="node_exporter", fstype=~"ext.|xfs|zfs", mountpoint!~"^(/sys|/proc|/dev|/run|/boot).*"}
) * 100
threshold: 85
for: "5m"
condition_type: "gt"
need_reduce: true
reducer_type: "max"
no_data_state: "OK"
exec_err_state: "Error"
labels:
service: "system"
severity: "warning"
team: "infra"
summary: |
Мало места на диске: {{ printf "%.0f" $values.B.Value }}% занято на {{ $labels.mountpoint }} ({{ $labels.instance }})
description: |
На {{ $labels.instance }} точка монтирования {{ $labels.mountpoint }} заполнена на {{ printf "%.0f" $values.B.Value }}%.
При достижении 100% запись на диск станет невозможной — сервисы начнут падать.

View File

@ -0,0 +1,21 @@
name: "DEV ADIBROV - High CPU Usage"
expression: |
(
1 - avg by(instance) (rate(node_cpu_seconds_total{job="node_exporter", mode="idle"}[5m]))
) * 100
threshold: 85
for: "5m"
condition_type: "gt"
need_reduce: true
reducer_type: "max"
no_data_state: "OK"
exec_err_state: "Error"
labels:
service: "system"
severity: "warning"
team: "infra"
summary: |
Высокая нагрузка CPU: {{ printf "%.0f" $values.B.Value }}% на {{ $labels.instance }}
description: |
Использование CPU на {{ $labels.instance }} превышает 85% на протяжении 5 минут.
Это может привести к деградации производительности всех сервисов на хосте.

View File

@ -0,0 +1,19 @@
name: "DEV ADIBROV - High System Load"
expression: |
node_load5{job="node_exporter"} / on(instance) machine_cpu_cores{job="cadvisor"}
threshold: 2
for: "10m"
condition_type: "gt"
need_reduce: true
reducer_type: "max"
no_data_state: "OK"
exec_err_state: "Error"
labels:
service: "system"
severity: "warning"
team: "infra"
summary: |
Высокий LA на {{ $labels.instance }}: {{ printf "%.2f" $values.B.Value }} на ядро
description: |
Средняя нагрузка (load average 5m) на {{ $labels.instance }} превышает 2x количество ядер CPU.
Система перегружена — процессы ждут в очереди на выполнение.

View File

@ -0,0 +1,21 @@
name: "DEV ADIBROV - High Memory Usage"
expression: |
(
1 - (node_memory_MemAvailable_bytes{job="node_exporter"} / node_memory_MemTotal_bytes{job="node_exporter"})
) * 100
threshold: 90
for: "5m"
condition_type: "gt"
need_reduce: true
reducer_type: "max"
no_data_state: "OK"
exec_err_state: "Error"
labels:
service: "system"
severity: "warning"
team: "infra"
summary: |
Высокое использование памяти: {{ printf "%.0f" $values.B.Value }}% на {{ $labels.instance }}
description: |
На {{ $labels.instance }} занято {{ printf "%.0f" $values.B.Value }}% оперативной памяти.
При исчерпании памяти ядро начнёт убивать процессы (OOM killer).

View File

@ -0,0 +1,19 @@
name: "DEV ADIBROV - Node Instance Down"
expression: |
up{job="node_exporter"}
threshold: 1
for: "2m"
condition_type: "lt"
need_reduce: true
reducer_type: "min"
no_data_state: "Alerting"
exec_err_state: "Alerting"
labels:
service: "system"
severity: "critical"
team: "infra"
summary: |
Хост недоступен: {{ $labels.instance }}
description: |
Node exporter на {{ $labels.instance }} не отвечает уже более 2 минут.
Хост может быть недоступен или упал node_exporter.