VictoriaMetrics

Сравнение ролей в стэках VM и Prometheus:

Роль	Prometheus-стек	VM-стек
Скрейпинг метрик	Prometheus	VMAgent
Выполнение recording/alert rules	Prometheus	VMAlert
Хранение данных (single node)	Prometheus TSDB	VMSingle
Хранение данных (кластер)	Thanos / Cortex	VMCluster (vminsert + vmselect + vmstorage)
Управление CRD и компонентами	Prometheus Operator	Victoria Metrics Operator
Алертинг (роутинг уведомлений)	Alertmanager	Alertmanager (или VMAlertmanager)
Проксирование / балансировка запросов	—	VMAuth
Дедупликация и долгосрочное хранение	Thanos Compactor	VMBackup / VMRestore
Federation / агрегация нескольких кластеров	Thanos Query	VMSelect с несколькими источниками
Буферизация при недоступности remote	—	VMAgent stateful queue (PVC)
CRD для scrape targets	ServiceMonitor / PodMonitor / PrometheusRule	VMServiceScrape / VMPodScrape / VMRule
Push метрик (вместо pull)	Pushgateway	VMAgent /api/v1/import или Pushgateway
Аномалии / ML	—	VMAnomaly

docker

docker-compose.yml

version: '3.7'

services:
  victoriametrics:
    image: victoriametrics/victoria-metrics:v1.75.0
    command:
      - '--retentionPeriod=2'
    volumes:
    - ${PRJPATH-/data/docker/victoriametrics}/victoria-metrics-data:/victoria-metrics-data:rw
    ports:
      - 8428:8428
    restart: on-failure:3
    deploy:
      replicas: 1
      resources:
        limits:
          memory: 2g
      restart_policy:
        condition: any
    healthcheck:
      test: wget -nv --spider http://localhost:8428 > /dev/null || exit 1
      interval: 60s
      timeout: 5s
      retries: 3
      start_period: 15s

Запросы

Показать все значения label (cluster):

curl "http://docker21:8428/api/v1/label/cluster/values"

Показать все метрики с фильтром по label (cluster=Infrastructure):

curl 'http://vm:8428/api/v1/query?query=up&extra_label=cluster=Infrastructure'

vmalert

Rules

Создать rules из чарта:

helm template vm-stack vm/victoria-metrics-k8s-stack   --version 0.79.0   --set vmalert.enabled=true   --namespace monitoring | yq e 'select(.kind == "VMRule")' | yq e '.spec' - > k8s-rules.yaml

Проверить:

cat k8s-rules.yaml | yq e '.groups[].name'

Пример вывода:

alertmanager.rules
---
etcd
---
general.rules
---
k8s.rules.container_cpu_limits
---
k8s.rules.container_cpu_requests
---
k8s.rules.container_cpu_usage_seconds_total
---
k8s.rules.container_memory_cache
---
k8s.rules.container_memory_limits
---
k8s.rules.container_memory_requests
---
k8s.rules.container_memory_rss
---
k8s.rules.container_memory_swap
---
k8s.rules.container_memory_working_set_bytes
---
k8s.rules.pod_owner
---
kube-apiserver-availability.rules
---
kube-apiserver-burnrate.rules
---
kube-apiserver-histogram.rules
---
kube-apiserver-slos
---
kube-prometheus-node-recording.rules
---
kube-scheduler.rules
---
kube-state-metrics
---
kubelet.rules
---
kubernetes-apps
---
kubernetes-resources
---
kubernetes-storage
---
kubernetes-system-apiserver
---
kubernetes-system-controller-manager
---
kubernetes-system-kubelet
---
kubernetes-system-scheduler
---
kubernetes-system
---
node-exporter.rules
---
node-exporter
---
node-network
---
node.rules
---
vm-health
---
vmagent
---
vmalert
---
vmcluster
---
vmoperator
---
vmsingle

Запустить vmalert для кластера Infrastructure:

docker-compose.yaml

services:
  vmalert-infrastructure:
    image: victoriametrics/vmalert:${TAG-latest}
    command:
      - -rule=/rules/k8s-rules.yaml
      - -datasource.url=${VM_URL-http://victoriametrics:8428}?extra_label=cluster=Infrastructure
      - -remoteWrite.url=${VM_URL-http://victoriametrics:8428}
      - -remoteRead.url=${VM_URL-http://victoriametrics:8428}
      - -evaluationInterval=30s
      - -external.label=cluster=Infrastructure
      - -notifier.blackhole
      - -loggerFormat=json
    volumes:
      - ./k8s-rules.yaml:/rules/k8s-rules.yaml