Diagnostic guide for active Prometheus cardinality problems — slow queries, OOMing Prometheus, high Grafana Cloud Active Series or DPM bills, "too many samples" ingest errors, series churn, or rapid memory growth.
grafana-prometheus-cardinality-troubleshooter/SKILL.md
---
name: prometheus-cardinality-troubleshooter
license: Apache-2.0
description: >
Diagnostic guide for active Prometheus cardinality problems — slow queries, OOMing
Prometheus, high Grafana Cloud Active Series or DPM bills, "too many samples" ingest
errors, series churn, or rapid memory growth. Walks through tsdb status endpoints,
per-metric and per-label drill-downs, common-culprit galleries, and remediation paths.
Use when the user is *currently experiencing* a cardinality fire. For preventing
cardinality issues at the source, route to prometheus-label-strategy. For post-ingest
aggregation, route to adaptive-metrics. For DPM-specific analysis, route to dpm-finder.
---
# Prometheus Cardinality Troubleshooter
You are an expert in diagnosing live Prometheus cardinality problems. When a user reports a Prometheus performance, memory, or cost issue that smells like cardinality, use this guide to triage systematically.
This skill is **diagnostic and operational**. For schema design and prevention, route to `prometheus-label-strategy`.
---
## Before You Remediate: The One Rule
Under pressure, the tempting move is to `labeldrop` the high-cardinality label at scrape time. **Do not.** You cannot remove, at scrape time, any label that makes a series unique — not `pod`, not `instance`, not anything that distinguishes one real series from another. It looks like it stops the bleeding; it actually **breaks the data**: