Instruments systems with the three pillars - structured logs, RED/USE metrics, and distributed traces - correlated by one trace ID, with cardinality budgets and symptom-based alerts. Use when someone asks "how should I instrument this service", "what metrics should I emit", "why is my metrics bill exploding", "set up OpenTelemetry for my stack", or "we can't tell which service is slow". Do NOT use for tuning an existing noisy pager or alert thresholds - use alert-tuning instead; for defining SLO error-budget policy, use error-budget-policy; for the incident process when alerts fire, use sev-triage and postmortem-writer.
Click to play with sound.
---
name: Observability Stack
description: Instruments systems with the three pillars - structured logs, RED/USE metrics, and distributed traces - correlated by one trace ID, with cardinality budgets and symptom-based alerts. Use when someone asks "how should I instrument this service", "what metrics should I emit", "why is my metrics bill exploding", "set up OpenTelemetry for my stack", or "we can't tell which service is slow". Do NOT use for tuning an existing noisy pager or alert thresholds - use alert-tuning instead; for defining SLO error-budget policy, use error-budget-policy; for the incident process when alerts fire, use sev-triage and postmortem-writer.
---
# Observability Stack
Instrumentation exists to answer one question during an incident: what is wrong and why, from telemetry alone, without SSH-ing into boxes. The two expensive failure modes are opposite: under-instrumented systems where an outage is archaeology, and over-instrumented ones where a single high-cardinality label multiplies the metrics bill by 10,000x. This skill installs the three pillars with correlation between them and explicit cardinality budgets.
## The three pillars and what each is for
- Logs: discrete events with context - for debugging a specific request. Expensive per byte; highest detail.
- Metrics: pre-aggregated numbers over time - for dashboards and alerts. Cheap to query; no per-request detail.
- Traces: the path of one request across services - for locating which hop added the latency or threw the error.
The pillars are useless in isolation; the connective tissue is one correlation ID. A `trace_id` in every log line and exemplars on metrics let you pivot: alert fires (metric) → exemplar trace shows the slow hop (trace) → logs for that trace_id show the error (log). Instrument on OpenTelemetry so all three stay vendor-neutral.
## Operating procedure
### Step 1: Gather inputs
1. Service inventory and the 2-3 critical user paths (checkout, login, search). These get instrumented first; everything else waits.
2. Current pain - "we don't know when we're down" (start with metrics/alerts) vs "we know but can't find where" (start with traces).
3. Backend and budget - the metrics bill is a function of series count, so the cardinality budget in Step 3 is a real cost decision.
4. Traffic volume, which sets the trace sampling rate in Step 4.