Produces a rigorous, honest evaluation report for an ML model – real baselines, business-matched metrics with confidence intervals, slice-level error analysis, calibration checks, and a go/no-go recommendation.
Produces a rigorous, honest evaluation report for an ML model - real baselines, business-matched metrics with confidence intervals, slice-level error analysis, calibration checks, and a go/no-go recommendation. Use when someone asks "is this model good enough to ship", "write up the eval for this model", "did we actually beat the old model", or before any model promotion, stakeholder demo, or retraining decision. Do NOT use for evaluating LLM or prompt outputs - use llm-evaluation instead; for analyzing randomized product experiments use ab-test-analyzer; for documenting a shipped model for consumers use model-card-writer.This is the skill's description: the text your AI reads to decide when the skill applies.
model-evaluation-report/SKILL.md
---
name: Model Evaluation Report
description: Produces a rigorous, honest evaluation report for an ML model - real baselines, business-matched metrics with confidence intervals, slice-level error analysis, calibration checks, and a go/no-go recommendation. Use when someone asks "is this model good enough to ship", "write up the eval for this model", "did we actually beat the old model", or before any model promotion, stakeholder demo, or retraining decision. Do NOT use for evaluating LLM or prompt outputs - use llm-evaluation instead; for analyzing randomized product experiments use ab-test-analyzer; for documenting a shipped model for consumers use model-card-writer.
---
# Model Evaluation Report
A model that beats a naive baseline on aggregate accuracy can still be worse than useless in production. Honest evaluation means choosing the right metric before looking at results, comparing against baselines that actually matter, and finding the slices where the model fails - because those slices are where the incident reports will come from.
## Operating procedure
Order matters: the primary metric and baselines are chosen before results are examined, or the report is p-hacked by construction.
### Step 1: Gather inputs
Collect from the requester before computing anything; label unknowns as guesses.
1. Task type: classification, regression, or ranking.
2. The business decision the model drives, and the asymmetry of errors (is a false positive or a false negative more expensive, and roughly by how much).
3. The evaluation dataset: source, size, and confirmation it is disjoint from training data and from any data used for model selection.
4. What the model replaces: prior production model, rule-based system, or nothing.
5. Class balance (classification) or target distribution (regression).
### Step 2: Fix the primary metric before looking at results
… load the full skill through Skill MeClick to play with sound.