Turns frozen eval cases into graded references and human verdicts – writes labeling files people fill (expected facts, amounts, tool calls, escalation, a known-good reference output; pass/fail verdicts on real model outputs for judge calibration), counts only rows a person confirmed.
Turns frozen eval cases into graded references and human verdicts - writes labeling files people fill (expected facts, amounts, tool calls, escalation, a known-good reference output; pass/fail verdicts on real model outputs for judge calibration), counts only rows a person confirmed, merges two labelers with an adjudication file, and measures inter-annotator agreement (Cohen's kappa on verdicts, agreement on references); the gate fails on unresolved disagreements, low coverage of the test set, low human agreement or PII in labels. Use when someone asks "label our eval cases", "write expected answers for the test set", "set up human grading", "do our labelers agree", "we need ground truth for the judge", or as the third step of an LLM release eval. Do NOT use for labeling training data at scale (use a labeling platform), for designing a rubric from scratch without labeling (use llm-evaluation), or for crowd-sourcing workflows.This is the skill's description: the text your AI reads to decide when the skill applies.
This is the skill's header. The step-by-step instructions are paid content and aren't shown here.
eval-case-labels/SKILL.md
---
name: eval-case-labels
description: Turns frozen eval cases into graded references and human verdicts - writes labeling files people fill (expected facts, amounts, tool calls, escalation, a known-good reference output; pass/fail verdicts on real model outputs for judge calibration), counts only rows a person confirmed, merges two labelers with an adjudication file, and measures inter-annotator agreement (Cohen's kappa on verdicts, agreement on references); the gate fails on unresolved disagreements, low coverage of the test set, low human agreement or PII in labels. Use when someone asks "label our eval cases", "write expected answers for the test set", "set up human grading", "do our labelers agree", "we need ground truth for the judge", or as the third step of an LLM release eval. Do NOT use for labeling training data at scale (use a labeling platform), for designing a rubric from scratch without labeling (use llm-evaluation), or for crowd-sourcing workflows.
---
… the full instructions are part of a premium pack and load in your AI once you have access.