Calibrates an LLM-as-judge against human labels before its verdicts can count toward a release – runs the judge on human-labeled dev outputs and reports agreement, Cohen's kappa, the confusion matrix, length bias (does it pass long answers humans failed?) and position bias (pairwise, both orders.
Calibrates an LLM-as-judge against human labels before its verdicts can count toward a release - runs the judge on human-labeled dev outputs and reports agreement, Cohen's kappa, the confusion matrix, length bias (does it pass long answers humans failed?) and position bias (pairwise, both orders, against the human-confirmed reference); BLOCKED without enough labels, FAIL below the contract's kappa or above its bias limits, and every later gate refuses judge verdicts unless this gate passed for the same judge model, prompt and code. Use when someone asks "can we trust our LLM judge", "calibrate the grader model", "is GPT/Claude-as-judge biased", "does the judge agree with our reviewers", "check the judge for position bias", or as the fifth step of an LLM release eval. Do NOT use for choosing a judge model by reputation, for general reading on judge methods (use llm-evaluation), or for grading where a deterministic rule works (use eval-graders).This is the skill's description: the text your AI reads to decide when the skill applies.
This is the skill's header. The step-by-step instructions are paid content and aren't shown here.
judge-calibration/SKILL.md
---
name: judge-calibration
description: Calibrates an LLM-as-judge against human labels before its verdicts can count toward a release - runs the judge on human-labeled dev outputs and reports agreement, Cohen's kappa, the confusion matrix, length bias (does it pass long answers humans failed?) and position bias (pairwise, both orders, against the human-confirmed reference); BLOCKED without enough labels, FAIL below the contract's kappa or above its bias limits, and every later gate refuses judge verdicts unless this gate passed for the same judge model, prompt and code. Use when someone asks "can we trust our LLM judge", "calibrate the grader model", "is GPT/Claude-as-judge biased", "does the judge agree with our reviewers", "check the judge for position bias", or as the fifth step of an LLM release eval. Do NOT use for choosing a judge model by reputation, for general reading on judge methods (use llm-evaluation), or for grading where a deterministic rule works (use eval-graders).
---
… the full instructions are part of a premium pack and load in your AI once you have access.