Defines and tests the graders that score an LLM app's outputs – deterministic graders (exact, normalized, regex, JSON Schema validity with a stated stdlib subset, numeric tolerance, required and forbidden phrases, citations-in-retrieved-context for RAG.
Defines and tests the graders that score an LLM app's outputs - deterministic graders (exact, normalized, regex, JSON Schema validity with a stated stdlib subset, numeric tolerance, required and forbidden phrases, citations-in-retrieved-context for RAG, weighted rubric checklists) and an LLM-judge grader through the judge adapter with a strict reply schema and both-orders pairwise judging; the gate grades the graders - unsupported schema keywords, bad regexes, references they need but the labels lack, graders that fail a human-confirmed correct answer, and graders that pass an empty output. Use when someone asks "how do we grade our LLM outputs", "write evals for JSON output", "check the citations are grounded", "is our grader too strict", "set up an LLM-as-judge", or as the fourth step of an LLM release eval. Do NOT use to pick metrics for a research benchmark (BLEU, ROUGE, BERTScore; use llm-evaluation or wshobson-llm-evaluation), or to grade free-form creative writing where no reference exists.This is the skill's description: the text your AI reads to decide when the skill applies.
This is the skill's header. The step-by-step instructions are paid content and aren't shown here.
eval-graders/SKILL.md
---
name: eval-graders
description: Defines and tests the graders that score an LLM app's outputs - deterministic graders (exact, normalized, regex, JSON Schema validity with a stated stdlib subset, numeric tolerance, required and forbidden phrases, citations-in-retrieved-context for RAG, weighted rubric checklists) and an LLM-judge grader through the judge adapter with a strict reply schema and both-orders pairwise judging; the gate grades the graders - unsupported schema keywords, bad regexes, references they need but the labels lack, graders that fail a human-confirmed correct answer, and graders that pass an empty output. Use when someone asks "how do we grade our LLM outputs", "write evals for JSON output", "check the citations are grounded", "is our grader too strict", "set up an LLM-as-judge", or as the fourth step of an LLM release eval. Do NOT use to pick metrics for a research benchmark (BLEU, ROUGE, BERTScore; use llm-evaluation or wshobson-llm-evaluation), or to grade free-form creative writing where no reference exists.
---
… the full instructions are part of a premium pack and load in your AI once you have access.