Starts a release eval of an LLM application – writes the .evals/eval-contract.json every later step obeys (how to call the app's baseline and candidate builds as a provider-agnostic command or allowlisted HTTP endpoint, the judge likewise, task type, slices, graders, quality thresholds.
Starts a release eval of an LLM application - writes the .evals/eval-contract.json every later step obeys (how to call the app's baseline and candidate builds as a provider-agnostic command or allowlisted HTTP endpoint, the judge likewise, task type, slices, graders, quality thresholds, p95 latency and cost-per-1k budgets with your own price table, sample sizes, red-team settings, expiring waivers), installs the shared scripts and example Anthropic/OpenAI adapters, and validates it all with a PASS/FAIL/BLOCKED gate. Use when someone asks "set up evals for our LLM app", "is the new prompt/model safe to ship", "build a regression gate for our chatbot", "evaluate our RAG assistant before release", "eval harness for our AI feature", or before any other skill in this pack. Do NOT use for general advice on eval methods without running anything (use llm-evaluation or wshobson-llm-evaluation), for writing or tuning prompts (use prompt-engineer, or the free AI Engineer Toolkit pack), for benchmarking a base model on public datasets (use orchestra-lm-evaluation-harness), or for runtime guardrails in production traffic (use orchestra-nemo-guardrails).This is the skill's description: the text your AI reads to decide when the skill applies.
This is the skill's header. The step-by-step instructions are paid content and aren't shown here.
eval-contract/SKILL.md
---
name: eval-contract
description: Starts a release eval of an LLM application - writes the .evals/eval-contract.json every later step obeys (how to call the app's baseline and candidate builds as a provider-agnostic command or allowlisted HTTP endpoint, the judge likewise, task type, slices, graders, quality thresholds, p95 latency and cost-per-1k budgets with your own price table, sample sizes, red-team settings, expiring waivers), installs the shared scripts and example Anthropic/OpenAI adapters, and validates it all with a PASS/FAIL/BLOCKED gate. Use when someone asks "set up evals for our LLM app", "is the new prompt/model safe to ship", "build a regression gate for our chatbot", "evaluate our RAG assistant before release", "eval harness for our AI feature", or before any other skill in this pack. Do NOT use for general advice on eval methods without running anything (use llm-evaluation or wshobson-llm-evaluation), for writing or tuning prompts (use prompt-engineer, or the free AI Engineer Toolkit pack), for benchmarking a base model on public datasets (use orchestra-lm-evaluation-harness), or for runtime guardrails in production traffic (use orchestra-nemo-guardrails).
---
… the full instructions are part of a premium pack and load in your AI once you have access.