Runs the baseline and candidate builds of an LLM app over the frozen test set (concurrency limit, retries, timeouts, repeats, cache keyed by input, app version and code) and decides whether the candidate regressed – grades every repeat, finds flaky cases whose verdict changes between repeats.
Runs the baseline and candidate builds of an LLM app over the frozen test set (concurrency limit, retries, timeouts, repeats, cache keyed by input, app version and code) and decides whether the candidate regressed - grades every repeat, finds flaky cases whose verdict changes between repeats, compares paired pass/fail with McNemar's exact test and a paired bootstrap CI on the difference, overall, per quality dimension and per slice, checks every threshold, and emits a GitHub Actions workflow that runs the whole gate on pull requests. Use when someone asks "did the new prompt make things worse", "regression test our LLM app in CI", "is this model swap safe", "compare the candidate to production on our evals", "is that drop significant or noise", or as the sixth step of an LLM release eval. Do NOT use for online A/B tests on live traffic, for evaluating a base model on public benchmarks (use orchestra-lm-evaluation-harness), or for unit tests of deterministic code.This is the skill's description: the text your AI reads to decide when the skill applies.
This is the skill's header. The step-by-step instructions are paid content and aren't shown here.
eval-regression-gate/SKILL.md
---
name: eval-regression-gate
description: Runs the baseline and candidate builds of an LLM app over the frozen test set (concurrency limit, retries, timeouts, repeats, cache keyed by input, app version and code) and decides whether the candidate regressed - grades every repeat, finds flaky cases whose verdict changes between repeats, compares paired pass/fail with McNemar's exact test and a paired bootstrap CI on the difference, overall, per quality dimension and per slice, checks every threshold, and emits a GitHub Actions workflow that runs the whole gate on pull requests. Use when someone asks "did the new prompt make things worse", "regression test our LLM app in CI", "is this model swap safe", "compare the candidate to production on our evals", "is that drop significant or noise", or as the sixth step of an LLM release eval. Do NOT use for online A/B tests on live traffic, for evaluating a base model on public benchmarks (use orchestra-lm-evaluation-harness), or for unit tests of deterministic code.
---
… the full instructions are part of a premium pack and load in your AI once you have access.