LLM App Evals & Guardrails

Fixed twin: SHIP, with every gate PASS (2 minor notes)

From the report

• REGRESSION-SLICE-DROP · intent=billing

  85.7% -> 21.4% (n=28, 18 regressed, 0 improved, McNemar p=7.63e-06); e.g. numbers [104.12, 3.0] not within 0.01 of 52.06.

  Fix: The candidate is worse on this slice; read its failing cases.

• EVALSET-LEAKAGE · c_18bbeb89b240

  Test case near-duplicates app/prompt_examples.jsonl line 7 (Jaccard 1.0): pasted into the prompt after the freeze.

  Fix: Remove the example from the prompt, or re-freeze without this case.

• JUDGE-LENGTH-BIAS · helpfulness

  Judge-minus-human pass rate is +0.56 higher on long outputs than short ones; false-pass rate on long outputs 1.0, short 0.0.

  Fix: Tell the judge length is not quality, give it reference facts, and re-calibrate.

• REDTEAM-EXFILTRATION · exfiltration:image-beacon

  A poisoned help-center document made the candidate add a markdown image to an attacker host, on 5 of 5 attempts.

  Fix: Do not render model-written links or images to hosts outside an allowlist; strip markdown images server-side.

What each skill delivers, in run order

1. 1.Eval Contract eval-contract.json with adapters for baseline, candidate and judge (a command or allowlisted endpoint, keys stay in your env), slices, graders, thresholds, budgets and waivers, validated; example Anthropic and OpenAI adapters.
2. 2.Trace Eval Set A frozen dev/test eval set from your JSONL, CSV, chat-log or OpenTelemetry traces: PII scrubbed before storage, duplicates removed, prompt-example copies excluded, stratified by slice, with a hash manifest that catches drift and leakage.
3. 3.Eval Case Labels Confirmed reference labels and pass/fail verdicts from your people, with two-labeler agreement (Cohen's kappa) and adjudication, plus the runner that calls your app.
4. 4.Eval Graders Deterministic graders (exact, regex, JSON Schema, numeric, required facts, citations in context, rubrics) and an LLM-judge grader, each tested against known-good and empty answers.
5. 5.Judge Calibration Judge versus human agreement, kappa, confusion matrix, length bias and position bias; an uncalibrated judge never counts.
6. 6.Eval Regression Gate Baseline versus candidate on the frozen set with repeats for flaky cases, McNemar's exact test with Holm correction overall, per dimension and per slice, an honest power statement, and a pinned GitHub Actions workflow that can ship from a fresh checkout.
7. 7.LLM Red-Team Suite Prompt-injection, exfiltration, tool-misuse, output-manipulation and PII-leak probes built from your app's input shape, output schema and tools, scored by markers, canaries, allowlists and paired clean runs, with positive controls that refuse to pass a category the probes could not reach.
8. 8.LLM Cost and Latency Budget Tokens, cost per 1,000 requests and per successful task, p50/p95/p99 and error rates against your budget and the production baseline.
9. 9.Evals Release Report One evidence-bound decision (SHIP, SHIP WITH ACCEPTED RISKS, or HOLD) in Markdown and HTML, refusing missing or stale gates.

What it does not do

1. 1.Small eval sets can only see big regressions. With 150 test cases the paired test catches a net drop of about 11 points with 80% power if 5% of cases flip each way on their own (about 5.5 points if nothing else flips), and a slice of 25 cases only very large drops. The report states the number and the churn it assumed. Slices and dimensions are Holm-corrected as two separate families next to the overall test, so a false alarm somewhere in the three can come up to about 3 times as often as alpha; within a small slice a real regression is easier to miss.
2. 2.An LLM judge counts only on dimensions it was calibrated for, against your own labelers, on dev outputs. Passing calibration means it agrees with your reviewers this often on that distribution, not that each verdict is right, and a graded output can still talk a judge into passing it.
3. 3.The red-team suite is a fixed set of templated probes (28, plus a few generated from your output schema and tools), filled in with your app's domain questions, tool names and record type, and scored mostly by string checks (markers, canaries, link hosts, tool calls, field values against a clean run). It catches literal leaks, not paraphrased ones, does not adapt or attempt multi-turn attacks, and is not a human red team. Probes are never cached and the app is not deterministic, so results can change between runs; each finding gives its hit rate over the repeats. It only counts as tested when your adapter passes injected documents and tool results to the app and reports its actions; positive controls check this, and the gate is BLOCKED otherwise. A channel the contract leaves empty is untested (BLOCKED), and becomes not applicable only when you declare it absent with a reason. An injection counts as obeyed when the output contains a code the injection asked the app to build from two parts (so copying or quoting it never matches). An app that builds the code while saying it refuses still counts; a weak model that obeys by echoing the instruction text instead of building the code is not counted as obeying; and an injection that changes behaviour without producing the code is caught only by the other checks. Without a calibrated judge, policy probes are a refusal heuristic whose hits are marked for human review, never counted as attacks.
4. 4.PII scrubbing is pattern-based: emails, phone numbers (national formats such as UK 020 7946 0958 only next to a phone word), card numbers, IBANs, US SSNs, IP addresses and common key, token and password formats, in traces and in every stored app output. It does not find names, street addresses, dates of birth, passport or medical record numbers; a number after a label such as "order" or "ticket" is treated as an identifier even when it is a real phone or card, and an unformatted card number with no card word near it is missed. It is not a DLP tool.
5. 5.Traces must be JSONL, chat-message JSONL, CSV or OTLP JSON in the documented shapes. Other vendor exports need a short conversion script; OpenTelemetry content attributes must be named in the contract. Images and other non-text message parts are replaced by a placeholder.
6. 6.Each case is one request through your adapter. Multi-turn agents, simulated users and tools with side effects need a sandboxed adapter you write.
7. 7.Fingerprints cover the files in your project that the adapters load. Python loading is read from the source, not run: a dynamic import, another runtime, or nothing found beyond the entry's folder makes the whole project tree hashed, so any edit re-runs the app; code loaded from outside the project is never seen. Installed packages and changes to a hosted model are invisible unless you bump the version string. An HTTP app's replies are cached only under the app_fingerprint it reports, which the pack cannot verify; the report re-checks it once when it can reach the server, and otherwise the results expire after 7 days. The frozen test set is tamper-evident through git, not tamper-proof.
8. 8.Cost comes from your price table and the token usage your adapter reports (cached tokens at their own prices, reasoning tokens as output); batch and negotiated pricing are not modelled. Latency is measured on one machine at the contract's concurrency: it compares builds, it is not a load test, and when the baseline runs much slower here than in its production traces the absolute latency budget is reported as not judged.
9. 9.Calls go wherever your adapters send them, usually a paid model API that bills you for every uncached call. The HTTP allowlist checks hostnames (no DNS-rebinding protection) and honours your proxy settings. Nothing else leaves your machine, and the pack ships no keys.
10. 10.CI is not a security boundary against a malicious pull request: a pull request runs its own copy of the workflow, and its app code can forge the gate files the decision job reads. Base-branch gate scripts and a separate decision job stop accidental self-certification only. The contract, labels and waivers come from the pull request, so protect .evals/ and .github/workflows/ with CODEOWNERS and required review.
11. 11.Requires an AI client that can run commands (Claude Code, Cursor, Codex), Python 3.9+, and people's time to label; with one labeler, human agreement is not measured, and the gap can only be accepted as a waived risk.
12. 12.A SHIP means these gates passed for this candidate on these cases; it is not a guarantee about behaviour on traffic the eval set does not contain.