Builds an LLM eval set from real production traces – imports JSONL, CSV, chat-message logs or OpenTelemetry (OTLP JSON) span exports in documented shapes (anything else is BLOCKED, not guessed), scrubs emails, phones, Luhn-valid card numbers, SSNs and API keys before anything is stored.
Builds an LLM eval set from real production traces - imports JSONL, CSV, chat-message logs or OpenTelemetry (OTLP JSON) span exports in documented shapes (anything else is BLOCKED, not guessed), scrubs emails, phones, Luhn-valid card numbers, SSNs and API keys before anything is stored, drops exact and near-duplicate traces (MinHash on word shingles), keeps out traces that copy the app's own prompt examples, samples a stratified dev/test split by intent, route and length, and freezes the test set with a hash manifest; the gate checks drift from the manifest, leakage into prompt examples or training data, coverage of every declared slice and PII remaining. Use when someone asks "turn our traces into an eval set", "build a test set from production logs", "we have Langfuse/OTel/JSONL exports, make evals from them", "is our eval set leaking into the prompt", "our test set keeps changing", or as the second step of an LLM release eval. Do NOT use for synthetic benchmark generation with no real traffic (use llm-evaluation for test-design advice), for a general PII/DLP scan of a data lake, or for model training data curation.This is the skill's description: the text your AI reads to decide when the skill applies.
This is the skill's header. The step-by-step instructions are paid content and aren't shown here.
trace-eval-set/SKILL.md
---
name: trace-eval-set
description: Builds an LLM eval set from real production traces - imports JSONL, CSV, chat-message logs or OpenTelemetry (OTLP JSON) span exports in documented shapes (anything else is BLOCKED, not guessed), scrubs emails, phones, Luhn-valid card numbers, SSNs and API keys before anything is stored, drops exact and near-duplicate traces (MinHash on word shingles), keeps out traces that copy the app's own prompt examples, samples a stratified dev/test split by intent, route and length, and freezes the test set with a hash manifest; the gate checks drift from the manifest, leakage into prompt examples or training data, coverage of every declared slice and PII remaining. Use when someone asks "turn our traces into an eval set", "build a test set from production logs", "we have Langfuse/OTel/JSONL exports, make evals from them", "is our eval set leaking into the prompt", "our test set keeps changing", or as the second step of an LLM release eval. Do NOT use for synthetic benchmark generation with no real traffic (use llm-evaluation for test-design advice), for a general PII/DLP scan of a data lake, or for model training data curation.
---
… the full instructions are part of a premium pack and load in your AI once you have access.