Checks an LLM app release against its cost and latency budget from the eval run logs – tokens per call, cost via your own price table (USD per million input/output tokens per model), cost per 1,000 requests and per successful task, p50/p95/p99 latency, error and timeout rates.
Checks an LLM app release against its cost and latency budget from the eval run logs - tokens per call, cost via your own price table (USD per million input/output tokens per model), cost per 1,000 requests and per successful task, p50/p95/p99 latency, error and timeout rates, increases against the production baseline, the slowest slice, and a sanity check that the eval's token mix matches production traces; a model without a price or calls without token usage are BLOCKED, never estimated silently. Use when someone asks "how much will the new model cost us", "is the candidate slower", "what's our cost per conversation", "set a latency budget for the AI feature", "did the prompt change blow up tokens", or as the eighth step of an LLM release eval. Do NOT use for load or capacity testing a deployed endpoint (use a load-testing tool such as k6 or Locust), for cloud bill analysis, or for provider price comparisons from memory.This is the skill's description: the text your AI reads to decide when the skill applies.
This is the skill's header. The step-by-step instructions are paid content and aren't shown here.
llm-cost-latency-budget/SKILL.md
---
name: llm-cost-latency-budget
description: Checks an LLM app release against its cost and latency budget from the eval run logs - tokens per call, cost via your own price table (USD per million input/output tokens per model), cost per 1,000 requests and per successful task, p50/p95/p99 latency, error and timeout rates, increases against the production baseline, the slowest slice, and a sanity check that the eval's token mix matches production traces; a model without a price or calls without token usage are BLOCKED, never estimated silently. Use when someone asks "how much will the new model cost us", "is the candidate slower", "what's our cost per conversation", "set a latency budget for the AI feature", "did the prompt change blow up tokens", or as the eighth step of an LLM release eval. Do NOT use for load or capacity testing a deployed endpoint (use a load-testing tool such as k6 or Locust), for cloud bill analysis, or for provider price comparisons from memory.
---
… the full instructions are part of a premium pack and load in your AI once you have access.