AI Engineer Toolkit

Builds MCP servers with proper tool definitions, error handling, and auth patterns.

Builds evaluation harnesses for LLM products - golden datasets, deterministic checks, calibrated LLM-as-judge rubrics, and CI regression gates that turn "seems good" into a tracked number. Use when someone asks "how do I know this prompt change didn't break anything", "set up evals for my RAG pipeline", "is LLM-as-judge reliable", "why did quality drop after the model swap", or is shipping an LLM feature with no quality measurement. Do NOT use for classical ML model reporting (precision/recall, ROC curves, confusion matrices on trained classifiers) - use model-evaluation-report instead; do NOT use for analyzing online A/B experiments - use ab-test-analyzer instead.

Designs reliable multi-agent LLM systems - choosing a topology, writing handoff contracts between agents, deciding what runs in parallel vs series, and setting context, retry, and cost budgets that stop runaway loops. Use when someone asks "should I split this into multiple agents", "my agents keep looping", "how do I pass context between agents", "orchestrator vs pipeline vs router", or is architecting an agent workflow. Do NOT use for tuning a single prompt - use prompt-engineer instead; do NOT use for measuring agent output quality - use llm-evaluation instead; for making a product agent-operable end to end, use build-on-agent-native.

Runs a thorough multi-source research process - decompose the question, gather independent sources, cross-check, synthesize - and delivers a structured, cited brief with confidence levels and gaps. Use when someone asks "research this topic in depth", "give me everything on X", "what does the evidence say about Y", or needs a broad question answered from secondary sources with citations. Do NOT use for designing a study that collects new data - use primary-research instead; for distilling an already-gathered pile of 20+ sources into themes, use research-synthesis; for verifying one discrete claim, use fact-checker.

Designs ML features with leakage-safe pipelines, correct categorical encoding by cardinality, numeric transforms, and validation that a feature earns its place. Use when someone asks "how should I encode this high-cardinality column", "why does my model score great offline and fail in production", "what features should I build from this table", or "should I scale these inputs". Do NOT use for building the serving infrastructure that stores and versions features across models - use feature-store-design instead; for detecting when live feature distributions shift after deployment - use data-drift-monitor instead; for initial dataset exploration and profiling - use eda-playbook instead.

Designs typed, observable, recoverable error handling - an explicit taxonomy of retryable vs terminal failures, Result types at boundaries, single-point logging, and retry policies with real backoff numbers. Use when someone asks "should I retry this error", "how do I structure error types", "why are my logs full of duplicate stack traces", "where do I put the try/catch", or is designing failure paths for an API, job, or client. Do NOT use for building the circuit breaker component itself - use circuit-breaker-builder instead; do NOT use for handling 429s against third-party APIs - use rate-limit-handler instead; do NOT use for writing user-facing error copy - use error-message-writer instead.