Data Engineering Pipeline

Designs ClickHouse schemas and queries for fast analytics - MergeTree engine selection, ORDER BY key design, partitioning, materialized views, and projections. Use when someone asks "why is my ClickHouse query slow", "how should I order my sorting key", "should I use a materialized view or projection", "how do I deduplicate events", or is modeling an event or metrics table for OLAP. Do NOT use for tuning row-store OLTP databases like Postgres or MySQL - use sql-query-optimizer instead; for general relational schema design use database-schema; for reading query plans on traditional databases use explain-plan-reader; for the streaming ingestion side use kafka-pipelines.

Designs layered data quality checks across completeness, validity, consistency, uniqueness, timeliness, and accuracy, with severity tiers, freshness SLAs, and anomaly baselines wired into dbt and CI. Use when someone asks "how do I stop bad data reaching dashboards", "set up dbt tests for this model", "our pipeline loaded duplicate rows again", "what data quality checks should this table have", or "how fresh does this source need to be". Do NOT use for detecting distribution drift in ML features and predictions - use data-drift-monitor instead; for one-off exploration and profiling of a new dataset - use eda-playbook instead; for infrastructure and application telemetry - use observability-stack instead; for removing personal data from datasets - use pii-scrubber instead.

Decomposes and forecasts time-indexed data with STL, ARIMA/SARIMA, and Prophet, validated by time-ordered backtests against a seasonal-naive baseline. Use when someone asks "forecast next quarter's demand", "is this series seasonal", "why is my ARIMA forecast flat", "how do I backtest a forecast", or has any metric indexed by time that needs prediction or decomposition. Do NOT use for explaining what a trend means for strategy - use trend-analysis instead; for estimating the causal impact of an intervention on a series use causal-inference; for translating a forecast into an ARR or MRR plan use revenue-modeling.

Designs ML features with leakage-safe pipelines, correct categorical encoding by cardinality, numeric transforms, and validation that a feature earns its place. Use when someone asks "how should I encode this high-cardinality column", "why does my model score great offline and fail in production", "what features should I build from this table", or "should I scale these inputs". Do NOT use for building the serving infrastructure that stores and versions features across models - use feature-store-design instead; for detecting when live feature distributions shift after deployment - use data-drift-monitor instead; for initial dataset exploration and profiling - use eda-playbook instead.

Diagnoses slow SQL from the execution plan, names the cost driver, and delivers the rewritten query plus index DDL with the expected plan change and write-side cost stated. The general entry point for slow-query work that routes deep cases to specialist skills. Use when someone asks "why is this query slow", "can you optimize this SQL", "what index do I need", "this endpoint got slow and it's the database", or pastes an EXPLAIN output. Do NOT use for deep index strategy across a whole schema - use index-advisor instead; for pure query-shape rewrites when indexes are already right - use query-rewriter instead; for ORM-driven repeated-query patterns - use n-plus-one-hunter instead; for line-by-line EXPLAIN interpretation training - use explain-plan-reader instead; for table partitioning decisions - use partition-planner instead.

Designs MongoDB schemas, indexes, and aggregation pipelines that perform - embed-vs-reference decision rules, the 16MB document limit, ESR compound-index ordering, explain-plan verification, and operational settings for replica sets and sharding. Use when someone asks "should I embed or reference this", "why is my Mongo query slow", "design a MongoDB schema for X", "how do I structure this aggregation pipeline", or "what should my shard key be". Do NOT use for relational or SQL schema design - use database-schema instead; for tuning SQL queries use sql-query-optimizer; for Postgres-on-Supabase work use supabase-expert.