Data engineers shipping production pipelines: ingest, transform, serve & trust the data.
Stand up data pipelines you can run in production and trust. Reach for this when you're moving data end to end - streaming ingestion off Kafka, batch transforms in Spark, an analytics store in ClickHouse - and need quality checks and query tuning so the numbers downstream are correct and fast. Built for the engineer who owns the pipeline, not the analyst querying its output.
Click to play with sound.
Arranged in the author's recommended order. Walk through them in sequence, or open any one on its own.
Designs Kafka streaming pipelines end to end – partition-count sizing math, key choice, consumer-group sizing, offset strategy, delivery semantics, poll tuning, and dead-letter handling – with production configs.
Writes and tunes PySpark jobs – join strategy and broadcast size limits, shuffle-partition sizing, skew diagnosis and salting, UDF avoidance, caching, and output file layout – with concrete size and skew thresholds.
Designs ClickHouse schemas and queries for fast analytics – MergeTree engine selection, ORDER BY key design, partitioning, materialized views, and projections.
Designs layered data quality checks across completeness, validity, consistency, uniqueness, timeliness, and accuracy, with severity tiers, freshness SLAs, and anomaly baselines wired into dbt and CI.
Decomposes and forecasts time-indexed data with STL, ARIMA/SARIMA, and Prophet, validated by time-ordered backtests against a seasonal-naive baseline.
Designs ML features with leakage-safe pipelines, correct categorical encoding by cardinality, numeric transforms, and validation that a feature earns its place.
Diagnoses slow SQL from the execution plan, names the cost driver, and delivers the rewritten query plus index DDL with the expected plan change and write-side cost stated. The general entry point for slow-query work that routes deep cases to specialist…
Designs MongoDB schemas, indexes, and aggregation pipelines that perform – embed-vs-reference decision rules, the 16MB document limit, ESR compound-index ordering, explain-plan verification, and operational settings for replica sets and sharding.