Writes correct, vectorized pandas for cleaning, joining, reshaping, and aggregating tabular data, with validated joins and deliberate dtype and missing-data handling. Use when someone asks "why did my merge duplicate rows", "how do I clean this CSV in pandas", "my groupby numbers look wrong", "this apply is too slow", or is transforming a DataFrame for analysis or a pipeline. Do NOT use for first-pass profiling of an unfamiliar dataset - use eda-playbook instead; for data too large for one machine use spark-jobs; for answering business questions directly in SQL use sql-to-insights.
Click to play with sound.
---
name: Pandas Expert
description: Writes correct, vectorized pandas for cleaning, joining, reshaping, and aggregating tabular data, with validated joins and deliberate dtype and missing-data handling. Use when someone asks "why did my merge duplicate rows", "how do I clean this CSV in pandas", "my groupby numbers look wrong", "this apply is too slow", or is transforming a DataFrame for analysis or a pipeline. Do NOT use for first-pass profiling of an unfamiliar dataset - use eda-playbook instead; for data too large for one machine use spark-jobs; for answering business questions directly in SQL use sql-to-insights.
---
# Pandas Expert
Most pandas bugs are silent: a join that fans out rows, a dtype that upcast to object, a NaN that vanished from a sum. The output looks plausible and is wrong. This skill produces transformations that validate themselves - every join asserted, every coercion counted, every aggregate reconciled - so errors surface at the line that caused them instead of in a dashboard three weeks later.
## Operating procedure
Follow the steps in order. Types must be fixed before cleaning, cleaning before joining, joining before aggregating - each later step silently produces wrong numbers if an earlier one was skipped.
### Step 1: gather inputs
Before writing any transformation, establish:
- The grain of each table: what one row represents. If the user cannot state it, derive it with `df.duplicated(subset=candidate_keys).sum()` and label the result a guess until confirmed.
- The expected output: shape, grain, and one or two spot-check numbers the user already trusts (last month's total, a known customer's count).
- Whether the source can be mutated. Default to no: work on a copy, never mutate the caller's DataFrame.
- Approximate size. Pandas wants roughly 3-5x the on-disk dataset size in free RAM for comfortable transformation work. Above a few GB in memory, reach for chunked reads (`pd.read_csv(..., chunksize=...)`), pyarrow-backed dtypes, or hand off to spark-jobs.
### Step 2: inspect before touching
Run `df.info()`, `df.describe()`, `df.isna().sum()`, and `df.nunique()` first. You are looking for: object columns that should be numeric or datetime, null counts, and cardinality. This is a two-minute sanity pass, not a full audit - for a genuinely unfamiliar dataset, run eda-playbook first.