Data Analysis Advanced

Advanced · Data & Analysis track · ~35 min hands-on + readings and quiz

← Back to the Data & Analysis track

Scaffold full pipelines with AI — then audit the subtle failure modes.

What you’ll be able to do

  • Use AI to scaffold a multistep modeling pipeline
  • Audit AI-built pipelines for leakage, validation, and metric choice
  • Know the limits of AI on tabular and time-series data

Overview

For advanced work, AI can scaffold a full pipeline — feature prep, modeling, evaluation. The failure modes get subtler: data leakage, the wrong validation strategy, overconfident metrics. Your job is to audit each step and the assumptions behind it.

Recent work cautions that “foundation models” don’t automatically win on tabular or time-series data — classic methods often match or beat them. Skepticism is part of the method.

Practice activities

Activity 1 · Advanced — Audit a modeling pipeline

Time ~25 min · Tools ChatGPT Edu (or Claude) + R or Python

Goal. Build a pipeline, then rigorously audit it for the errors that scale.

Setup. Download the Crash Data 2020–2024 (CSV). Outcome: a serious-crash indicator (fatal or suspected-serious injury). Use only fields known at the time of the crash.

Steps.

  1. Scaffold it:

    Build a complete R or Python pipeline that predicts a serious-crash indicator (fatal or major injury) from crash characteristics in this Philadelphia data: load, clean, split, fit a logistic model, and report performance. Use only fields known before or at the time of the crash.

  2. Audit it, point by point:

    • Leakage — are any predictors really the outcome in disguise (injury counts, fatality flags)? Remove them.
    • Validation — is the split appropriate? Prefer a temporal split (train on earlier years, test on later).
    • Metric — serious crashes are rare, so accuracy misleads. Look at recall, precision, and PR-AUC.

    Audit this pipeline for data leakage, an inappropriate validation split, and a misleading metric given class imbalance. Show me exactly where each problem is and the minimal fix.

  3. Fix one real flaw and re-run.

Expected result. A pipeline with at least one substantive flaw found and corrected, and honest metrics.

Check your work. Confirm no predictor encodes the outcome, and that the test set is genuinely held out (ideally later years).

Common pitfalls. Leakage inflates performance and hides easily. High accuracy on a rare outcome is meaningless. This is observational data — don’t imply causation.

Stretch (optional). Add calibration (are the predicted probabilities trustworthy?) and report it.

Activity 2 · Advanced — Compare a baseline to a complex model

Time ~20 min · Tools ChatGPT Edu (or Claude) + R or Python

Goal. Decide whether model complexity actually earns its place.

Setup. Same outcome and temporal split as Activity 1.

Steps.

  1. Compare:

    Compare a simple logistic-regression baseline to a gradient-boosted model on the same temporal split, reporting PR-AUC and calibration. Tell me whether the complex model is justified.

  2. Inspect the comparison yourself — don’t accept the verdict blindly.

  3. Write one paragraph justifying your choice, weighing interpretability, performance, and maintenance.

Expected result. A side-by-side comparison and your reasoned decision.

Check your work. Is the performance gap real and meaningful, or within noise? Would you trust the complex model in practice?

Common pitfalls. A higher AUC doesn’t automatically justify complexity. Recent evidence shows classic methods often match boosting and “foundation” models on tabular data.

Stretch (optional). Log your prompts, package versions, and a random seed so a colleague could reproduce the comparison (see the Reproducible AI topic).

Activity 3 · Advanced — Build, run, and audit the pipeline in one Claude Code session

Time ~30 min · Tools Claude Code

Goal. Have Claude Code build the serious-crash pipeline, actually execute it, and then audit the fitted model — coefficients, splits, metrics — instead of auditing code by eye.

Setup. Download the Crash Data 2020–2024 CSV into a working folder, cd into it, and start claude. Same outcome as Activity 1: a serious-crash indicator (fatal or suspected-serious injury), predicted only from fields known at the time of the crash.

Steps.

  1. Ask for the pipeline — and insist it runs, not just writes:

    Read crash_data.csv and inspect the columns. Write a Python script that builds a logistic regression predicting a serious-crash indicator (fatal or suspected-serious injury) using only fields knowable at the time of the crash. Split, fit, and evaluate — then run the script and show me the output. If it errors, fix it and re-run until it works.

  2. Interrogate the fitted model, not the source code:

    Print the model’s coefficients sorted by absolute magnitude, the class balance in train and test, and the exact split boundary. Which features are doing the most work, and why?

    Injury-severity counts, EMS/transport fields, or anything derived from the outcome sitting at the top of the coefficient list is your leakage, visible in the actual model rather than hypothetical in the code.

  3. Now make it audit and repair its own work:

    Audit this pipeline for (a) predictors that encode the outcome, (b) validation strategy — I want a temporal split, train on 2020–2022, test on 2023–2024, not a random split, and (c) metrics appropriate for class imbalance: recall, precision, and PR-AUC, not accuracy. Fix every problem you find, re-run, and show me the before/after metrics side by side.

Expected result. A working, executed pipeline; at least one substantive flaw caught in the fitted model (leaky feature, random split, or accuracy-only reporting); and a before/after comparison where the honest metrics are visibly worse — that drop is the audit working.

Check your work. Ask Claude Code to print min and max crash dates for train and test — they must not overlap. Re-inspect the corrected model’s top coefficients and confirm nothing there is only knowable after the crash. If performance barely changed after removing a “leaky” feature, question whether it was actually leaking.

Common pitfalls. Claude Code will happily “fix” a flaw by explaining it without editing the code — confirm the script changed and the numbers changed with it. And the standing rule: Claude Code is not Drexel-supported and is approved for Low Risk (public) data only — this public crash CSV qualifies; your real project data likely does not without a Third-Party Risk Assessment.

Stretch (optional). Ask it to add a trivial baseline (predict the majority class, or a one-feature model) and report how much the full model beats it on PR-AUC.

Check your readiness

Answer these, then check — your score suggests whether to dive in or skim the readings first.

Useful resources

  • tidymodels — modeling and validation framework in R.
  • scikit-learn — machine learning in Python.
  • Quarto — reproducible analysis documents.