Quant Research Workflow: Jupyter Notebooks and Feature Engineering

The output of quant research is not a notebook — it is a deployable signal with documented assumptions. Most research dies in Jupyter because the notebook becomes an unreadable artifact. A disciplined workflow turns exploration into production.

Notebook structure

Use one notebook per research question, with fixed sections in order: hypothesis, data load, feature engineering, signal, backtest, validation, conclusion. A notebook without a hypothesis statement at the top is tourism, not research.

Keep notebooks runnable top-to-bottom after Kernel → Restart. If it only works with state from a previous run, it is broken. Run nbstripout before commit so diffs show logic, not outputs.

Feature engineering principles

Features encode your economic hypothesis. A feature without a stated reason is a fishing expedition.

1. State the mechanism first. "Trend persistence is stronger in low-volatility regimes" — then build a feature combining slope and realized vol. Reverse engineering features from a target is overfitting by construction.
2. Stationarity check. Compute the Augmented Dickey-Fuller statistic; reject non-stationary features or difference them. A feature that trends will fool any linear model.
3. Look-ahead audit. For every feature, write the timestamp at which it would have been knowable. rolling(20).mean() on close at time t is only valid if computed on bars up to t-1. Off-by-one indexing is the leading cause of fake edges.
4. Distribution and outliers. Clip or winsorize features at the 1st and 99th percentile. A single 20-sigma print (a stock split misfeed) will dominate gradient boosting.

Validation before backtest

Before any PnL curve, compute: feature autocorrelation (target should not be predictable from lagged target alone), feature-target mutual information, and stability across non-overlapping windows. A feature that flips sign in its relationship to the target across 2018 vs 2022 is not a feature — it is noise.

The backtest honesty rules

• Walk-forward: train 2016–2019, test 2020; then train 2016–2020, test 2021. Never report a single in-sample fit.
• Costs: assume 5 bps round-trip minimum, even if your broker charges less. Slippage is real.
• Capacity: if the strategy needs 10% of daily volume, mark it as unscalable.

From notebook to production

When the signal is validated, port it out of the notebook into a versioned module with unit tests. The notebook is the lab; the module is the product. If the production signal and the notebook diverge, the notebook is wrong.

The deliverable

A research deliverable is: the hypothesis, the features with mechanisms, the walk-forward results with costs, the failure conditions (when does this stop working?), and the production port. Anything less is a notebook that will never trade.