Backtesting Engine Design: Event-Driven Simulation, Fill Modelling, and the Biases That Fake Your Returns

Event-driven, not vectorized

Vectorized backtests — compute a signal column over the whole price series, shift it, multiply by returns — are fast and are the single largest source of fake results. The entire history is present in memory while the signal is computed, so any accidental use of future information is invisible. A rolling normalization over the full series, a fillna that back-propagates, a threshold tuned on the whole sample: all of these leak, and none of them raise an error.

An event-driven engine makes leakage structurally impossible. Time advances one event at a time, and the strategy is handed a read-only view containing only what has already occurred. If it cannot see the future, it cannot use it.

def run(events, strategy, broker, portfolio):
    for event in events:                       # strictly chronological
        if event.type == "BAR":
            portfolio.mark_to_market(event)
            # Strategy sees a view that physically cannot reach past the event timestamp.
            orders = strategy.on_bar(MarketView(as_of=event.ts))
            for order in orders:
                broker.submit(order, arrival_ts=event.ts)

        elif event.type == "FILL":
            portfolio.apply(event)             # costs already deducted
            strategy.on_fill(event)

    return portfolio.results()

One detail carries most of the realism: an order submitted on a bar cannot fill at that bar's close. It fills at the next available price after the decision, with the delay you would actually experience. Filling at the price that triggered the signal is the most common way a mediocre strategy becomes a spectacular one on paper.

The bias checklist

The subtle one is restatement. A company reports earnings in February and restates them in November; a vendor's 'historical fundamentals' table often shows only the restated figure, dated to the original quarter. A strategy reading that table trades in March on information that did not exist until November, and no amount of careful event ordering catches it — the leak is in the data, not the loop.

Fills and costs decide the outcome

For anything trading more than a few times a month, cost modelling matters more than signal quality. A strategy earning 8 basis points per trade is profitable at 3bps of cost and dead at 12. Getting this wrong is not a rounding error — it inverts conclusions.

• Spread: cross it. Buy at ask, sell at bid. If you only have bar data, charge at minimum half the typical spread for that symbol and time of day.
• Slippage scaled to size: model impact as a function of order size relative to period volume, not as a flat constant.
• Participation cap: refuse to fill more than a realistic share — commonly 1 to 10% — of the bar's volume, and carry the remainder forward or cancel it.
• Fees, commissions, exchange rebates, and for crypto perps, funding payments accrued per interval.
• Short borrow cost and, importantly, borrow availability — a backtest that shorts hard-to-borrow names for free is describing a market that does not exist.
• Gaps and halts: an order resting through an overnight gap fills at the gap, not at your limit.

Make every cost a pluggable component with its own tests, and always run the final backtest at a pessimistic setting. If a strategy only works under optimistic costs, you have learned something valuable and cheap.

Walk-forward and the multiple-comparisons trap

If you test two hundred parameter combinations and report the best, you have found the luckiest one, not the best one. This is the central statistical failure of systematic trading research, and it is invisible from inside a single backtest.

1. Split history into sequential train and test windows — for example, optimize on two years, evaluate on the following six months, then roll forward.
2. Reoptimize inside each training window only, and record only the out-of-sample results.
3. Concatenate the out-of-sample segments; that concatenation is your honest equity curve.
4. Hold back a final period that you look at exactly once, at the end. Every additional look burns it.
5. Count and report the number of configurations tested. Deflate expected Sharpe accordingly — an unreported search of hundreds of variants makes a Sharpe of 1.5 unremarkable.

Prefer strategies whose parameters sit on a plateau rather than a peak. If performance collapses when a lookback moves from 20 to 22, you have fitted noise. Robustness across neighbouring parameters is a far better predictor of live behaviour than any in-sample metric.

One strategy interface, two brokers

The strategy must not know whether it is in a simulation. It receives market events and emits orders; a SimulatedBroker or a LiveBroker sits behind the same interface. This is what makes 'it worked in backtest but not live' a debuggable claim instead of a shrug — the divergence has to be in data, latency, or fills, and each of those is measurable.

• Run live and simulated side by side on the same feed for a period, and reconcile every divergence before committing capital.
• Log the exact market view that produced each decision so any live trade can be replayed offline.
• Kill switches on drawdown, order rate, and position limits — enforced in the broker layer, where the strategy cannot override them.
• Paper trade for long enough to see a regime you did not train on. It is the only test that is not retrospective.