Your Backtest Is Lying to You

Maciej Lewandowski · July 2026 · Engineering notes

I built a scoring and backtesting system for my own investing: a multi-factor stock ranker on one side, a replay engine on the other. The first equity curve it produced was gorgeous. Smooth, steep, barely a drawdown. I did not feel clever. I felt suspicious, because I have written enough systems to know that when a program tells you something wonderful on the first run, the program is usually describing itself rather than the world.

It was. The curve was a bug report written in the notation of finance. Here is what was in it, and why almost every one of these failures is a familiar systems problem wearing a ticker symbol.

Look-ahead bias, the version that actually gets you

Nobody writes buy(tomorrow_close). The real leaks are quieter:

Indicators computed on the full series, then sliced. You load ten years of prices, compute a moving average or a z-score over the whole array, and then iterate day by day. Every value in that array was computed with knowledge of the entire history, including the part that had not happened yet. The loop looks causal. The data is not.
Deciding during a bar on that bar's close. A rule that says "if today closes above the average, buy today at the close" is only executable if you knew the close before it printed. Signal time and execution time are different clocks, and the gap between them is where honest backtests lose most of their edge.
Fundamentals applied before publication. A P/E ratio dated to the end of the fiscal quarter is a ratio nobody could compute until the filing came out, weeks later. Ranking a universe on data that was still confidential is not a factor model, it is time travel.
Restated financials. Vendors typically serve you the corrected numbers. Your backtest then trades on figures that were revised after the fact, in exactly the direction reality later confirmed.

The fix is not vigilance. Vigilance fails at 1am when you are adding one more feature. The fix is an invariant that the engine cannot violate: at simulated time T, the only data reachable is data whose timestamp is less than or equal to T, enforced by the API, not by the author's discipline. Point-in-time storage with two timestamps per record, the event time and the time it became knowable, and an accessor that takes T and physically cannot return anything else. Make the wrong thing unrepresentable and the class of bug disappears instead of moving around.

Survivorship: your universe is a list of winners

Ask where your universe of stocks came from. If the answer is "today's index members" or "everything my data provider currently lists", the backtest is running on companies pre-selected for the property of still existing. Everything that went bankrupt, got delisted, or was acquired at a discount has been quietly removed from history, and those were precisely the positions that would have hurt.

Testing "the S&P 500 constituents" with today's membership list compounds the problem, because index membership itself is a momentum and quality filter. Companies join after doing well and leave after doing badly. Backtesting today's members over the last decade means buying, in 2016, the firms that a committee would later decide had earned their place. You need historical constituent lists with the dates they changed, and a universe that includes the dead. A strategy that only works on survivors has discovered survival, not alpha.

The point-in-time problem, generally

Survivorship and look-ahead are two symptoms of one disease: history is not a fixed object. It gets rewritten under you.

Dividend and split adjustments retroactively change every prior price in the series, so the number your rule compared against last month is not the number it compares against today.
Vendors revise silently. A backfill, a corrected corporate action, a changed methodology, and yesterday's file is not today's file.
Tickers get reused. The same three letters can be two unrelated companies across a decade, which is why identity should hang off a stable security identifier and never off the symbol.

Run the same backtest against "the same" data twice, a month apart, and get two different equity curves. That is not a market observation. That is a reproducibility failure, and it means the number you are staring at belongs to your data pipeline rather than to your strategy.

Overfitting: the best of 200 tries is a measurement of noise

Search a parameter grid of 200 combinations and report the winner, and you have not found a strategy, you have run a lottery and reported the ticket that hit. The more configurations you test, the higher the best in-sample result you should expect from pure chance, which means the winner's performance needs to be discounted by how hard you searched before it means anything. Two hundred coin-flippers, one of them lands eight heads in a row, and there is nothing special about his wrist.

The countermeasures are unglamorous. Hold out data you genuinely never look at. Use walk-forward: fit on a window, trade the next window blind, roll, and read only the stitched out-of-sample result. Count your experiments honestly, including the ones you abandoned, because the ones you abandoned are part of the search. And remember that "my strategy beat buy-and-hold from 2015 to 2020" is a single sample from a single regime, reported by someone who chose the dates after seeing the data.

Costs the naive engine forgets

The first version of any engine fills every order instantly at the close, in unlimited size, for free. Reality charges for all three:

Spread. You buy at the ask and sell at the bid. Round trip, you pay the spread every time.
Slippage and market impact. Your own order moves the price against you, and the effect grows with size and shrinks with liquidity.
Borrow costs. Short legs are not free, and the hardest-to-borrow names are exactly the ones the signal loves.
Taxes and commissions. Jurisdiction-specific, turnover-sensitive, and brutal to high-frequency rebalancing.

A strategy whose per-trade edge is smaller than the spread is not an edge, it is an invoice. And an engine that fills you at a price nobody was actually offering is not simulating a market, it is simulating a wish.

Regime dependence: n equals one

Markets are not stationary. The relationships your factors exploit are conditional on a monetary and structural backdrop that changes on a timescale of years. A ten-year backtest feels like a lot of data because it contains thousands of daily bars, but on the variable that dominates asset returns, the interest rate cycle, it may contain exactly one observation. Thousands of samples of the noise, one sample of the thing that matters. Ask what happens to your strategy across a regime it has never seen, and if the honest answer is "unknown", say so in the report.

The engineering frame

Here is the reframing that changed how I built mine. A backtest is a replay engine. It is the same category of system as any event-sourced service that must reconstruct the past faithfully, and the properties that make those systems trustworthy are exactly the ones that make a backtest trustworthy:

Deterministic replay. Same inputs, same seed, same outputs, always. If your run varies, your results are not measurements.
Immutable event log. Market data and fundamentals are appended facts, never updated rows. Corrections arrive as new events with their own knowable-at timestamp.
No mutation of history. Adjustments are computed at read time from the event log, not baked destructively into the stored series.
Clock discipline. Event time and processing time are different fields with different meanings, and confusing them is the whole of look-ahead bias restated in distributed-systems vocabulary.
Pinned, versioned datasets. Every run records the snapshot identifier, the code commit and the parameters, so a result can be reproduced months later or declared void.

If your backtest is not reproducible byte for byte from a pinned dataset, you are not measuring a strategy. You are measuring your data pipeline.

The checklist

Point-in-time access only: at time T the engine can reach nothing timestamped after T, enforced by the API rather than by care.
Signal time and execution time are separate, with a realistic gap between them.
The universe includes delisted, bankrupt and acquired names, reconstituted from historical membership with dates.
Identity keys off a stable security identifier, never off the ticker.
Costs are modelled explicitly: spread, slippage, impact, borrow, commissions, taxes. Then ask whether the edge survives.
Out-of-sample and walk-forward results are the only ones you quote, and you count every experiment you ran.
Every run is pinned to a dataset snapshot plus a code commit, and reruns match byte for byte.
The report states which regimes were covered and, more importantly, which were not.

None of this makes a backtest true. It makes it falsifiable, which is the most a simulation of the past can offer. The beautiful curve is the null hypothesis. Your job is to spend a week trying to break it, and to be genuinely pleased when you succeed, because finding the leak in your own engine is enormously cheaper than finding it in the market.