If you run the same backtest twice and get two different answers, you can’t trust either one. Reproducibility, the same inputs producing the same result, is what turns a backtest from an anecdote into evidence. It’s also one of the first things to quietly break in a research setup.
What breaks reproducibility
- Uncontrolled randomness. Any randomness without a fixed seed makes results wander between runs.
- Code drift. You tweak the strategy and lose track of which version produced which result. A month later, you can’t tell what you actually tested.
- Data revisions. Vendors restate and backfill data. The “same” historical window can return different numbers later, so a result tied to mutable data isn’t truly reproducible.
- Environment differences. Library versions and platform quirks can shift results subtly.
- Hidden state. A run that depends on the order of earlier runs, cached values, or wall-clock time won’t reproduce cleanly.
How to get it back
- Pin the strategy version. Tie every result to an immutable, identifiable version of the code, so a number always maps back to exactly what produced it.
- Fix the data window. Be explicit about the instruments and date range, and be aware that underlying data can be revised.
- Control randomness. Seed anything stochastic, or design the strategy to be deterministic.
- Use a deterministic engine. The backtester itself should produce the same result given the same code, config, and data, no run-to-run variation of its own.
A realistic standard
Be precise about what “reproducible” means. Bit-for-bit identical results across every possible data revision and environment is a high bar; data vendors alone can move that target. A more practical and honest standard is deterministic replay: given the same code, the same config, and the same data window, the engine produces the same result every time. That’s enough to trust your experiments and compare runs fairly.
How Periscøpe helps
Periscøpe is built for this. Publishing a strategy creates an immutable, checksummed version, and every run pins the exact version it used, so a result always maps back to the code that produced it. The backtester is a deterministic, calendar-aware replay: same code, same config, same data window, same result. That makes runs comparable and re-runnable rather than one-off outputs, short of guaranteeing bit-identical numbers across underlying data revisions, which no honest tool can promise.