algorithmic trading / backtesting / rust / performance / software engineering
How to Read a Flamegraph: I Cut My Backtest Time in Half
An options backtest across about 5,600 stocks took almost ten minutes, and a genetic optimizer running hundreds of backtests per generation would need two days per generation. A flamegraph samples the program thousands of times and stacks up where the time went; a wider bar means that code ate more of it.
The real flamegraph from the engine showed more than half the CPU preparing option data. Stopping one scan early took a setup step from two minutes to six seconds, filtering option columns before copying them made the run almost 40% faster, and borrowing records replaced cloning them. Spreading a hot loop across every core made it 73% slower, so it was thrown out. The same backtest now runs in about four minutes. Part 3 of 3.
Transcript
0:00Your data loads fast now, so the slowest part of every backtest is your engine.
0:03You wanted anyone to backtest an options strategy on every minute of history across fifty-six hundred stocks.
0:10One of those runs took almost ten minutes, and nobody waits that long.
0:14A genetic optimizer runs hundreds of backtests per generation, so a single generation would take two days.
0:19So you stop guessing and start measuring.
0:22A flamegraph samples your program thousands of times and stacks up where the time went.
0:26When a bar is wider, that code ate more of the time.
0:30This flamegraph comes from my real engine.
0:32More than half the CPU went to preparing option data.
0:36One function kept scanning for stocks after it had already found all of them.
0:40I made it stop early, and that setup step went from two minutes to six seconds.
0:45I was also copying every option column before filtering it.
0:48When I filtered first, the run got almost forty percent faster.
0:52And I was cloning whole records in places where borrowing them was enough.
0:56One of my ideas made things worse.
0:58Spreading a hot loop across every core made it seventy-three percent slower, so I threw it out.
1:04That same backtest now runs in less than half the time, about four minutes.
1:09Follow for more.
Join the conversation
Loading conversation…