algorithmic trading / backtesting / parquet / data engineering / options trading
Parquet for Backtesting: My $230 Disk Bill Is Now $34 a Month
Adding minute-level options data to an event-driven backtester changes the problem. A trading day has 390 minutes, one stock can have thousands of contracts, and the minute-level option quotes alone take up more than a terabyte. Pulling millions of rows out of a database makes every backtest crawl.
Parquet splits data into chunks that each remember their smallest and largest values, so a backtest asking for SPY in March skips every chunk that cannot contain it. The first version copied 200 GB onto each of seven servers, about $230 a month of duplicate disk. One copy on Tigris, streamed in pieces, brought that to $34, and Tigris charges nothing to download your own data. Part 2 of 3.
Transcript
0:00Your event-driven backtester works, and your daily prices come straight from your database.
0:04Then you get ambitious and add options, minute by minute.
0:08A trading day has three hundred ninety minutes, and one stock can have thousands of contracts.
0:13My minute-level option quotes alone take up more than a terabyte.
0:17Every backtest now drags millions of rows out of your database.
0:21Your backtests crawl, and your MongoDB bill climbs every month.
0:25So what do you do?
0:26You switch to Parquet files.
0:28Parquet splits your data into chunks, and every chunk remembers its smallest and largest values.
0:33When a backtest asks for SPY in March, it skips every chunk that can't contain SPY in March without ever downloading it.
0:39My first version copied two hundred gigabytes onto every server.
0:43I had seven servers, so I was paying two hundred thirty dollars a month for duplicate disk.
0:48Now one copy lives on Tigris, and every backtest streams only the pieces it needs.
0:53That disk bill dropped to thirty-four dollars.
0:56Tigris also charges nothing when you download your own data.
1:00On S3, one of my options backtests would pay about thirty cents in transfer fees every time it ran.
1:06After all that, the slowest part of every backtest was the engine itself.
1:10Follow for part three.
Join the conversation
Loading conversation…