backtesting / algotrading / software engineering / build in public
Can You Vibe Code a Backtester in 5 Hours? The 5 Hour, 5 Week and 5 Month Versions
A creator claimed an AI backtesting tool could be vibe coded within five hours. I've been building a backtesting engine for almost six years and I'm still not done, so here is what five hours, five weeks and five months actually get you.
Five hours gets you daily Yahoo Finance prices and textbook indicators, with survivorship bias, broken ticker changes (Facebook to Meta) and lookahead bias. Five weeks fixes most of that and adds minute data, then hits the next wall: reused tickers like FB, split data you have to verify, sparse options quotes, and over 100 TB of historical options data from Massive to process. Five months gets you ticker changes and mergers from SEC filings, point-in-time universes like the top 100 by market cap, and a live trading engine, and then results drift, golden files run every day, and a minute-by-minute options backtest can need hundreds of gigabytes. This is not financial advice.
Transcript
0:00"Thomas Quant is both bottom of his class in college and a failure in real world quant work."
0:07"And his idea, an AI backtesting tool, could be vibe coded within five hours."
0:11Let's fact check this video. This woman claims that she can build a robust backtesting engine
0:17in 5 hours or less. I'm willing to bet $100 that she can't and here's why. I've been building a
0:26backtesting engine for nearly 6 years and I'm still not 100% done. So let me tell you what a
0:355-hour version actually gets you. Within 5 hours, you can build a backtesting engine that works on
0:42Yahoo Finance or open, high, low, closed data. Those trading strategies that you can create
0:48are very simplistic. So like typical Tendergate indicators that you can imagine. SMA, EMA, RSI,
0:56MACD, those types of indicators. If you use Rust in the open source Chrono TA library,
1:02then you might be able to cut that down to 4 hours. Here are the problems with the 5-hour
1:09version. Number 1, it'll still have survivorship bias because you haven't modeled stocks getting
1:15delisted. Secondly, when the ticker of the company changes, your backtesting system simply won't
1:22account for it. So when you query data for Meta, it's not going to include any data for Facebook.
1:29Number 3, it's going to have look-ahead bias unless you use an existing open source library.
1:36So that's the 5-hour version. Let me tell you what the 5-week version buys you. Within 5 weeks,
1:42you can fix most of these problems. You can make sure your system doesn't incorporate look-ahead
1:47bias. You can natively handle ticker changes and merger events. And you can start incorporating
1:53basic minute-by-minute data into your backtesting system. But you're going to encounter a lot more
1:59edge cases that you didn't foresee originally. For example, Facebook now trades as Meta, right?
2:06But did you know the ticker FB trades as a new stock? Are you going to handle that? And also,
2:12where are you going to get your data for stocks? Are you planning to just use a vendor like Massive?
2:16Well, guess what? A lot of their data is actually wrong. And then let's talk about options. Are you
2:22going to try to process every single quote that you can incorporate into your system? I would hope
2:28not unless you have just a whole bunch of supercomputers. So then maybe you'll aggregate it.
2:34You'll only use minute-by-minute data and you'll store yourself the bid-ask price at the end of
2:41each minute. How are you going to backtest that data? Because in some periods, there are no quotes.
2:47In other periods, nobody bought a contract. How exactly are you going to handle that when your
2:53backtests look crazy? By the end of five weeks, you'll barely just finished storing or processing
3:00over 100 terabytes of quote market data from Massive. You're not going to be able to fetch
3:06it efficiently, but you at least stored it within your data lake. Now let's get to the five-month
3:13version. You follow ticker changes and merger events, and the data comes from official SEC
3:19filings, so you know you're getting what actually happened. Your options backtests are finally
3:25somewhat accurate, and you can start backtesting several stocks at the same time. You don't have
3:30to just trade Apple or Nvidia. You can maybe backtest the top 100 stocks by market cap.
3:36If you're smart and planned ahead, you'll even figure out how does that number change across
3:42points in time because the top 10 stocks by market cap is different last week than it is
3:48right now. And maybe you would have connected it with a live trading engine and connected a single
3:55broker at this time, but you're still going to run into issues. Their backtest results is going to
4:01appear to drift, and you're not going to know why unless you created a robust verification system.
4:07You have golden files that runs every day, and it looks at the changes in your backtest across time,
4:15and you also want to make sure that they're not getting any slower. A minute-by-minute historical
4:21options backtest can require hundreds of gigabytes of data, so you want to make sure that you're
4:27processing that data efficiently and processing the most number of backtests that you possibly
4:33can. And every single day, you'll find the tiniest edge case that alerts you at 3 a.m.
4:41Now, if I'm wrong, I'm happy to be proven wrong. Show me what you can do. Build
4:49QuantPad in five hours, and I'll give you $100. Good luck.
Join the conversation
Loading conversation…