Trading System Design: Scaling Backtests and Live Trading

0:00If you want a senior level software engineering interview then you're gonna have to pass the system design round And

0:08to do this you should have built a very large very complex highly scalable system And I'm gonna use this

0:18example of an algorithmic trading system to prepare you for your senior level role This is part 3 on how

0:29to build your own algorithmic trading system from scratch Imagine you want to run 100 000 backtests And these backtests

0:41can be very small maybe just 6 months of data using open high low closed data Or they can be

0:53very large They can use 5 years of minute by minute historical options data on the entire US stock market

1:07And your goal is to process all of them all 100 000 backtests This is what a senior engineer should

1:19be able to do And here's just one way you can approach it Number 1 you'll absolutely need your different

1:30backtest workers but this time you don't want just one you want as many as you need You want 10

1:4120 even 50 backtest workers picking items from the queue simultaneously To handle the load you could have all the

1:54backtest queue workers be the same size or each worker can be a little bit more specialized You might have

2:02a worker specifically focused on the historical options backtest and it might have a much larger disk where it can

2:12store some of the data You might have a different worker focusing on the smaller backtest and maybe it can

2:22emit 20 backtests at the same exact time Each worker can pull the data from a central source of truth

2:32like S3 or Tigris and the goal is to build a horizontally scalable system to process all of your data

2:43Another example is live trading You might have had one mini live trading worker that ingested the market data that

2:53looped through each user's portfolios and generated signals that created orders for their trading strategies But when you have a

3:04thousand that one little worker just isn't going to work So you might discuss sharding You might take all of

3:14these different live trading workers and shard it by the user ID And you know the trade offs You chose

3:23the user ID because you wanted each shard to have access to the market data credentials for that user You

3:34don't want user one's market data credentials to power user two's portfolio You have to keep them together and you

3:44understand the trade off you're making Or you might go with a different approach You might pull the market data

3:53out of the live trading service entirely Build a market data service and that's the central source of truth for

4:04all things market data So you can build any number of workers and you can chart it by the complexity

4:13of the trading strategies so each worker can be sized approximately the same You have all of your monitors you

4:23have all of your alerts you know exactly when things go wrong down to the line number and in today's

4:31age the alert should even trigger an AI agent that autonomously resolves the issues and sends you the PR Having

4:42this level of systems design knowledge can get you the senior level software engineering job of your dreams Good luck