Trading System Design: APIs, Queues, and Workers

0:00Most computer science students apply to hundreds of jobs and they're confused on why they can't find the software engineering

0:06jobs of their dreams And the answer is your projects look like everyone else's You don't have anything to help

0:17you stand out and what you need to do is a project that no one else has done So in

0:24this video I'm going to teach you how to build your own algorithmic trading system and in this video we're

0:33going to design it as if we're a mid level software engineer So let's go back to our previous version

0:42a single loop in a TypeScript process Your trading bot worked perfectly when it was just one user but when

0:51a hundred more people sign up it completely fails Your back tests are slow you can't run any sort of

1:02complex optimizations on your strategies and your single node process CPU is at a hundred percent It just doesn't work

1:13at all Here's what a mid level software engineer should do The big idea is that you want to build

1:23a distributed system Your web server should be very light and basically just respond to the user's requests It should

1:34be stateless it shouldn't have any heavy CPU intensive process The only thing it does is takes the request and

1:44sends a response Anything slower goes to separate servers separate workers and separate jobs you can use something like a

1:54queue like sqs or bullmq you can create different running workers like an order queue worker a data pipeline worker

2:07a system alerts worker that sends you notifications when anything goes down you can have your own separate back testing

2:18service your own separate live trading service and even a genetic optimization service all doing their specific roles you want

2:32to pick the database that's right for the job you don't want to store minute by minute historical options data

2:41in mongo db your mongo db or postgres data just has the application state information about your users maybe some

2:54ai chat logs just the simple stuff that isn't too heavy and then you're going to use the heavy technologies

3:03for what you need so you might use something like tigris to store parquet files that contain your historical market

3:14data You might use Postgres with pgvector so you can run things like semantic search And the key idea is

3:26you're building a distributed system ideally a fault tolerant self healing distributed system And that's what separates you from a

3:41junior software engineer And if you're trying to get the senior job then you're going to need to go even

3:51further So like and follow for part three