Building an equity trading system, from scratch
Using a microservice-based architecture
Preamble
When I first became interesting in programming, I always wondered how a complex system like Uber worked. When I asked this question, I never received satisfactory answers. Some of this was my fault, as even if someone went over the details with me, I’m not sure I would have truly understood it at the time. Even today, it may take years to fully understand a complex distributed system.
This post is the start of a series where I’ll go through some of the designs used in my own trading system, in the hopes that it will help others. I currently develop systems at Shenandoah Research.
Part I — From Scratch
Part II — The Details (work in progress)
Part III — Monitoring (work in progress)
Part IV — Context (work in progress)
From the Start
Follow the Unix Philosophy every step of the way. The last thing you want is to spend excess amounts of time trying to maintain your custom solutions. Use existing frameworks or tools when possible. Take this from someone that has created a lot of custom solutions. Eventually, you will learn the limitations of the tools and then and only then should you invest any time in creating custom solutions.
From a software design point of view, there is hardly anything unique about (basic) equity trading systems. At the end of the day, you need a system that is capable of entering and exiting the market (reliably!). Start with this. You can incorporate accounting, visualization and other features later.
Things to keep in Mind
Single source of truth (SSOT)
When designing the system, ensure your data and state comes from only one source. For example, there should only be one source for whether your position is open or closed. If you have a position tracker component, ensure all other components get the position status from this component (instead of the broker API for instance).
Determine your data structures before starting
What is a position? What properties does it have? Will you have a position with attributes of “entry_orders” and “exit_orders” ? Or will it just be “orders” with methods to determine which orders are entry/exit? For an example of just how complex this can get, take a look at TDAmeritrade’s order schema.
This is a deceptively hard question. I can promise that you will eventually have to go back and change your original designs, so it is best to limit that by thinking about the problem upfront.
Incorporate Fail safes
One of the most famous instances of catastrophic “software” failure is that of Knight Capital’s runaway trading system. There will always be bugs. Try to make it impossible very hard for a bug to cause loss of capital.
For example, when I create an execution service, I incorporate a step that ensures certain conditions are met before the trade can be sent to the broker. One of those conditions is that there have been less than X number of trades in the last Y seconds. This is a general catch all that would prevent too many orders going through in the case of an infinite loop.
Many brokers also have “circuit breaker” like features that may help prevent catastrophic loss — and while these are nice features, I would always say design your systems so that you don’t have the pleasure being introduced to them.
Testing
There will assuredly be times where you have to change the way certain components behave. One of the challenges when building trading systems is only being able to really test during market hours. Unfortunately, this overlaps with a decent chunk of the time most people also develop their system.
As such, design your system in a way that you can run simulated orders from the start. For instance, for your execution component, create a “paper” broker first. This way, you can send many simulated trades through your entire system without worry, at any hour of the day. This isn’t a substitute for testing against your actual broker — but it will save you a headache.
Bonus Tips!
This falls more into the realm of general programming, but keep responsibilities separate. If you have a position manager component, ensure it sticks to managing positions. If you then want to introduce “risk management” do that in a different component and pursue composition.
Increase shareholder value! It sounds silly, but you are the shareholder of your system. You don’t need to make it perfect, make it functional. You can worry about CSS when you’re rich :)
Building things
I personally have set up my system to be easily extendable by utilizing GitLab. Code changes are committed, tests are run and applications are deployed. This is referred to as continuous integration and development (CI/CD), which is part of DevOps. One need not go down the rabbit hole of DevOps, however I would say getting a basic CI/CD pipeline working is a must, as your system will be in a state of constant change, at least starting out. It will make you a more efficient developer and save you much needed time. I would recommend figuring out how to deploy a very simple “Hello World” app in the language of your choice before starting to build your system.
Languages and frameworks
The applications themselves are mostly Python-based. I use frameworks like FastAPI and Django. I would 100% recommend using a frameworks, as you want to spend as little time fussing over implementation as possible. For dashboards and visualization, I use NextJS (a react framework) w/TypeScript. I also have a few apps in Go. In my opinion, the language of choice doesn’t matter too much as long as you’re efficient.1 You just don’t want to be re-inventing the wheel every other day.
So what do you actually build?
Some may now ask what I mean when I say “applications” and that’s a fair question, especially if you’ve never built or designed a system. When I say applications, I mean applications in a pretty literal sense. Computer programs that handle a specific function. Some examples are:
Broker Interface
Portfolio Constructor
Risk Control
Alpha Model
These applications can then work together via RPC (REST may be more familiar). For example; Portfolio Constructor asks the Alpha Model what trades it recommends. It can then check with the Risk Control service to see if the portfolio is acceptable. Risk Control gets the information it needs from the Broker Interface, etc.
We will touch on additional features later on in the series but I hesitate to even write them down here, as its very easy to have scope creep.
In Summary, how do I start?
First, get familiar with CI/CD. After that, start planning out the components of your system (refer to Languages and Frameworks section). These components will work together to reliably enter and exit the market. I’d start with figuring out how to put on trades and represent a position in code. Have your component open and close a few positions. Test this thoroughly.
In the next section (Part II — Details) I’ll go over more details.
Keep in mind the Unix Philosophy when planning your system. If you do this, extending the system to incorporate other features will be much easier.
There are some caveats. Python is slow. However, I’ve found that to rarely be a problem for my goals. If your system will be doing lots of loops or processing lots of data in real time, it may be best to use another language like Go.