Chapter 2: The Critical Components of a Trading System

In the previous chapter, we learned how to create high-frequency trading (HFT) strategies. In this chapter, we are going to study how to convert these strategies into real-time software that will connect to an exchange to actually apply the theory that you've previously learned. We will describe the functioning of a trading system capable of trading assets.

In this chapter we will cover the following topics:

• Understanding the trading system
• Making a trading system trade with exchanges
• Order book management
• Strategy making decisions on when to trade

By the end of this chapter, you will be capable of designing a trading system, connecting a trading system to an exchange, and building a limit order book.

Understanding the trading system

Designing a trading system for HFT trading requires much more than knowledge of programming and trading. The following chapters of this book will describe these parts in depth, which will give you an edge in designing an HFT system. In this section, we are going to talk about the fundamentals of trading system design. One of the most critical parts of designing a system is having a detailed description of the requirements. The goal of a trading system is to support your trading ideas. Any trading strategies start by getting data and end up with making a decision based on this data. A trading system will oversee collecting market data (that is, price updates) and sending orders to the exchange. Additionally, it will collect answers from the exchange containing information on the orders. These market updates could be representing any state of the orders: canceled, rejected, filled, or partially filled. It will also compute metrics measuring the performance of your portfolio (such as profit and loss, risk metrics, or information about the different processes of the trading system).

When deciding whether to create this type of software, we need to keep the following points in mind:

• Asset class: Knowing which asset class will be employed in the trading system will change the data structure of this software. Each asset class is unique and has its own set of characteristics. Building a trading system for US equities won't be the same as building a system for foreign exchange (FX). Stocks in the United States are mostly traded on two exchanges, the New York Stock Exchange (NYSE) and NASDAQ. These two exchanges have roughly 7,000 firms (symbols) listed. FX contains six main currency pairs, six minor currency pairs, and six exotic currency pairs, as opposed to stocks. We can add additional currency pairs, but no more than 100 will be available. Unlike the US equity market, where we can have two main exchanges, in the FX market, there will be hundreds of exchanges. The number of symbols and the number of exchanges will change the architecture of a trading system.

• Trading strategy type (high frequency, long-term position): The software architecture will be influenced by the trading strategy type. HFT tactics necessitate transmitting orders in a very short period of time. In the case of US stocks, a standard trading system will decide to send an order in microseconds. A Chicago Mercantile Exchange (CME) trading system has a latency in the order of nanoseconds. Based on this finding, technology will play a significant role in the software design process. If we only consider the programming language, Python is not well suited to speed, and we would rather choose C++ or Java. If we wish to take a long-term position, such as one that lasts many days, the speed with which a trader may obtain liquidity faster than others is irrelevant.
• Number of users (or trading techniques): As the number of traders grows, so does the variety of trading tactics. This indicates that the number of orders will increase. Before submitting an order to an exchange, we must ensure that the orders we are about to send are valid; we must ensure that the overall position for a specific instrument has not been reached.

Trading strategies are being moderated by an increasing number of rules in the trading sector. We shall test the compliance of the orders that we wish to send in order to ensure that our trading strategy complies with the regulations. All of these tests will increase the amount of time it takes to calculate. If we have a lot of orders, we'll have to conduct all of these verifications in sequence for one instrument. If the program isn't fast enough, the orders will take longer to process. The more users you have, the more scalable the trading system must be.

These variables change the way you think about the trading system you're going to create. Let's discuss the design of a simple trading system in the following section.

Trading system architecture

The following schema represents the trading system architecture. On the left part of this diagram, we can see the venues. Venues are a more generic term for any platform matching buy and sell orders for the securities and/or derivatives of multiple parties. In other words, a venue can be a trading exchange, an ECN, an aggregator, or a bank. Trading systems communicate with venues to collect price updates from all the participants and send orders. To do so, a trading system needs a piece of software called Gateways, which will ensure the communication between the trading system and the venues. The Book Builder will build the limit order book from the data collected from Gateways. Finally, the Strategy will send the order to the venues through the Order Manager. The Order Manager is responsible for collecting all the orders coming from the strategies of the system and keeps track of the life cycle of the orders. All of these components are a part of the critical path of sending orders to the market.

Figure 2.1 – Trading system architecture design

Additionally, we observe other less critical services, such as command and control, in charge of starting the components of the system. The viewers are critical in algorithmic trading because they will give you a status of all the components of the system, the orders and trades, and the metrics you consider important to monitor your trading strategies. Algorithmic trading automates trading. Therefore, it is important to keep track of the health of your trading system and trading strategies. It is particularly important to understand here that in HFT trading, a few microseconds can end up in colossal losses. Having viewers and user interfaces capable of efficiently reporting alerts is critical.

In the next section, we will talk more in-depth about the critical components of a trading system.

Gateways connecting to trading exchanges

Gateways are the components of a trading system communicating with the exchange and the trading system. They are essential because they are the greediest in terms of execution time. By design, they must get data from the network and provide this data to the rest of the system. This operation is demanding in terms of system resources and operating system. Price updates are collected by a trading system, which then transmits orders on your behalf. To do so, you must first code all the procedures you would do if you were trading without a trading system. If you want to make money by purchasing low and selling high, you must first decide what product you will trade. You should get the order from the other merchants after you have chosen these products.

The other traders will inform you of their willingness to trade a financial asset by identifying the size, price, and quantity. You can pick the trader with whom you will negotiate a transaction once you have received enough orders for the product you wish to trade. The price of this item will influence your selection. If you plan to resell this item in the future, you will need to acquire it for a cheap price. When you reach an agreement on a price, tell the other trader that you want to buy at the listed price. You will then own this product when the transaction is completed.

Data collection

Gateways collect price updates from the trading venues you have chosen (exchanges, ECNs, and dark pools). This component (shown as a Gateway in the following figure) is one of the most important in the trading system. This component's job is to get the book for the instruments from the exchange into the trading system. This component will be connected to the network and will be able to communicate with it via exchanges receiving and sending streams.

The position of the trade system's gates is depicted in the following diagram. They are the trading system's inputs and outputs:.

Figure 2.2 – Gateways in charge of collecting price updates and sending orders

The following points are depicted in the preceding diagram:

• Traders, exchanges, ECNs, and dark pools are represented by the venues.
• Different protocols might be used to connect the venues (they are represented using arrows).
• Wires, wireless networks, the internet, microwaves, and fibers are all options to transfer data. In terms of speed, data loss, and bandwidth, each of these network media has its own set of characteristics.
• The arrows for price updates and orders are bidirectional because we can have data sent to/received from the venues.
• To begin receiving pricing updates, the gateway will establish a network connection with the venue, verify itself, and subscribe to a certain instrument (we will explain this part in the next section).
• The order-processing gateway also receives and sends communications. When an order is placed, it is forwarded to the venue over the network.
• An acknowledgment of this order will be sent if the venue receives it. A message will be issued to the trading system after this order meets a matching order. If the venue does not receive an order, the acknowledgment will not be sent. It is up to the trading system to declare that an order has timed out. In this situation, a trader will need to intervene and check the problem that occurred in the system.

Making a trading system trade with exchanges

A trading system contains a number of functional components that are responsible for trading and risk management, as well as monitoring the trading process on one or more exchanges. A trading strategy becomes a part of the trading system once it is coded. As input, you'll need price data, and as output, your orders. This will give out trading signals. We'll need gateways to finish this flow because they're the most important components.

The functional components of a trading system, the gateway's interface, and the trading system's interaction with the outside world are depicted in the following diagram:

Figure 2.3 – The functional components of a trading system

The gateways gather and send orders based on pricing and market reactions. Their primary function is to establish a link and translate data received from the outside world into the data structure needed by the trading system.

The following points are depicted in the preceding diagram:

• This trading plan will be on your machine when you apply your trading strategy. The trade will take place on a different computer.
• Because these two devices are located at different locations, they must connect over a network.
• The methods of communication employed by the system might vary depending on its location.
• A single wire will be utilized if the trading system is collocated (the machines are in the same facility), which will minimize network latency.
• The internet might be another mode of communication if we adopt a cloud solution. The communication will be substantially slower than with a direct connection in this instance.

Examine the following figure, which displays the communication that occurs between the gateways:

Figure 2.4 – Communication between exchange and trading system

We understand the following points from the preceding diagram:

• When we examine the communication handled by the gateways more closely, we can see that the venues may use various protocols.
• To transform protocols into trading system data structures, gateways will need to be able to process a variety of protocols.

We learned about how a trading system connects to trading exchanges. We will now talk about how communication can happen and which protocol we are using for receiving market updates and sending orders.

Examining the API for communication

The rules of communication between machines are defined by network protocols. They specify how these devices will be identified on the network and how they will communicate with one another. The User Datagram Protocol (UDP) and Transmission Control Protocol (TCP) over Internet Protocol (IP) protocols are used in trading finance. Furthermore, we employ a software protocol that specifies how to send an order and obtain a price update. At the software level, the communication API will establish communication rules. The communication API is provided by the entity with which you wish to transact. This document contains all of the messages you will need to get price updates and place orders.

Examples of trading API documents can be found at https://en.wikipedia.org/wiki/List_of_electronic_trading_protocols. Before we go into the trading API, let's go over some networking fundamentals.

The fundamentals of networking

The network is responsible for allowing computers to connect with one another. To share data, networks require a physical layer. For a network to achieve a certain level of speed, dependability, or even security, selecting the appropriate medium (communication layer) is crucial. We use the following terms in trade finance:

• Wire: Electrical currents with a narrow bandwidth.
• Fiber: More bandwidth.
• Microwave: It's simple to set up and has a lot of bandwidth, but it's susceptible to storms.

Depending on the sort of trading technique you choose, the media will change. In the Open Systems Interconnection (OSI) model (developed in Chapter 5, Networking in Motion), selecting the appropriate medium is part of the first layer of the network. The physical layer is the name given to this layer. There are six more layers on top of this one that describe the sort of communication.

Like most of the communication, finance is also using IP. This is part of the ISO model's network layer. This IP establishes the rules for network packet routing. The transport layer is the final layer we'll discuss. TCP and UDP are the two most well-known protocols in banking. These two procedures are diametrically opposed. TCP is a protocol that allows two machines to communicate with each other. All messages sent initially will be delivered first. UDP lacks a means for determining whether network packets were received by the network. All of the exchanges will use either TCP or UDP as their protocol.

In Chapter 5, Networking in Motion, we will go deeper into the study of these protocols. Let's learn about order book management in the following section.

Order book management

The primary goal of data handling is to copy the limit order book from the venues into your trading system. The book builder will be in charge of gathering the pricing and categorizing them for your tactics in order to integrate all of the many books you obtain.

The pricing changes are transformed by the gateway and then passed to the book builder, as shown in the following diagram. The book builder will use the books that the gateways have received from the venues, as well as gather and sort any pricing changes:

Figure 2.5 – Book builder getting price updates from Gateway IN

In the following diagram, we use an example of an order book for a given financial product. The order book will contain two parts, one for the bids and one for the offers. For each part, we will store the orders represented by the venue, the volume, and the price. Every venue will send its own order books. The goal of the book builder is to create a book taking into account the three books coming from each venue. The data represented in this diagram is artificial:

Figure 2.6 – Trading system building the book out of three different venues

The following is depicted in the diagram:

• You can see that there is an order for each row in these books.
• For example, a trader on Venue 1's offer list is ready to purchase 1,000 shares for $1.21. On the other hand, there's a list of people who are eager to sell.
• The offer (or ask) price will almost always be greater than the bid price. Indeed, it would be far too simple to profit if you could purchase for less than you could sell.
• The book builder's job is to collect the three books from the three places that the gates have collected. The book builder organizes and sorts the three books.

We have learned how the trading system gets price updates and how to build a limit order book. We will now explain in detail the different functionalities of the order book.

Order book considerations

A limit order book collects all price updates (orders) and arranges them in a way that makes the trading strategy's work easier. Exchanges utilize the order book to keep track of bids and offers. When trading, we receive the book from the exchange in order to figure out an indication of the asset prices, which prices are the best, or just to get a sense of the market. We'll have to utilize the network to convey changes to the exchange book because the exchange is on another platform/server. We have two options for doing so:

• The first way entails sending the entire book. Sending the full book is consuming in terms of time. Indeed, if we have large trading exchanges (such as NYSE or NASDAQ), there are millions of orders sent within one second. If each time the exchange receives a new order, the full book is sent, the network will be saturated and it will take too long to send price updates.
• The second way will send a full snapshot of the book as described previously. Then, the exchange will send incremental updates. The book is a critical part of the trading system since it will provide the trading strategy with the information to decide when to send an order or not. An order book contains bids and orders that are presently on the exchange. When a price update is sent to our trading system, the other market participants receive the same update concurrently. All the other market participants can also decide to run after this price update. When the exchange receives many orders (explained in detail in Chapter 3, Understanding the Trading Exchange Dynamics), the orders received first will be executed first. That's why the book plays a large part in the latency and all the operations of the book must be optimized.

For the life cycle of the orders, we will need to handle the following operations:

• Insertion: An insertion is a book entry that adds a new order to the book. This should be a quick operation. Since we must sort the bids and the offers for any received price updates, the method and data structure we choose for this operation are crucial. To insert a new order, we'll need to use a data structure with an O(1) or O(log n) complexity.
• An amendment/modification will use the order ID to seek the order in the book. This operation should have the same level of difficulty as the insertion.
• Cancelation: Using the order ID, a cancelation allows an order to be withdrawn from the book.

The data structure chosen and the method connected with it have a significant impact on performance. If you're creating an HFT system, you'll need to make the appropriate decisions. The order book that we are implementing in HFT is called an order-based book. Since this is a critical component in the system, it is very important to consider the complexity of the execution of this book.

An efficient data structure to model the order book must ensure the following:

• Constant look-up, fast quantity update: An order book stores a lot of orders for one given instrument. Large exchanges can get millions of orders per second. Because we will have a growing number of orders in this book, it is important to keep a constant look-up time for order IDs. We will have to look up order IDs millions of times per second to update these orders. Additionally, we will need to retrieve orders with the best price rapidly. Looking up orders by price cannot be linear in terms of complexity. Therefore, we will use fast indexing (having a logarithmic time to find an order for a specific price).
• Iteration in order of prices: When buying or selling large quantities, we may need to find many orders to reach a given volume. In this situation, we will start with the best price, then we will go to the second-best price, and we will keep on going like this. In this situation, it is also critical for the execution speed to reach the next best price with very low complexity.
• Retrieving best bid and ask in constant time: Since we will mainly work with the best prices, we need to have a data structure capable of returning the best orders for bids and offers.

We will need to take into account the following considerations:

• Organize order identifiers to order information in a huge associative array (for C++, it could be a std::unordered map or std::vector).
• The order metadata includes references to the order book and price-level it belongs to, therefore, after checking up the order, the order book and price-level data structures are only a single dereference away. When using an Order Execute or Order Reduce action, having a reference to the price allows for an O(1) decrease. You may preserve pointers to the next and previous orders in the queue if you wish to keep track of time priority as well.
• Because the majority of changes occur near the inside of the book, employing a vector for each book's price levels will result in the quickest average price lookup. Because the desired price is usually just a few levels from the interior and a linear search is simpler on the branch predictor, optimizer, and cache, searching linearly from the end of the vector is, on average, quicker than a binary search. Of course, pathological orders can exist outside of the book, and an attacker might, theoretically, transmit a large number of updates at the end of the book to slow down your implementation. In reality, however, this usually yields a cache-friendly, almost O(1) implementation for insert, lookup, update, and delete (with an O(N) memcpy in the worst-case scenario).
• This has O(1) best-case behavior for insertion, lookup, deletion, and update, with extremely low constants. Unfortunately, because of the cache, TLB, and compiler-friendliness, you may achieve O(N) worst-case behavior with a low probability and still have extremely excellent constants. It's also very quick, almost ideally so, when it comes to Best Bid and Offer (BBO) updates, which is what you're normally after.

By explaining how a book should be implemented in HFT, we can already see why we need to develop an in-depth knowledge of computer operating systems and programming. In the following section, we will cover in depth the use of these components to achieve the best performance.

Strategy making decisions on when to trade

The trading strategy is the system's brain. This is where we will put our algorithm that represents our trading concept into action. Let's take a look at the diagram:

Figure 2.7 – Trading strategy receiving data from the book builder to make a decision on when to trade

The diagram shows that the trading strategy is divided into two main components, signal and execution:

• The signal component of this strategy only focuses on generating signals. However, having the intention (a signal) does not guarantee you will get the liquidity you are interested in. For instance, in HFT, it is highly likely your orders will be rejected because of the speed of your trading.
• The execution part of the strategy will take care of handling the response from the market. This part decides what to do for any responses from the market. For instance, what should happen when the order is rejected? You should continue trying to get equivalent liquidity and another price.

In this section, we learned about the trading strategy; we will now learn all about the order management system (OMS) being the last critical piece of the trading system.

The OMS

The OMS gathers orders submitted from the strategy. The order life cycle is tracked by the OMS (creation, execution, amendment, cancelation, and rejection). The OMS collects trading strategy orders. If an order is invalid or malformed, the OMS may reject it (too large a quantity, wrong direction, erroneous prices, excessive outstanding position, or order type not handled by the exchange). The order does not leave the trading system when an error is identified in the OMS. The rejection occurs sooner. As a result, the trading strategy can react more quickly than if the order is rejected by the exchange. Let's have a look at the following figure, which depicts the OMS's key features:

Figure 2.8 – Order manager collecting all the orders in a trading system

Let's now discuss the critical components of a trading system.

Critical components

A trading system's key components include gateways, a book builder, strategies, and an OMS. They bring together all of the capabilities you'll need to get started in trading. We calculate a trading system's performance in terms of speed by aggregating the processing times of all important components. When a price update enters the trading system, we start a timer, and when the order generated by this price update leaves the system, we terminate the timer. This period is known as the tick-to-trade or tick-to-order period.

The OMS gathers orders submitted from the strategy. The order life cycle is tracked by the OMS (creation, execution, amendment, cancelation, and rejection). The OMS collects trading strategy orders. If an order is invalid or malformed, the OMS may reject it (too large a quantity, wrong direction, erroneous prices, excessive outstanding position, or order type not handled by the exchange). The order does not leave the trading system when an error is identified in the OMS. The rejection occurs sooner. As a result, the trading strategy can react more quickly than if the order is rejected by the exchange.

Non-critical components

Non-critical components are those that aren't directly related to the choice to submit an order. They change settings, collect data, and report this data. When designing a strategy, for example, you'll have a set of parameters that you'll need to alter in real time. You'll need a component that can transmit data to the trading strategy component. We'll use a component called command and control for that.

Command and control

The link between traders and the trading system is known as command and control. It might be a command-line system or a user interface that receives orders from traders and routes them to the necessary components. Take a look at the following diagram:

Figure 2.9 – User interface for a trading system

We covered the command and control service responsible for interacting with all the trading system components. We will now see the remaining functions of a trading system.

Services

Additional components may be added to the trading system. We will talk about the following components (it is not an exhaustive list):

• Position server: This keeps track of all the trades. It updates the positions for all the traded financial assets. For instance, if a trade is made for 100,000 EUR/USD at a price of $1.2, the notional position will be $120,000. If a trading system component needs the position amount for EUR/USD, it will subscribe to the position server for getting position updates. The order manager or the trading strategy may want to know this information before allowing an order to go out. If we want to limit the position to $200,000 for a given asset, another order to get 100,000 EUR/USD will be rejected.
• Logging system: This gathers all the logs from the components and will write a file or modify a database. A logging system helps with debugging, figuring out the causes of issues, and also just reports.
• Viewers (read-only user interface view): These display the views for trading (such as positions, orders, trades, and task monitoring).
• Control viewers (interactive user interface): These provide a way to modify parameters and start/stop components of the trading system.
• News server: This gathers news from many news companies (such as Bloomberg, Reuters, and Ravenpack) and provides this news in real time or on demand to the trading system.

This section covers the critical and non-critical components of a trading system. We will now conclude this chapter by summarizing what we have learned.

Summary

We learned how to create a trading system in this chapter. The trading system we created includes all of the necessary components for you to design a trading system and start trading.

It takes years to learn how to construct a trading system. Because of the difference between the asset classes, it is likely that you will become an expert in one asset class rather than another. We created the bare minimum of features that a trading system should have. We must learn how to link this component to a trading system in order for it to be completely functioning.

In the following chapters, we will explain in detail how trading systems should be implemented and especially related to the operating system and the hardware. In the next chapter, we have more pieces of knowledge to learn about the trading exchange.