Knock‐knock.

• —Who is there?
• —Bot.
• —Bot who?
• —Bot and sold, it's a stat‐arb world.

Do you wonder why the markets have changed so much? Where's it all heading? How will it affect you? You are not alone. Today's markets are very different from what they used to be. Technological advances morphed computers and infrastructure. Changes in regulation allowed dozens of exchanges to coexist side by side. The global nature of business has ushered in round‐the‐clock deal making. All of this has created stratospheric volumes of data. The risks that come along with automated trading in real‐time are numerous. Now, the inferences from these data allow us to go to previously untapped depths of markets and discover problems and solutions that could not even be imagined 20 years ago.

Do you remember Bloomberg terminals? If so, you are reading this book not so long after it was written. JP Morgan's January 2016 announcement “to pull the plug” on thousands and thousands of Bloomberg terminals is a leading example of the sweeping disruption facing investment managers. Billion‐dollar hedge fund Citadel followed suit on August 16, 2016, by announcing that it was taking on Symphony messaging as Bloomberg's replacement. Symphony, who? Many still struggle to wrap their head around the situation, with social media platforms like LinkedIn buzzing with discussions about pulling the plug on traditional sources of market data. Yet, here is fact: The competition is not sleeping, but working hard. And now, the competition is so strong that Bloomberg, Thomson Reuters, and others may end up in significant financial peril if they ignore fintech. Is your company also oblivious to changes in innovation?

The unfortunate truth is that many established firms are completely unprepared for the fast train of innovation currently passing them by. Old, manual procedures may have been fine in the past, but with innovation sweeping through, risk management executives have to be ready to see established operating models and platforms go out the door as newer, untried approaches take their place.

Consider the investment advisory industry. Reliance on charming brokers to seduce ever‐dwindling pools of clients into paying for their commissions and overhead expenses remains the business model for some firms. At the same time, a number of well‐established startups deliver cutting‐edge portfolio‐management advice to investors right over the Internet, with some charging as little as $9.95 per month.

Global banks like Barclay's and Credit Suisse have exited the US wealth management arena while at the same time hundreds of millions of dollars in venture funding have been channeled to fintech startups working to streamline financial advice and beyond.

The bet has been wagered that new innovative and cost‐efficient business models are here to stay. Innovation can take the form of a completely new approach to conducting business or through advances in the information used for the existing way of conducting business. As an illustration, while many finance professionals are still debating market structure and whether a new exchange will help people avoid high‐frequency traders, companies like AbleMarkets deliver a streaming map of high‐frequency trading activity directly to subscribers' desktops, leaving nothing to chance and helping to significantly improve trading performance across all markets. Similar innovations are going on in insurance, risk management, and other aspects of financial services, and firms that are not up to par on what's going on are at a significant risk of failure.

EVERYONE IS INTO FINTECH

Have you ever missed opportunities in the markets because you felt you were disrupted? We have been in a unique and fortunate position to be immersed in the heart of fintech innovation and to observe first‐hand the extent of what is becoming a true disruption to businesses that, in turn, disrupted financial markets in the late 1970s and 1980s. Think of this as Finance 3.0. The possibilities are endless, and the new players are already embedded in most facets of traditional finance. These new players are not boiler rooms—most founders have advanced degrees and the most recent scientific innovations at their fingertips.

According to the Conference Board, investment in financial technology, trendily abbreviated into fintech, grew by 201 percent in 2014 around the world. In comparison, overall venture capital investments have only grown by 63 percent. The digital revolution is well underway for banks, asset managers, and customers. The impact on the financial institutions from the many startups that are trying unproven ideas is beginning to crystallize. Venture capitalists are betting that the once‐stodgy financial industry is about to experience a considerable transformation.

The pace of change for the financial world is speeding up, and startups and venture capitalists are hardly alone in the fintech craze. Apple, Amazon, and Google, among others, have already launched financial services platforms. They have aimed at niches where they can establish a strong position. Threatened by these new entrants, traditional financial stalwarts are hearing the pitch: Adapt to the new environment or perish.

Banks are launching their own internal funds and hiring significant numbers of developers for internal builds. Why now? In his latest annual letter to shareholders, Jamie Dimon, CEO of JPMorgan Chase, wrote that “Silicon Valley is coming.” While this statement went unnoticed by the news, it reflects the torrent of venture capital flowing into fintech. Estimates by the Economist, shown in Figure 1.1, suggest that 2014 was the watershed year for fintech startups.

The Current State of Big Data Finance

What is big data finance? For many financial practitioners, big data is still just a buzzword, and finance is business as usual. However, looking at the hottest‐financed areas of business, one uncovers particular trends that move beyond buzz into billion‐dollar investments. According to Informilo.com, for instance, the fastest‐growing areas of big data in finance in 2015 were:

• Payment services
• Online loans
• Automated investing
• Data analytics

Each of these areas, in turn, translates into automation. The payment services businesses, such as TransferWise, harness technology to commoditize counterparty risk computations. Counterparty risk is a risk of payment default by a money‐sending party. Some 20 years ago, counterparty risk was managed by human traders, and all settlements took at least three business days to complete, as multiple levels of verification and extensive paper trails were required to ensure that transactions indeed took place as reported. Fast‐forward to today, and ultra‐fast technology enables transfer and confirmation of payments in just a few seconds, fueling a growing market for cashless transactions.

Similarly, the loan markets used to demand labor‐intensive operations. Just 10 years ago, the creditworthiness of a bank's business borrowers were often judged during a round of golf and drinks with the company's executives. Of course, quantitative credit‐rating models such as the one by Edward Altman of New York University have proved invariably superior for predicting defaults over most human experts, enabling faster online loan approvals. Online loan firms now harness these quantitative credit‐modeling approaches to produce fast, reliable estimates of credit risk and to determine the appropriate loan pricing.

Can anyone issue loans over the Internet or facilitate payments? According to recent industry reports, yes, the founders of many loan startups that originated during the credit squeeze of 2009—have little prior background in lending.

The key issues in lending are (1) having capital to lend, and (2) estimating credit risk of the borrowers correctly. The pricing of the loan service, interest, is then a function of the credit rating. If and when a borrower defaults, the loan should be optimally paid out from the interest. More generally, the average loan interest should exceed the average loan amount outstanding in order for the lender to make money.

The lending business is central to banking, and banks have had a near monopoly over the lending business for a very long time. New approaches to lending have emerged that compete with banks. Banks fund loans with deposits, whereas peer‐to‐peer lending is funded by investors. The leading players in this new approach to lending are the LendingClub and Prosper in the United States and Funding Circle and Zopa in the United Kingdom. In 2015, Zopa passed the Great Britain pound (GBP) 1 billion mark. Zopa's growth is shown in Figure 1.2.

With peer‐to‐peer lenders prospering with their new model, not only have banks noticed, but in some cases, started to acquire the upstart companies. SunTrust Bank acquired FirstAgain in 2012, later rebranding it LightStream.

New technologies are making their presence felt in wealth management as well. The topics of the robo‐advising and a broad group of analytics are the most diverse and least exact. Robo‐advising takes over the job of traditional portfolio management. The idea behind robo‐advising is that a computer, programmed with algorithms, is capable of delivering portfolio‐optimized solutions faster, cheaper, and at least as good as its human counterparts, portfolio managers. Given a selected input of parameters to determine the customer's risk aversion and other preferences (say, the customer's life stage and philosophical aversion to selected stocks), the computer then outputs an investing plan that is optimal at that moment.

Automation of investment advice enables fast market‐risk estimation and the associated custom portfolio management. For example, investors of all stripes can now choose to forgo expensive money managers in favor of investing platforms such as Motif Investing. For as little as $9.95, investors can buy baskets of ETFs preselected on the basis of particular themes. Companies such as AbleMarkets.com offer real‐time risk evaluation of markets, aiding the judgment of market‐making and execution traders with real‐time inferences from the market data, including the proportion of high‐frequency traders and institutional investors present in the markets at any given time.

Not only are the changes aimed at managing the portfolios of the retail investor but also in the way companies are raising capital from these same investors. Crowdfunding has become a popular way for ideas to turn into projects with real funding. Kickstarter is one of the more popular sites.

And companies like Acuity Trading, Selerity, and iSentium are trying to harness data from platforms like Twitter to give an indication of investor “sentiment,” which, in turn, gives them an idea of which way to trade.

The information‐driven revolution is changing more than the investing habits of individuals. Institutional investors are increasingly subscribing to big data information sources, the more uncommon or uncorrelated is the data source, the more valuable it is. Each data source then drives a small profit in market allocations, and, when combined, all of the data sources deliver meaningful profitability to the data acquirers. This uncommon‐information model of institutional investing has become known as Smart Beta or the Two Sigma model, after the hedge fund that grew 400% in just three years after the model adoption.

Underlying all these developments are the advances in scalable architecture and data management. Ultra‐fast computation and data processing are critical enablers of other innovative forms of financial research and investing. Several companies have lately generated multibillion‐dollar valuations by providing analytics in the software‐as‐a‐service (SaaS, pronounced “sass”). For instance, Kensho is delivering the power of human‐language queries in customers' data, which have been rolled out across Goldman Sachs.

Risk managers face a daunting challenge. Finding a risk event is the needle in a haystack. With automation and big data, the haystack becomes a mountain, and that mountain is virtual. The potential to catch issues could never have been stronger, but the ways of doing so are drastically novel.

THE MILLENNIALS ARE COMING

Why is technology transforming financial services now? Where was it 20 years ago, when computers and the Internet already existed? The short answer is the millennials, a generation of young people loyal to their smart phones and technology platforms and caring little for other brands, such as those of banks. With this generation of people now in the workforce, the choices that this group of 84 million make can provide the momentum to carry change. The millennials, born between 1980 and 2000, are expected to hold $7 trillion in liquid assets by 2020.

Recent findings in the Millennial Disruption Index (MDI) paint a startling portrait of preferences so different from older generations and so aligned with corporate digital heavyweights that financial services may change further dramatically. For example, according to the MDI study, one in three millennials will switch banks in the next 90 days. Additionally, over 50 percent of the 10,000+ respondents consider all banks to share the same value proposition. In other words, millennials don't see any difference among financial institutions. With over 70 percent of respondents saying, “They would be more excited about a new offering in financial services from Google, Amazon, Apple, Paypal, or Square than from their own nationwide bank,” it is clear that change is before us. Such findings open the door for brands like Google to enter the market and build a stable business with the millennials before bringing in older generations.

Traditional banks are feeling the threats of new entrants. Apple, Google, and Amazon are now all actively participating in the financial services industry. Whether through payments, cloud infrastructure, or investments into other fintech companies, firms considered technology leaders are focusing on financial services. The technology giants have even created their own lobbying group to avoid getting mired in regulatory red tape encasing banks. (See “An Excerpt about the Silicon Valley Lobbying Entity.”)

How can this affect you? For years, financial services companies focused their investments on meeting regulatory changes or incremental improvements—automation, workflow, and so on. The essential business model went untouched. What's changing now is that new startups are bringing a Silicon Valley approach, and they are entering financial services with bold new business ideas.

The same message resonates for most investors: institutional or retail, global macro or small‐cap, trading in the dark pools or lit exchanges. The sudden demand for new technology concerns all aspects of the financial ecosystem. At least some of the demand is based on the idea that operating models need to become leaner to offer services at lower price points, utilize a labor force based all over the world, and compete with new players. While slimming their offerings makes banks less prominent, it may enable them to face the challenge of new well‐heeled Silicon Valley entrants as they get into the business of financial services.

How do you protect your company in an environment of disruptive change? How do you anticipate shocks to the markets precipitated by new dynamics at play? How do you ensure you know your customer when more and more of your company's process are moving to new platforms? These are some of the questions we explore in the following chapters.

How is the current environment different from the one, say, just 10 years ago? Today, many companies have adopted the Digital One company strategy with the idea to integrate social media, mobile technology, cheap computing power, fast analytics, and cloud data storage.

SOCIAL MEDIA

Social media alone creates change, and not just because of all the new tools connecting billions of individuals worldwide. People use social networks to gain immediate access to information that is important to them. The increased independence that people feel when they can access their networks whenever and wherever they want makes these networks a treasured part of the way they spend their day.

For investors, social media may mean wide access to a variety of information on the go. On the train and feel like learning the business model of some obscure public company? Not an issue. At the airport, but thought of investing in a specific municipal bond and need more information on the jurisdiction? Here it is. A successful fintech business has a social network that reaches investors both proactively and responsively. By offering a social experience, the business can provide traditional services in a setting that is consistent with the social network's way of navigating. Analyzing a customer's use of the social network allows a company to respond to clients in a tailored fashion, offering messages and ideas that are consistent with what the customer wants.

The implications of social media, however, go far beyond the communication and customer service experience a business can have with prospects and clients. Unlike news, social media is a powerful user‐generated forum where ideas collide, opinions are formed, and beliefs are floated, often completely under the radar of traditional media. The participants who offer the opinions often join in anonymously, concealing their identity in a degree of masquerade where they feel comfortable to disclose their thoughts honestly and passionately. The same degree of honesty is often impossible in our politically correct daily interactions, even with the nearest friends behind closed doors. The chatroom‐formed opinions then often trickle into the stock markets as people trade on their beliefs, putting their money where their mouths are.

Harvesting and interpreting social media content has thus been a boon for a range of financial businesses. Machine‐collected sentiment on specific stocks has been shown to predict intraday volatility and future returns. The AbleMarkets Social Media Index, for example, has consistently predicted short‐term volatility over the past six years, and is used by investors, execution traders, and risk management professionals.

Is all social media content created equal? As you have guessed it, this is very far from being the case. With proliferation of automatic social media tools, for instance, a lot of the content comprises “reposts” and “retweets” of information found elsewhere. This duplication of materials sometimes is worthwhile and reflects the copying party's agreement or endorsement of the original content. In many instances, however, duplicate content appears to be streamed simply to fill the informational void of a given social media participant's stream.

Another social media hazard is fake news. This may come in the form of individuals' posts or, much worse, via fraudulent posts on hijacked accounts of other users. A classic in the latter category was a Twitter post on the Associated Press account informing followers of an explosion at the White House on April 23, 2013.

Separating the wheat from the chaff in the social media space is not a job for dilettantes, and requires advanced machine‐learning algorithms. In today's market environment, where the profit margins are thin and every bit of information is valuable, correct inferences are critical and experience in dealing with various circumstances is worth a lot.

MOBILE

How is mobile affecting your business? The prevalence of mobile devices has already driven business of all shapes and sizes to offer their services through an online channel. Why are people choosing to transact over the mobile channel? Accessing a service at a convenient time without any concern of intrusions during the experience is a very powerful use case. There are no lines, no puddles to navigate on the way to the service, and the customer can jump between the transaction and doing something else as needed.

Furthermore, mobile takes instant gratification to a new level. Are you sitting on the beach, yet have a sudden urge to send money back to your parents in Canada? TransferWise will take your order right there and then. Need to apply for a loan at the same time? No problem—100 or so new apps will be at the ready to process your information and issue preapproval in a matter of minutes, if not seconds.

The ability to fulfill your latest craze or wish anywhere at any time is clearly driving much of market innovation. In response to people's 24/7 newly found ability to demand financial services, companies like the Chicago Mercantile Exchange (CME) now offer around‐the‐clock trading in selected futures. Whenever you want it, you can bet your money on the latest thought or piece of research.

Adding to the real‐time 24/7 availability of services is the proliferation of smart watches. Whereas “traditional” mobile devices may be securely packed out of site, say, in your back pocket, the wrist gadget is much harder to ignore. And the millennials reportedly love it. In response, the development of smartwatch applications devoted exclusively to all things financial has exploded. According to Benzinga, there are at least 22 fintech apps coming to Apple Inc.'s smartwatch (see “Financial Services Applications Being Developed for the Apple Smartwatch”). And there is no mention of Bloomberg or Thomson Reuters on this list. Are they wise to stay away from the smartwatch, or will someone else just step in and replace them altogether?

CHEAPER AND FASTER TECHNOLOGY

What would it mean to you if your technology costs dropped? Over the past 30 years, the costs of computing have been falling steadily and, sometimes, exponentially. Some 30 years ago, a computer of decent processing power cost as much as US$20 million and was so big that it required its own highly air‐conditioned room. Today, a machine with comparable specifications can be picked up at a local Best Buy for about $200, and it is about the size of a high school yearbook. The decline in the costs of computer technology has been driven by several factors:

1. Broadly‐based demand for fast, superior computing by retail users, such as video gamers, has created a business case for a larger‐scale manufacturing of computers, reducing costs.
2. Investments in research and development by Silicon Valley consumer‐oriented companies, such as Google and Apple, have resulted in faster, leaner, and more affordable solutions.
3. Overseas investments by countries such as Singapore enabled foreign production of top‐quality components at a fraction of the cost, reducing overall ticket prices of machines.

Lower costs have permeated every aspect of computing from data storage to analytic power, allowing innovations such as cloud computing to flourish.

CLOUD COMPUTING

The term cloud refers to a collection of computers, each with its separate processing and storage engines, which are interconnected and operate with a single interface. The interface is a complex computer program with built‐in intelligence to automatically distribute the workload and the storage capacity among the participating machines. The cloud enables companies to reduce their data storage and processing costs by outsourcing at least some of their infrastructure and data storage.

A great example of a successful cloud deployment is Tradier.

According to Forbes, Tradier offers a brokerage‐account management system, a trading engine, and some market data. It then hands them off to application developers who can launch their own trading platforms, mobile apps, algorithmic trading systems, or other customized features for their customers, who are traders and investors who want to play the markets their own way. Account settings and market data are based in the cloud, so customers can log in to, and trade from, any of Tradier's developer partners.

As Dan Raju, the CEO of Charlotte, N.C.–based Tradier, explains it, “Tradier has decoupled the individual brokerage account from the front‐end investing experience.” Raju believes that his firm is offering a democratic platform that gives everyone access to the same cloud‐based engine that powers retail trading. He thinks of the developers as delivering “that most innovative last mile” to the trader, while the nuts and bolts of account management, tax reporting, funding, and so on are handled by Tradier.

BLOCKCHAIN

Blockchain, a technology underlying Bitcoin and gaining an increasingly wider acceptance in financial settlement, is an example of a cutting‐edge technology made possible by the cloud. The key idea underlying blockchain is an algorithm allowing users to simultaneously update the cloud database while maintaining the database's integrity, all in real time. Applied to financial trading, blockchain enables brokers and other institutions that handle their orders and money to reconcile their ledgers in real time. In other words, blockchain shortens the settlement procedures from T + 3 and T + 1 (still a standard in many financial instruments today) to real time. Shorter settlement times, in turn, allow for real‐time margin calculation and lower margin‐related risks. These developments, once adopted, will lead to even more real‐time trading.

This won't happen overnight. The complexities involved in moving all trading toward real time are nontrivial. Topics like margin, securities lending, and over‐the‐counter (OTC) trading introduce time‐consuming administrative procedures or custom trades that are not perfectly suited to the standardized type of blockchain discussed at this time.

Of course, the value of blockchain extends far beyond financial settlement. It is a tool that allows multiple parties to do business together ensuring reliability and at the same time without the threat of corrupting data. The financial businesses that are likely to be affected by blockchain technology require real‐time electronic negotiations, such as over‐the‐counter trading, loan origination, and any kind of workflow that was historically done slowly due to the high degree of error and the complexity of transactions. In short, before blockchain, many tasks had to be executed by one party at a time to prevent corrupting data. With blockchain, many parties can do tasks at the same time without worrying about possible overwrites, miscommunications, and so on.

FAST ANALYTICS

TransferWise and loan‐issuing apps did not emerge as a function of an ability to quickly send requests on the go. Beneath every successful money transfer and loan approval is a complex analysis that determines the risk of each operation.

At the core of all the super‐fast information sharing is data analytics. Take, for instance, any near‐instantaneous loan approval process. All loans are subject to credit risk—the risk that the loan is not repaid on time, if at all. Typically, the higher the probability that the loan is repaid in full and on schedule, the lower are the interest rates the lender needs to charge the borrower to make the transaction worthwhile. The reverse also holds: The higher is the probability that the borrower defaults, the higher are the rates the lender needs to charge to compensate for the risk of a default. The creditworthiness of the borrower can be forecasted using various factors, of which free cash flow and its relationship to the existing short‐term and long‐term debt, as well as other factors from Edward Altman's model, are critical. The ability to gather and process the required data points in real time are making the here‐and‐now loan approvals possible.

In general, risk, to many financial practitioners, has implied a multiday Monte Carlo simulation, something impossible to accomplish in a matter of hours, let alone seconds. Now, with new technologies, über‐fast processing of data is not only feasible, it is already in deployment in many applications.

How does data processing accelerate over time? Several applications running atop cloud architecture help dissect vast amounts of data faster than a blink of an eye. MapReduce was a first generation of fast software that allowed data mining extensive volumes of information and helped propel Google Analytics to its current lead. Still, newer, faster applications are here. Spark, an application that also runs on top of a cloud architecture, outperforms MapReduce and delivers lightning‐fast inferences through advanced management of computer resources, data allocation, and, ultimately, super‐fast computational algorithms rooted in the same technology that allows real‐time image and signal processing.

To understand why customers make decisions, companies harness the data available to them. In the past, customer segmentation studies were fixed in a point in time and used a variety of analytical approaches. Why go through this effort? By identifying types of customers who have similar tendencies to make similar decisions, a company can tailor their marketing, products, and investments. But that is the traditional approach.

With all forms of transactional and social data available and with enormously more computing power, companies can predict future behavior of clients almost at the same pace as clients are making their own decisions. For example, where will the aggressive high‐frequency traders trade in five minutes? New technologies, such as the one of several offered by AbleMarkets, can answer this question on the fly.

Traditional players need to review their technology spend and consider that while they are making incremental improvements, their clients may be evaluating a leap to an insurgent with a category‐killing new app.

Not only are startups working to provide discrete services with the likes of Google but also entire business models are being created to challenge established ways of doing business. For example, robo‐investing is a substitute for online brokers as well as full‐service brokers and financial planners. The idea has been around for a while; however, in the last five years the momentum has started to grow. According to Corporate Insights, robo‐advisers had gathered $20 billion in assets by the end of 2014, which is a small portion of the $24 trillion in retirement assets in the United States. The growth and the high‐profile venture capital funding of Betterment and Wealthfront have led players such as Vanguard to launch their own robo‐advisers. The growth of these companies is a topic the entire investment management industry is watching and the question becomes will the baby boom generation adopt this form of wealth management in their retirement or is this service geared to the millennials.

The innovation to use predictive technology is not just about consumer habits. Of course, future fintech solutions will churn through transaction history to spot trends and use that information to provide intelligent recommendations on decisions such as what credit card to pay off first, how much to put down on a home, or how to save for a new car. They'll even suggest things like whether it's better to buy or lease a car. However, the majority of changes from predictive analytics will occur at the institutional level, resulting in sweeping organizational and operational changes at most financial services.

For institutional asset managers, predictive analytics assess future volatility, price direction and likely decisions by fund managers. A pioneer in predictive analytics for investment management is AbleMarkets, which brings aggressive high‐frequency trading (HFT) transparency to market participants. AbleMarkets estimates, aggregates, and delivers simple daily averages of aggressive HFT so that professionals can improve their prediction of the market's reaction to events, assessments of future volatility, and shorter‐term price movement. It is used for portfolio management, volatility trading, market surveillance by hedge funds, pension funds, and banks.

What is different now? Computers are now involved in many economic transactions and can capture data associated with these transactions, which can then be manipulated and analyzed. Conventional statistical and econometric techniques such as regression often work well, but there are issues unique to big data sets that may require different tools. First, the sheer size of the data involved may require more powerful data manipulation tools. Advanced databases and computer languages are required for most large data sets; after all, even the latest version of Excel stops at some one million rows. What if your data set contains five billion records? Second, we may have more potential predictors than appropriate for estimation, so we need to do some kind of variable selection. A popular technique called principal component analysis does just that: it estimates clusters of properties common among the records. Those clusters next become important variables in slicing and dicing the data. Third, large datasets may allow for more flexible relationships than simple linear models. Machine learning techniques such as decision trees, support vector machines, neural nets, deep learning, and so on may allow for more selective ways to model complex relationships.

What are the old‐timers, who want to survive and thrive in the new competitive environment, to do? First, one needs to understand the lay of the new land. The borders have been redrawn, the capitals have moved, and Finance 3.0 is simply not the business it used to be.

IN THE END, IT'S ALL ABOUT REAL‐TIME DATA ANALYTICS

Much of this book is devoted to the innovation in the growing field of data analytics. In the last 20 years, finance has seen nothing short of an explosion of data. Just 20 some years ago, the only data available to investors comprised five figures reported in the long tables in the newspapers on the following day (T + 1). The data comprised daily open, high, low, close and volume for the previous trading day. No information about the market conditions beyond these numbers was available even within the banks and other market makers: by law, only the latest 21 days of intraday data were required to be handy at most institutions. Data storage was expensive, number crunching took forever, the profit margins were thick enough to avoid any additional data‐driven work.

As technology became cheaper and more sophisticated at the same time over the following decades, the market participants began reevaluating the cost–benefit equation of more data. Quant traders and portfolio managers were the first to deploy data analysis to improve financial functions in a semi‐algorithmic framework. Using mostly daily data and armed with the latest inferences from physics and other research fields, the quants sought answers to challenges associated with portfolio risk, derivatives pricing, diversification, and other issues. Their findings paved the way to modern exchange‐traded funds (ETFs), passively managed, yet actively traded indexes.

As the daily‐data field became saturated, researchers turned to intraday data. The late 1990s saw the birth of high‐frequency trading and execution algorithms, requiring a higher degree of processing speed. With Regulation Alternative Trading Systems (Reg ATS, 2000), the volume of data increased further as a number of new trading venues and exchanges came online. Regulation National Market Systems (Reg NMS, 2005) has further driven data storage and processing, by requiring the compilation of market quotes in the government's Security Information Processor (SIP) system and the following redistribution of SIP data back to trading venues. The introduction of SIP has shored up the real‐time nature of data on many exchanges and contributed a great deal to the volumes, depth, and sophistication of financial data we observe today. And the regulatory shift is chasing the data advances to their utmost frontiers. For example, the latest regulations about pre‐ and post‐trade analytics coming from MiFiD II and the intraday liquidity risk management from Basel all demand new, faster, ever more powerful data and analytics.

And the data sets are still growing. As new asset classes and new exchanges come online, trading hours extend and trading becomes more and more global, generating volumes of new data. In addition, the world of data outside of financial services has a direct influence on what is going on within the markets, and making use of this data requires storage and real‐time processing. Taken in aggregate, the news delivered by companies like Dow Jones, along with the blog posts by random individuals, and even the Internet activity collected by the data behemoths like Google, can all be used to understand and improve upon market movements. And it is all happening right this moment, while you are reading this sentence.

This book is written for investors who are interested in the impact of the latest revolution to affect finance and what that means for their decision making. The book is not heavy on the models, although references are provided, where appropriate. Instead, the book discusses at length the perceived and documented impact these disruptions will have on companies and what that will mean for the markets. With market crashes, interest rate uncertainty, and wars threatening to disrupt the market stability, it is more important than ever to have a balanced data‐driven perspective on what is really going on in today's markets.

Have you ever been concerned that the big data revolution and real‐time disruption is leaving you and your investment portfolio behind? This book seeks to close the gap in knowledge so that you can be more confident in making investing decisions going forward.

END OF CHAPTER QUESTIONS

1. What is fintech?
2. Why is fintech boom happening now?
3. What are the primary enablers of fintech innovation?
4. What are the hottest areas of fintech innovation?
5. What are the biggest risks of fintech innovation?