Complexity of the Financial Markets and Big Data

Institutional and individual investors and financial advisers are looking for advanced technologies which not only help them in making investment decisions or recommendations with more confidence, but also offer a different perspective on the financial markets. The capital markets constitute a very complex system that continuously evolves beyond established theories and thus cannot be sufficiently explained and predicted by traditional models. Moreover, market participants are overwhelmed by the increasing amounts of (big) data that needs to be digested and understood to be able to navigate through all kinds of market environments successfully.

Even the supposedly best professionals in the industry seem to fail to adapt quickly enough to technological advancements, rising competition and the constant flow of new market moving information. This can be seen from the irregularity of hedge fund returns and their declining ability to generate alpha over the last 15 years. Therefore, more complex self-learning systems are needed in order to model the financial market and adapt to the changes it continuously goes through.

AI and Deep Learning: Adaptable and Self-Learning Capital Markets Modelling and Forecasting

I Know First’s system incorporates multilayered neural networks, allowing it to model the markets without human-derived assumptions and maintain flexibility. It applies artificial intelligence and machine learning techniques to find patterns in large sets of historical stock market data and, based on these, can generate ranked forecasts. The algorithm learns from the most updated data and evolves with it. It adapts to new conditions and features and has enough learning experience and intelligence to be able to make predictions in circumstances not observed before.

This forecasting algorithm models the markets as non-stationary chaotic systems with fractal properties. It uses multi-representation, meaning multiple points of view and multiple data variants for each forecast, and data munging¹ to expose important features. Through explicit regularization processes – introducing additional information in order to solve an ill-posed problem or to prevent overfitting – and a fitness factor used to optimize the learning process, it is able to make predictions based on new data that the algorithm was not exposed to in the machine learning process. Moreover, principal component analysis is used to identify important (information-carrying) inputs and separate them from unimportant ones. It makes the data easy to explore, visualize and learn.

A genetic adaptable algorithm² is then constantly refining the predictor pool – an assembly of predictors used to create a forecast. The genetic algorithm repeatedly modifies a population of individual predictors, selects the best ones and rejects the worst ones. When the fitness criteria are changing, it adapts to new conditions. This method for solving optimization problems is based on natural selection, inspired by biological evolution.

Partially Versus Purely AI-Driven Investing

In the context of trading and investment management, one can differentiate between two main use cases of artificial intelligence. In the first one, the main goal when applying the techniques is to perform a certain analysis of capital markets data, for example price and trading volume pattern matching, detecting various market anomalies, interpreting sentiment via natural language processing (NLP) and finding relationships in order to come up with a set of outputs that are machine learned observations and conclusions on the current market environment. The big data is effectively condensed in that case into much smaller and more meaningful pieces of information. This first level of analysis can then be interpreted and eventually turned into action through a human filter (manager) or a strictly defined set of rules external to the AI module. Important to note is that the purpose of the outputs and their future application should be clearly defined in advance, when deciding on the setup of the AI module, in order to know how to make use of what the machine has learned.

From a business point of view, the machine processed and condensed information, along with corresponding descriptions and instructions, already provides additional insight into the markets and thus can be customized and sold as a product to the end-users, who are interested in integrating these insights into their investment decision-making processes.

In the case of I Know First, the condensed information pieces are the daily updated stock market forecasts. These are represented through positive or negative signals of different strengths to assess the investment opportunity, and the predictability level for each asset and forecasting timeframe. The unique predictability indicator is a confidence/quality measure of the respective signal. It is used to identify and focus on the most predictable assets according to the algorithm, and thus achieve risk-adjusted outperformance.

The advanced algorithmic forecasting system helps to detect the best investment opportunities and monitor current portfolio holdings when integrated into clients’ investment processes. Financial institutions, such as banks, wealth management companies and advisors, as well as brokers and online trading platforms, have the flexibility to customize the forecasting universe specifically to their needs and integrate the predictions into their research and advisory process or to offer it as a trading ideas generating tool to their clients. Through additional standardized subscription-based services, the AI-based forecasts further empower individual investors who usually do not possess the resources and expertise needed in order to develop, maintain and take advantage of AI technology when managing their wealth.

AI-Powered Investment Vehicles

The above-mentioned type of “intelligence” product can be extended to an investment vehicle. In this case, clients do not use the AI-based analytics by themselves but rely on their added value being optimally utilized in the implemented vehicle, which can easily be invested in. Here, AI prediction-based portfolio construction can be used. Algorithmic forecasting indicators are utilized in the development of systematic long-biased or long/short trading and allocation strategies. Depending on the chosen risk and trading activity profile, these strategies can be used to launch hedge funds, mutual funds or structure smart-beta or actively managed exchange traded funds (ETFs).

Opportunistic, more aggressive and frequently rebalanced strategies take the most recent changes in the daily updated forecasts into account and focus on the most predictable assets which offer a higher upside.

“AI-Smart”-Beta ETFs

In contrast, smart-beta products invest in a predefined set of companies, often constituents of a certain (sector) index or fundamentally sound “high-quality” companies, where, however, the default market cap allocation is adjusted according to the opportunities discovered by the predictive AI algorithm. The rebalancing happens either with a predefined frequency or in case a significant change in the forecast exceeds a threshold. Hence, the exposure is attained with the same assets as in a passive index-tracking (sector) ETF, with the weighting being adjusted to take advantage of the AI-driven predictive capabilities.

With these algorithmic forecasting and ranking systems, investment opportunities are assessed through predictability levels, signal directions and strengths, which are generated for various timeframes for each of the assets covered.

AI-Driven Sector (ETF) Rotation Strategies

A “bottom-up” built system allows us to separately follow the individual securities’ forecasts as well as group these by sectors and industries in order to derive “macro” conclusions and trends. A higher level of allocation decisions can be made through such “aggregated” forecasts and finally implemented as a sector (ETF) rotation strategy. Here, an ETF portfolio is constructed by investing in more promising sectors – as suggested by the predictive AI system on the constituents’ level.

Autonomous, Pure AI Hedge Funds and Reinforcement Learning

So far, we have covered the application of AI and machine learning technology on large and complex data sets in order to come up with outputs that give additional insights and are then used externally by managers in investment and by advisors in recommendation processes. A logical next step is the construction of a complete and closed AI-based “analysis-to-decision-to-execution” system, with minimal to no input from the fund manager. Reinforcement learning is used to transition to trading signals, and it can be seen as the integration of supervised learning and backtests.

Whereas, in supervised learning, a fitness function is optimized in the learning process from a training data set consisting of input–output examples, reinforcement learning is analogous to how a child learns from their mistakes. A machine explores the possible actions in a given environment and learns which ones give a desired result. A reward function is used to measure the success of learning. It can be singular (one step at a time), or cumulative. The best possible decision in any given state (current portfolio) is made and executed by the algorithm based on its learning experience up to this point. Such a system is not simply identifying, but predicting and adapting to the trends while making decisions in real time going forward, whereby the manager’s discretionary control over the investment process is removed. Of course, this could lead to certain transparency and thus regulatory issues, making pure and complete AI trading less feasible for publicly traded funds.

Plurality of Uncorrelated AI and Alternative Data-Driven Strategies

A recent study conducted by Eurekahedge³ on the performance of AI/machine learning hedge funds since 2011 indicates a low or even negative correlation to trend-following strategies and the average hedge fund returns, respectively. It is evidence of the ability of AI-based technology to find patterns and relationships in the data not identified by traditional models and not used in static rule-based strategies. By exploiting the rapidly growing computing power and the increasing amounts of new data sources, one can expect an even larger diversity of uncorrelated machine learned strategies and corresponding funds launched in the future.

Alternative data – such as, for example, high-quality satellite images revealing retailers’ parking lot traffic, oil tanker and other cargo movements, number of vehicles in industrial areas and facilities, crop field colour, etc. – can have predictive value for the retail market, manufacturing sector, agriculture and more. Furthermore, advances in NLP now allow us to better understand, check for consistency and use valuable and predictive information from immense amounts of unstructured (not organized, text-heavy) data. Investor and consumer sentiment shifts can be cost-efficiently measured through machine analysis and interpretation of data extracted from different news sources, articles, reports and blogs, social networking and sharing applications. These so far insufficiently used or ignored data points, if correctly and comprehensively extracted, interpreted, analysed and traded on via AI in a timely manner, will give the advantage of being ahead of the crowd until this information is “priced into” the financial markets and generates additional alpha.

AI and the Future of Finance

Independently of applying AI to generate predictive information and trading ideas to help market participants with their active investment decisions and trading strategies, utilizing it in the construction of smarter AI and alternative data-driven allocation schemes within more passive investment vehicles, or going “all-in” by allowing the machine to also take over the decision part and learning from its experience, artificial intelligence’s role in the investment management industry is on the rise and will become industry standard. Thus, it will be difficult to stay competitive without understanding and starting to incorporate this technology now, be it in the search for new alpha sources, the detection of risk events or in order to save costs and reduce fees.