Portfolio management and trading have benefited from the application of extended learning classifiers [11:13]. The use case is the management of a portfolio of exchange-traded funds (ETFs) in an ever-changing financial environment. Contrary to stocks, exchange traded funds are representative of an industry-specific group of stocks or the financial market at large. Therefore, the price of these ETFs is affected by the following macroeconomic changes:

Let's select the value of the 10-year Treasury yield as a proxy for the macroeconomic conditions, for the sake of simplicity.

The portfolio has to be constantly adjusted in response to any specific change in the environment or market condition that affects the total value of the portfolio, and can be done referring to the following table:

The first step is to create an initial set of rules regarding the portfolio. This initial set can be created randomly, much like the initial population of a genetic algorithm, or be defined by a domain expert.

The reader is invited to initialize the population of rules with as many relevant and financially sound trading rules as possible. Over time, the execution of the XCS algorithm will confirm whether or not the initial rules are indeed appropriate. The following diagram describes the application of the XCS algorithm to the composition of a portfolio of ETFs, such as VWO, TLT, IWC, and so on, with the following components:

The agent responds to the change in the allocation of ETFs in the portfolio by matching one of the existing rules.

Let's build the XCS agent from the ground.

There are three types of data that are manipulated by the XCS agent:

The XcsAction class was introduced for the evaluation of the genetic algorithm in the Trading signals section in Chapter 10, Genetic Algorithms. The agent creates, modifies, and deletes actions. It makes sense to define these actions as mutable genes, as follows:

class XcsAction(val sensorid: String, val target: Double)(implicit val discr: Discretization) extends Gene(sensorid, target, EQUAL)

The XcsAction class has the identifier of the sensor, sensorId, and the target value as parameters. For example, the action to increase the number of shares of ETF, VWO in the portfolio to 80 is defines as follows:

Val vwoTo80 = new XcsAction("VWO", 80.0)

The only type of action allowed in this scheme is setting a value using the EQUAL operator. You can create actions that support other operators, such as += used to increase an existing value. These operators need to implement the operator trait, explained in the Trading operators section in Chapter 10, Genetic Algorithms.

A discretization instance has to be implicitly defined in order to encode the target value.

Finally, the XcsSensor class encapsulates the sensorId identifier for the variable and value of the sensor, as shown here:

case class XcsSensor(val sensorId: String, val value: Double)
val new10ytb = new XcsSensor("10yTBYield", 2.76)

The next step consists of defining a rule as a pair of two genes: a signal and an action, as shown in the following code:

class XcsRule(val signal: Signal, val action: XcsAction)

The rule: r1: IF(yield 10-year TB > 2.84%) THEN reduce VWO shares to 240 is implemented as follows:

val signal = new Signal("10ytb", 2.84, GREATER_THAN)
val action = new XcsAction("vwo", 240)
val r1 = new XcsRule(signal, action)

The agent encodes the rule as a chromosome using 2 bits to represent the operator and 32 bits for values, as shown in the following diagram:

In this implementation, there is no need to encode the type of action as the agent uses only one type of action—set. A complex action requires encoding of its type.

Matching a rule to a new sensor consists of matching the sensor to the signal. The algorithm matches the new new10ytb sensor against the signal in the current population of s10ytb1 and s10ytb2 rules that uses the same sensor or variable 10ytb, as follows:

val new10ytb = new XcsSensor("10ytb", 2.76)

val s10ytb1 = Signal("10ytb", 2.5, GREATER_THAN)
val s10ytb2 = Signal("10ytb", 2.2, LESS_THAN)

val r23: XcsRule(s10ytb1, act12)
val r34: XcsRule(s10ytb2, act17)
…

In this case, the agent selects the rule r23 but not r34 in the existing population. The agent then adds the act12 action to the list of possible actions. The agent lists all the rules that match the sensor: r23, r11, and r46, as shown in the following code:

val r23: XcsRule(s10yTB1, act12)
val r11: XcsRule(s10yTB6, act6)
val r46: XcsRule(s10yTB7, act12)

The action with the most references, act12, is executed. The Q-learning algorithm computes the reward from the profit or loss incurred by the portfolio following the execution of the selected rules r23 and r46. The agent uses the reward to adjust the fitness of r23 and r46, before the genetic selection in the next reproduction cycle. These two rules will reach and stay in the top tier of the rules in the population, until either a new genetic rule modified through crossover and mutation or a rule created through covering, triggers a more rewarding action on the environment.

The purpose of the covering phase is to generate new rules if no rule matches the input or sensor. The cover method of an XcsCover singleton generates a new XcsRule instance given a sensor and an existing set of actions, as shown here:

def cover(sensor: XcsSensor, actions: List[XcsAction]) (implicit discr: Discretization): List[XcsRule] = {
  actions.foldLeft(List[XcsRule]()) ((xs, act) => {
    val rIdx = Random.nextInt(Signal.numOperators)
    val signal = new Signal(sensor.id, sensor.value, new SOperator(rIdx))
    new XcsRule(signal, XcsAction(act, Random)) :: xs
  })
}

You might wonder why the cover method uses a set of actions as arguments knowing that covering consists of creating new actions. The method mutates (operator ^) an existing action to create a new one instead of using a random generator. This is one of the advantages of defining an action as a gene. The mutation is executed by one of the constructors of XcsAction, as follows:

def apply(action: XcsAction, r: Random): XcsAction = 
    (action ^ r.nextInt(XCSACTION_SIZE))

The index of the operator type, rIdx, is a random value in the interval [0, 3] because a signal uses four types of operators: None, >, <, and =.

The Xcs class has the following purposes:

The XCS algorithm is initialized with a configuration, config, an initial set of rules, population, a fitness function, score, and an input to the Q-learning policy generate reward matrix for qlLearner. The goals and number of states are extracted from the input to the policy of the Q-learning algorithm.

In this implementation, the generic algorithm, gaSolver, is mutable. It is instantiated along with the Xcs container class. The Q-learning algorithm uses the same design, as any classifier, as immutable. The model of Q-learning is the best possible policy to reward rules. Any changes in the number of states or the rewarding scheme require a new instance of the learner.