There are a multitude of unsupervised algorithms, not only for neural networks. Examples are K-means, expectation maximization, methods of moments, and so on. Such algorithms assume the entire dataset as the knowledge to be learned, so one common feature through all the learning algorithms is that they do not have an input–output relationship in the current dataset. However, one wishes to find a different meaning of these data, and that's the goal of any unsupervised learning algorithm.
Bringing this concept into the neural network context, let's take a look at an ANN and how it deals with data in an unsupervised organization.
The neural network output is considered to be an abstraction of the original data points. As opposed to the supervised learning paradigm, there is no causality between the input data points or data variables; instead, we want the neural network to derive consequent variables that would be able to give another meaning to the presented data. While in supervised learning algorithms, we usually have a smaller number of outputs, for unsupervised learning, there is a need to produce an abstract data representation that may require a high number of outputs. However, except for classification tasks, their meaning is totally different from the one presented in the case of supervised learning. Usually, each output neuron is responsible for representing a feature or a class present in the input data. In most architectures, not all output neurons need to be activated at a time; usually, only a restricted set of output neurons is fired, meaning that a neuron is able to better represent most of the information being fed as the neural input.
In this chapter, we are going to explore two unsupervised learning algorithms: competitive learning and Kohonen SOMs.
As the name implies, competitive learning handles a competition between the output neurons to determine which one is the winner. To facilitate understanding, suppose we want to train a single layer neural network with two inputs and four outputs, as shown in the following figure:
Every output neuron is then connected to these two inputs; hence, for each neuron, there are two weights.
The competition starts after the data have been processed by the neurons. The winner neuron will be the one that produces the greatest output value. One additional difference compared to the supervised learning algorithm is that only the winner neuron may update its weights, while the others remain unchanged. This is the so-called winner-takes-all rule. This intention to bring the neuron "nearer" to the input causes it to win the competition.
Considering that every input neuron i is connected to all output neurons j through a weight wij. In our case, we would have a set of weights:
Provided that the weights of every neuron have the same dimensionality of the input data, let's consider all the input data points together in a plot with the weights of each neuron.
In this chart, let's consider the circles as the data points and the squares as the neuron weights. We can see that some data points are closer to certain weights, while the others are farther but nearer to others. The neural network performs computations related to the distance between the inputs and the weights:
The result of this equation will determine how "strong" a neuron is against its competitors. The winner neuron connections are then adjusted to the neurons according to the following update rule:
Where η denotes the learning rate. After many iterations, the weights are driven so near enough to the data points that triggers the greatest output values on the corresponding neuron, that weight updates are either too small or fall in a zig-zag pattern. Finally, when the network is already trained, the chart takes another shape:
As can be seen, the neurons form centroids that surround the points capable of making the corresponding neuron stronger than its competitors.
Tip
In an unsupervised neural network, the number of outputs is completely arbitrary. Sometimes, only some neurons are able to change their weights, while in other cases, all the neurons may respond differently to the same input, causing the neural network to never learn. For these cases, it is recommended either to review the number of output neurons or to consider another type of unsupervised learning.
There are basically two stopping conditions of competitive learning:
- Predefined number of epochs: This prevents our algorithm from running for a relatively long time without convergence
- Minimum value of weight update: This prevents the algorithm from running longer than necessary