In the next section, we'll recap artificial neural networks; they can do a good job in classification tasks such as classifying handwritten digits.
Suppose we have the network shown in Figure 1:
As mentioned earlier, the leftmost layer in this network is called the input layer, and the neurons within the layer are called input neurons. The rightmost or output layer contains the output neurons, or, as in this case, a single output neuron. The middle layer is called a hidden layer, since the neurons in this layer are neither inputs nor outputs. The term hidden perhaps sounds a little mysterious—the first time I heard the term, I thought it must have some deep philosophical or mathematical significance—but it really means not an input and not an output. It means nothing else. The preceding network has just a single hidden layer, but some networks have multiple hidden layers. For example, the following four-layer network has two hidden layers:
The architecture in which the input, hidden, and output layers are organized is very straightforward. For example, let's go through a practical example to see whether a specific handwritten image has the digit 9 in it or not.
So first, we will feed the pixels of the input image to the input layer; for example, in the MNIST dataset, we have monochrome images. Each one of them is 28 by 28, so we need to have 28 × 28 = 784 neurons in the input layer to receive this input image.
In the output layer, we will need only 1 neuron, which produces a probability (or score) of whether this image has the digit 9 in it or not. For example, an output value of more than 0.5 means that this image has the digit 9, and if it's less than 0.5, then it means that the input image doesn't have the digit 9 in it.
So these types of networks, where the output from one layer is fed as an input to the next layer, are called FNNs. This kind of sequentiality in the layers means that there are no loops in it.