In their 1998 paper, Gradient-Based Learning Applied to Document Recognition, LeCun et al. introduced the LeNet architecture.
The LeNet architecture consists of two sets of convolutional, activation, and pooling layers, followed by a fully-connected layer, activation, another fully-connected layer, and finally a softmax classifier. The following diagram illustrates the LeNet architecture:
Now, let's implement the LeNet architecture with the mxnet library in our project using the following code block:
## setting the working directory
setwd('/home/sunil/Desktop/book/chapter 19/MNIST')
# function to load image files
load_image_file = function(filename) {
ret = list()
f = file(filename, 'rb')
readBin(f, 'integer', n = 1, size = 4, endian = 'big')
n = readBin(f, 'integer', n = 1, size = 4, endian = 'big')
nrow = readBin(f, 'integer', n = 1, size = 4, endian = 'big')
ncol = readBin(f, 'integer', n = 1, size = 4, endian = 'big')
x = readBin(f, 'integer', n = n * nrow * ncol, size = 1, signed
= FALSE)
close(f)
data.frame(matrix(x, ncol = nrow * ncol, byrow = TRUE))
}
# function to load label files
load_label_file = function(filename) {
f = file(filename, 'rb')
readBin(f, 'integer', n = 1, size = 4, endian = 'big')
n = readBin(f, 'integer', n = 1, size = 4, endian = 'big')
y = readBin(f, 'integer', n = n, size = 1, signed = FALSE)
close(f)
y
}
# load images
train = load_image_file("train-images-idx3-ubyte")
test = load_image_file("t10k-images-idx3-ubyte")
# converting the train and test data into a format as required by LeNet
train.x <- t(data.matrix(train))
test <- t(data.matrix(test))
# loading the labels
train.y = load_label_file("train-labels-idx1-ubyte")
test.y = load_label_file("t10k-labels-idx1-ubyte")
# linearly transforming the grey scale image i.e. between 0 and 255 to # 0 and 1
train.x <- train.x/255
test <- test/255
# including the required mxnet library
library(mxnet)
# input
data <- mx.symbol.Variable('data')
# first convolution layer
conv1 <- mx.symbol.Convolution(data=data, kernel=c(5,5), num_filter=20)
# applying the tanh activation function
tanh1 <- mx.symbol.Activation(data=conv1, act_type="tanh")
# applying max pooling
pool1 <- mx.symbol.Pooling(data=tanh1, pool_type="max", kernel=c(2,2), stride=c(2,2))
# second conv
conv2 <- mx.symbol.Convolution(data=pool1, kernel=c(5,5), num_filter=50)
# applying the tanh activation function again
tanh2 <- mx.symbol.Activation(data=conv2, act_type="tanh")
#performing max pooling again
pool2 <- mx.symbol.Pooling(data=tanh2, pool_type="max",
kernel=c(2,2), stride=c(2,2))
# flattening the data
flatten <- mx.symbol.Flatten(data=pool2)
# first fullconnected later
fc1 <- mx.symbol.FullyConnected(data=flatten, num_hidden=500)
# applying the tanh activation function
tanh3 <- mx.symbol.Activation(data=fc1, act_type="tanh")
# second fullconnected layer
fc2 <- mx.symbol.FullyConnected(data=tanh3, num_hidden=10)
# defining the output layer with softmax activation function to obtain # class probabilities
lenet <- mx.symbol.SoftmaxOutput(data=fc2)
# transforming the train and test dataset into a format required by
# MxNet functions
train.array <- train.x
dim(train.array) <- c(28, 28, 1, ncol(train.x))
test.array <- test
dim(test.array) <- c(28, 28, 1, ncol(test))
# setting the seed for the experiment so as to ensure that the
# results are reproducible
mx.set.seed(0)
# defining that the experiment should run on cpu
devices <- mx.cpu()
# building the model with the network architecture defined above
model <- mx.model.FeedForward.create(lenet, X=train.array, y=train.y,
ctx=devices, num.round=3, array.batch.size=100, learning.rate=0.05,
momentum=0.9, wd=0.00001, eval.metric=mx.metric.accuracy,
epoch.end.callback=mx.callback.log.train.metric(100))
# making predictions on the test dataset
preds <- predict(model, test.array)
# getting the label for each observation in test dataset; the
# predicted class is the one with highest probability
pred.label <- max.col(t(preds)) - 1
# including the rfUtilities library so as to use accuracy
function
library(rfUtilities)
# obtaining the performance of the model
print(accuracy(pred.label,test.y))
# printing the network architecture
graph.viz(model$symbol,direction="LR")
This will give the following output and the visual network architecture:
Start training with 1 devices
[1] Train-accuracy=0.678916669438283
[2] Train-accuracy=0.978666676680247
[3] Train-accuracy=0.98676667680343
Accuracy (PCC): 98.54%
Cohen's Kappa: 0.9838
Users accuracy:
0 1 2 3 4 5 6 7 8 9
99.8 100.0 97.0 98.4 98.9 98.2 98.2 98.7 98.2 97.8
Producers accuracy:
0 1 2 3 4 5 6 7 8 9
98.0 96.9 99.1 99.3 99.0 99.3 99.6 97.7 98.7 98.3
Confusion matrix
y
x 0 1 2 3 4 5 6 7 8 9
0 978 0 2 2 1 3 7 0 4 1
1 0 1135 15 2 1 0 5 7 1 5
2 0 0 1001 2 1 1 0 3 2 0
3 0 0 0 994 0 5 0 1 1 0
4 0 0 1 0 971 0 1 0 0 8
5 0 0 0 3 0 876 2 0 1 0
6 0 0 0 0 2 1 941 0 1 0
7 1 0 7 1 3 1 0 1015 3 8
8 1 0 6 1 1 1 2 1 956 0
9 0 0 0 5 2 4 0 1 5 987
Take a look at the following diagram:
The code ran for less than 5 minutes on my 4-core CPU box, but still got us a 98% accuracy on the test dataset with just three epochs. We can also see that we obtained 98% accuracy with both the training and test datasets, confirming that there is no overfitting.
We see tanh is used as the activation function; let's experiment and see whether it has any impact if we change it to ReLU. The code for the project will be identical except that we need to find and replace tanh with ReLU. We will not repeat the code as the only lines that have changed from the earlier project are as follows:
ReLU1 <- mx.symbol.Activation(data=conv1, act_type="relu")
pool1 <- mx.symbol.Pooling(data=ReLU1, pool_type="max",
kernel=c(2,2), stride=c(2,2))
ReLU2 <- mx.symbol.Activation(data=conv1, act_type="relu")
pool2 <- mx.symbol.Pooling(data=ReLU2, pool_type="max",
kernel=c(2,2), stride=c(2,2))
ReLU3 <- mx.symbol.Activation(data=conv1, act_type="relu")
fc2 <- mx.symbol.FullyConnected(data=ReLU3, num_hidden=10)
You will get the following output on running the code with ReLU as the activation function:
Start training with 1 devices
[1] Train-accuracy=0.627283334874858
[2] Train-accuracy=0.979916676084201
[3] Train-accuracy=0.987366676231225
Accuracy (PCC): 98.36%
Cohen's Kappa: 0.9818
Users accuracy:
0 1 2 3 4 5 6 7 8 9
99.8 99.7 97.9 99.4 98.6 96.5 97.7 98.2 97.4 97.9
Producers accuracy:
0 1 2 3 4 5 6 7 8 9
97.5 97.2 99.6 95.6 99.7 99.2 99.7 98.0 99.6 98.2
Confusion matrix
y
x 0 1 2 3 4 5 6 7 8 9
0 978 0 3 1 1 2 12 0 5 1
1 1 1132 6 0 2 1 5 11 1 6
2 0 0 1010 1 0 0 0 1 2 0
3 0 2 4 1004 0 23 1 3 9 4
4 0 0 1 0 968 0 1 0 0 1
5 0 1 0 1 0 861 2 0 3 0
6 0 0 0 0 0 3 936 0 0 0
7 1 0 6 3 0 1 0 1010 1 9
8 0 0 2 0 1 0 1 0 949 0
9 0 0 0 0 10 1 0 3 4 988
With ReLU being used as the activation function, we do not see a significant improvement in the accuracy. It stayed at 98%, which is the same as obtained with the tanh activation function.
As a next step, we could try to rebuild the model with additional epochs to see whether the accuracy improves. Alternatively, we could try tweaking the number of filters and filter sizes per convolutional layer to see what happens! Further experiments could also include adding more layers of several kinds. We don't know what the result is going to be unless we experiment!