In this experiment, we will vary the units in the dense layer, the dropout rate, and the batch size to obtain values that help us improve classification performance. This also illustrates an efficient way of obtaining suitable parameter values through experimentation. We will start by creating a TransferLearning.R file using the following code:

# Model with RESNET50
pretrained <- application_resnet50(weights = 'imagenet',
                                   include_top = FALSE,
                                   input_shape = c(224, 224, 3))
# Flags for hyperparameter tuning
FLAGS <- flags(flag_integer("dense_units", 256),
               flag_numeric("dropout", 0.1),
               flag_integer("batch_size", 10))

# Model architecture
model <- keras_model_sequential() %>% 
         pretrained %>% 
         layer_flatten() %>% 
         layer_dense(units = FLAGS$dense_units, activation = 'relu') %>% 
         layer_dropout(rate = FLAGS$dropout) %>%
         layer_dense(units = 10, activation = 'softmax')
freeze_weights(pretrained)

# Compile
model %>% compile(loss = "categorical_crossentropy",
                  optimizer = 'adam',
                  metrics = 'accuracy')

# Fit model
history <- model %>% fit(trainx,
                         trainy,
                         epochs = 5,
                         batch_size = FLAGS$batch_size,
                         validation_split = 0.2)

In the preceding code, after reading the pretrained model, we declare three flags for the parameters that we want to experiment with. Now, we can use these flags in the model architecture (dense units and dropout rate) and in the code for fitting the model (batch size). We have reduced the number of epochs to five and for the optimizer, while compiling the model, we retain adam. We will save this R file, which we'll call TransferLearning.R, on the desktop of our computer.

The code for running this experiment is as follows:

# Set working directory
setwd('~/Desktop')

# Hyperparameter tuning
library(tfruns)
runs <- tuning_run("TransferLearning.R", 
                   flags = list(dense_units = c(256, 512),
                                dropout = c(0.1,0.3),
                                batch_size = c(10, 30)))

In the preceding code, we can see that the working directory is set at the location of the TransferLearning.R file. Note that the output from this experiment will be saved in this directory too. For running the hyperparameter tuning experiment, we will use the tfruns library. For the number of units in the dense layer, we will try 256 and 512 as the values. For the dropout rate, we will experiment with 0.1 and 0.3. Fnally, for the batch size, we will try 10 and 30. With three parameters, each being tried at two values, the total number of experimental runs will be 2³ = 8.

An extract from the results that were obtained from this experiment is as follows:

# Results
runs[,c(6:10)]
Data frame: 8 x 5 

  metric_val_loss metric_val_acc flag_dense_units flag_dropout flag_batch_size
1          1.1935         0.7525              512          0.3              30
2          0.9521         0.7725              256          0.3              30
3          1.1260         0.8200              512          0.1              30
4          1.3276         0.7950              256          0.1              30
5          1.1435         0.7700              512          0.3              10
6          1.3096         0.7275              256          0.3              10
7          1.3458         0.7850              512          0.1              10
8          1.0248         0.7950              256          0.1              10

The preceding output shows the loss and accuracy values based on the validation data for all eight experimental runs. For easy reference, it also includes parameter values. We can make the following observations from the preceding output:

•  The highest accuracy value (row 3) is obtained when the number of dense units is 512, the dropout rate is 0.1, and the batch size is 30.
• On the other hand, the lowest accuracy value (row 6) is obtained when the number of dense units is 256, the dropout rate is 0.3, and the batch size is 10. 

The code for obtaining the loss, accuracy, and confusion matrix using the test data for row 3 of the experiment is as follows:

# Loss and accuracy
model %>% evaluate(testx, testy)
$loss
[1] 1.095251
$acc
[1] 0.7975

# Confusion matrix
pred <- model %>% predict_classes(testx)
(tab <- table(Predicted = pred, Actual = data$test$y[1:2000,]))
         Actual
Predicted   0   1   2   3   4   5   6   7   8   9
        0 167   5  20   4   5   4   4  15  10   8
        1   1 176   0   3   0   0   1   1   2  15
        2   3   0 139   9   2   4   8   1   1   0
        3   0   0   3  92   6   6   5   0   0   0
        4   4   0  20  16 177  12  17  23   0   1
        5   0   0   7  50   1 149   1   9   0   0
        6   1   0   2  11   2   5 177   1   0   1
        7   0   0   0   5   3   4   1 143   0   0
        8  16   3   3   5   2   0   1   0 203   6
        9   4  14   1   4   0   1   1   0   1 172

# Accuracy for each category
100*diag(tab)/colSums(tab)
       0        1        2        3        4        5        6        7 
85.20408 88.88889 71.28205 46.23116 89.39394 80.54054 81.94444 74.09326 
       8        9 
93.54839 84.72906 

From the preceding results, we can make the following observations:

• Both the loss and accuracy values for the test data are better than the results we've obtained so far.
• The best classification results are obtained for category 8 (ship), with 203 correct image classifications out of 217 (93.5% accuracy).
• The lowest classification performance is for category 3 (cat), with 92 correct predictions out of 199 (46.2% accuracy).
• The worst misclassification is of 50 images from category 3 (cat) when they are misclassified as category-5 (dog).

In the next experiment, we will use another pretrained network: VGG16.