Our network architecture consists of two convolutional layers. Each convolutional layer is followed by one pooling layer, and we use a fully connected layer that is followed by an output layer; that is, conv1->pooling->conv2->pooling2->fully connected layer-> output layer.
First, we define the first convolutional layer and pooling layer.
The weights are actually the filters in the convolutional layers. So, the weight matrix will be initialized as [ filter_shape[0], filter_shape[1], number_of_input_channel, filter_size ].
We use a 5 x 5 filter. Since we use grayscale images, the number of input channels will be 1 and we set the filter size as 32. So, the weight matrix of the first convolution layer will be [5,5,1,32]:
W1 = initialize_weights([5,5,1,32])
The shape of the bias is just the filter size, which is 32:
b1 = initialize_bias([32])
Perform the first convolution operation with ReLU activations followed by max pooling:
conv1 = tf.nn.relu(convolution(X, W1) + b1)
pool1 = max_pooling(conv1)
Next, we define the second convolution layer and pooling layer.
As the second convolutional layer takes the input from the first convolutional layer, which has 32-channel output, the number of input channel, to the second convolutional layer becomes 32 and we use the 5 x 5 filter with a filter size of 64. Thus, the weight matrix for the second convolutional layer becomes [5,5,32,64]:
W2 = initialize_weights([5,5,32,64])
The shape of the bias is just the filter size, which is 64:
b2 = initialize_bias([64])
Perform the second convolution operation with ReLU activations, followed by max pooling:
conv2 = tf.nn.relu(convolution(pool1, W2) + b2)
pool2 = max_pooling(conv2)
After two convolution and pooling layers, we need to flatten the output before feeding it to the fully connected layer. So, we flatten the result of the second pooling layer and feed it to the fully connected layer.
Flatten the result of the second pooling layer:
flattened = tf.reshape(pool2, [-1, 7*7*64])
Now we define the weights and bias for the fully connected layer. We know that we set the shape of the weight matrix as [number of neurons in the current layer, number of neurons layer in the next layer]. This is because the shape of the input image becomes 7x7x64 after flattening and we use 1024 neurons in the hidden layer. The shape of the weights becomes [7x7x64, 1024]:
W_fc = initialize_weights([7*7*64, 1024])
b_fc = initialize_bias([1024])
Here is a fully connected layer with ReLU activations:
fc_output = tf.nn.relu(tf.matmul(flattened, W_fc) + b_fc)
Define the output layer. We have 1024 neurons in the current layer, and since we need to predict 10 classes, we have 10 neurons in the next layer, thus the shape of the weight matrix becomes [1024 x 10]:
W_out = initialize_weights([1024, 10])
b_out = initialize_bias([10])
Compute the output with softmax activations:
YHat = tf.nn.softmax(tf.matmul(fc_output, W_out) + b_out)