Now, it is time to train our model. As we have learned, first, we need to initialize all of the variables:

init = tf.global_variables_initializer()

Define the batch size, number of iterations, and learning rate, as follows:

learning_rate = 1e-4
num_iterations = 1000
batch_size = 128

Start the TensorFlow session:

with tf.Session() as sess:

Initialize all the variables:

    sess.run(init)

Save the event files:

    summary_writer = tf.summary.FileWriter('./graphs', graph=sess.graph)

Train the model for a number of iterations:

    for i in range(num_iterations):

Get a batch of data according to the batch size:

        batch_x, batch_y = mnist.train.next_batch(batch_size)

Train the network:

        sess.run(optimizer, feed_dict={ X: batch_x, Y: batch_y})

Print loss and accuracy for every 100^th iteration:

        if i % 100 == 0:

            batch_loss, batch_accuracy,summary = sess.run(
                [loss, accuracy, merge_summary], feed_dict={X: batch_x, Y: batch_y}
                )

            #store all the summaries    
            summary_writer.add_summary(summary, i)


            print('Iteration: {}, Loss: {}, Accuracy: {}'.format(i,batch_loss,batch_accuracy))

As you may notice from the following output, the loss decreases and the accuracy increases over various training iterations:

Iteration: 0, Loss: 2.30789709091, Accuracy: 0.1171875
Iteration: 100, Loss: 1.76062202454, Accuracy: 0.859375
Iteration: 200, Loss: 1.60075569153, Accuracy: 0.9375
Iteration: 300, Loss: 1.60388696194, Accuracy: 0.890625
Iteration: 400, Loss: 1.59523034096, Accuracy: 0.921875
Iteration: 500, Loss: 1.58489584923, Accuracy: 0.859375
Iteration: 600, Loss: 1.51407408714, Accuracy: 0.953125
Iteration: 700, Loss: 1.53311181068, Accuracy: 0.9296875
Iteration: 800, Loss: 1.57677125931, Accuracy: 0.875
Iteration: 900, Loss: 1.52060437202, Accuracy: 0.9453125