Next, we'll define our loss function. We'll use softmax cross-entropy as our loss function. TensorFlow provides the tf.nn.softmax_cross_entropy_with_logits() function for computing softmax cross-entropy loss. It takes two parameters as inputs, logits and labels:

• The logits parameter specifies the logits predicted by our network; for example, y_hat
• The labels parameter specifies the actual labels; for example, true labels, Y

We take the mean of the loss function using tf.reduce_mean():

with tf.name_scope('Loss'):
        loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y_hat,labels=Y))

Now, we need to minimize the loss using backpropagation. Don't worry! We don't have to calculate the derivatives of all the weights manually. Instead, we can use TensorFlow's optimizer. In this section, we the use the Adam optimizer. It is a variant of the gradient descent optimization technique we learned about in Chapter 1, Introduction to Deep Learning. In Chapter 3, Gradient Descent and Its Variants, we will dive into the details and see how exactly the Adam optimizer and several other optimizers work. For now, let's say we use the Adam optimizer as our backpropagation algorithm:

learning_rate = 1e-4
optimizer = tf.train.AdamOptimizer(learning_rate).minimize(loss)