The simplest form of image classification is binary classification. This is where we have a classifier that has just one object to classify, for example, dog/no dog. In this case, a loss function we are likely to use is the binary cross-entropy loss.

The cross entropy function between true labels p and model predictions q is defined as:

With i being the index for each possible element of our labels and predictions.

However, as we are dealing with the binary case when we have only two possible outcomes, y=1 and y=0, then p{} and q {} can be simplified down and we get:

This is equivalent 

Iterating over   training examples, the cost function L to be minimized then becomes this:

This is intuitively correct, as when , we want to minimize , which requires large  and when , we want to minimize , which requires a small .

In TensorFlow, the binary cross entropy loss can be found in the tf.losses module. It is useful to know that the name for the raw output  of our model is logits. Before we can pass this to the cross entropy loss we need to apply the sigmoid function to it so our output is scaled between 0 and 1. TensorFlow actually combines all these steps together into one operation, as shown in the code below. Also TensorFlow will take care of averaging the loss across the batch for us.

loss = tf.losses.sigmoid_cross_entropy(multi_class_labels=labels_in, logits=model_prediction)