We will now take a brief look at the graphs and the loss function, with TensorBoard. First, we will look at the graph output from TensorBoard:

We can clearly see that the encoding layer and decoding layer form the main components of the model, with the Bahdanau attention mechanism as another input in the decoding layer. The input embeddings are first fed into the encoding layer, whose output is fed to the attention mechanism, as well as the decoding layer. Finally, the decoding layer provides the predictions as output. The optimizer takes in the decoding layer output, the attention mechanism output, and the targets, to optimize on the cost function:

We also find that the cost steadily decreases with the number of training steps. In the next section, we will look at some recent papers that describe enhancements to the encoder-decoder model, with attention described here.