We just learned how the information in the cell state will be updated. Now, we will see how the information in the hidden state will be updated. We learned that the hidden state, , is used for computing the output, but how can we compute the output?

We know that the output gate is responsible for deciding what information should be taken from the cell state to give as output. Thus, multiplying and tanh (to squash between -1 and +1) of cell state, , gives us the output.

Thus, the hidden state, , is expressed as follows:

The following diagram shows how the hidden state, , is computed by multiplying and :

Finally, once we have the hidden state value, we can apply the softmax function and compute as follows:

Here, is the hidden-to-output layer weights.