We just learned how the update and reset gates work, but how do these gates help in updating the hidden state? That is, how do you add new information to the hidden state and how do you remove unwanted information from the hidden state with the help of the reset and update gates?

First, we will see how to add new information to the hidden state.

We create a new state called the content state, , for holding the information. We know that the reset gate is used to remove information that is not required. So, using the reset gate, we create a content state, , that holds only the required information.

The content state, , at time step is expressed as follows:

The following diagram shows how the content state is created with the reset gate:

Now we will see how to remove information from the hidden state.

We learned that the update gate, , helps to decide what information from the previous time step, , can be taken forward to the next time step, . Multiplying and gives us only the relevant information from the previous step. Instead of having a new gate, we just a take a complement of , that is , and multiply them with .

The hidden state is then updated as follows:

Once the hidden state is computed, we can apply the softmax function and compute the output as follows: