The update gate helps to decide what information from the previous time step, , can be taken forward to the next time step, . It is basically a combination of an input gate and a forget gate, which we learned about in LSTM cells. Similar to the gates about the LSTM cell, the update gate is also regulated by the sigmoid function.

The update gate, , at time step is expressed as follows:

Here, the following applies:

• is the input-to-hidden weights of the update gate
• is the hidden-to-hidden weights of the update gate
• is the bias of the update gate

The following diagram shows the update gate. As you can see, input is multiplied with , and the previous hidden state, , 0 and 1: