The input gate is responsible for deciding what information should be stored in the cell state. Let's consider the same example:

Harry is a good singer. He lives in New York. Zayn is also a good singer.

After the forget gate removes information from the cell state, the input gate decides what information it has to keep in the memory. Here, since the information about Harry is removed from the cell state by the forget gate, the input gate decides to update the cell state with information about Zayn.

Similar to the forget gate, the input gate is controlled by a sigmoid function that returns output in the range of 0 to 1. If it returns 1, then the particular information will be stored/updated to the cell state, and if it returns 0, we will not store the information to the cell state. The input gate at time step is expressed as follows:

Here, the following applies:

• is the input-to-hidden weights of the input gate
• is the hidden-to-hidden weights of the input gate
• is the bias of the input gate

The following diagram shows the input gate: