The forget gate, , is responsible for deciding what information should be removed from the cell state (memory). Consider the following sentence:

Harry is a good singer. He lives in New York. Zayn is also a good singer.

As soon as we start talking about Zayn, the network will understand that the subject has been changed from Harry to Zayn, and the information about Harry is no longer required. Now, the forget gate will remove/forget information about Harry from the cell state.

The forget gate is controlled by a sigmoid function. At time step , we pass input , and the previous hidden state, , to the forget gate. It return 0 if the particular information from the cell state should be removed and returns 1 if the information should not be removed. The forget gate, , at a time step, , is expressed as follows:

Here, the following applies:

• is the input-to-hidden layer weights of the forget gate
• is the hidden-to-hidden layer weights of the forget gate
• is the bias of the forget gate

The following diagram shows the forget gate. As you can see, input is multiplied with and the previous hidden state, , is multiplied with , then both will be added together and sent to the sigmoid function, which returns , as follows: