What makes LSTM cells so special? How do LSTM cells achieve long-term dependency? How does it know what information to keep and what information to discard from the memory?

This is all achieved by special structures called gates. As shown in the following diagram, a typical LSTM cell consists of three special gates called the input gate, output gate, and forget gate:

These three gates are responsible for deciding what information to add, output, and forget from the memory. With these gates, an LSTM cell effectively keeps information in the memory only as long as required.

In an RNN cell, we used the hidden state, , for two purposes: one for storing the information and the other for making predictions. Unlike RNN, in the LSTM cell we break the hidden states into two states, called the cell state and the hidden state:

• The cell state is also called internal memory and is where all the information will be stored
• The hidden state is used for computing the output, that is, for making predictions

Both the cell state and hidden state are shared across every time step. Now, we will deep dive into the LSTM cell and see exactly how these gates are used and how the hidden state predicts the output.