Another challenging problem faced by researchers is the long-term dependencies that one can find in text. For example, if someone feeds a sequence like I used to live in France and I learned how to speak... the next obvious word in the sequence is the word French.
In these kind of situation vanilla RNNs will be able to handle it because it has short-term dependencies, as shown in Figure 6:
Another example, if someone started the sequence by saying that I used to live in France.. and then he/she start to describe the beauty of living there and finally he ended the sequence by I learned to speak French. So, for the model to predict the language that he/she learned at the end of the sequence, the model needs to have some information about the early words live and France. The model won't be able to handle these kind of situation, if it doesn't manage to keep track of long term dependencies in the text:
To handle vanishing gradients and long-term dependencies in the text, researchers introduced a variation of the vanilla RNN network called Long Short Term Networks (LSTM).