Lemmatization is very similar to stemming, but is more akin to synonym replacement. A lemma is a root word, as opposed to the root stem. So unlike stemming, you are always left with a valid word that means the same thing. However, the word you end up with can be completely different. A few examples will explain this.
Make sure that you have unzipped the wordnet corpus in nltk_data/corpora/wordnet. This will allow the WordNetLemmatizer class to access WordNet. You should also be familiar with the part-of-speech tags covered in the Looking up Synsets for a word in WordNet recipe of Chapter 1, Tokenizing Text and WordNet Basics.
We will use the WordNetLemmatizer class to find lemmas:
>>> from nltk.stem import WordNetLemmatizer
>>> lemmatizer = WordNetLemmatizer()
>>> lemmatizer.lemmatize('cooking')
'cooking'
>>> lemmatizer.lemmatize('cooking', pos='v')
'cook'
>>> lemmatizer.lemmatize('cookbooks')
'cookbook'The WordNetLemmatizer class is a thin wrapper around the wordnet corpus and uses the morphy() function of the WordNetCorpusReader class to find a lemma. If no lemma is found, or the word itself is a lemma, the word is returned as is. Unlike with stemming, knowing the part of speech of the word is important. As demonstrated previously, cooking does not return a different lemma unless you specify that the POS is a verb. This is because the default POS is a noun, and as a noun, cooking is its own lemma. On the other hand, cookbooks is a noun with its singular form, cookbook, as its lemma.
Here's an example that illustrates one of the major differences between stemming and lemmatization:
>>> from nltk.stem import PorterStemmer
>>> stemmer = PorterStemmer()
>>> stemmer.stem('believes')
'believ'
>>> lemmatizer.lemmatize('believes')
'belief'Instead of just chopping off the es like the PorterStemmer class, the WordNetLemmatizer class finds a valid root word. Where a stemmer only looks at the form of the word, the lemmatizer looks at the meaning of the word. By returning a lemma, you will always get a valid word.
Stemming and lemmatization can be combined to compress words more than either process can by itself. These cases are somewhat rare, but they do exist:
>>> stemmer.stem('buses')
'buse'
>>> lemmatizer.lemmatize('buses')
'bus'
>>> stemmer.stem('bus')
'bu'In this example, stemming saves one character, lemmatization saves two characters, and stemming the lemma saves a total of three characters out of five characters. That is nearly a 60% compression rate! This level of word compression over many thousands of words, while unlikely to always produce such high gains, can still make a huge difference.
In the previous recipe, we covered the basics of stemming and WordNet was introduced in the Looking up Synsets for a word in WordNet and Looking up lemmas and synonyms in WordNet recipes of Chapter 1, Tokenizing Text and WordNet Basics. Looking forward, we will cover the Using WordNet for tagging recipe in Chapter 4, Part-of-speech Tagging.