The nltk.model.ngram
module in NLTK has a submodule, perplexity(text)
. This submodule evaluates the perplexity of a given text. Perplexity is defined as 2**Cross Entropy for the text. Perplexity defines how a probability model or probability distribution can be useful to predict a text.
The code for evaluating the perplexity of text as present in the nltk.model.ngram
module is as follows:
def perplexity(self, text): """ Calculates the perplexity of the given text. This is simply 2 ** cross-entropy for the text. :param text: words to calculate perplexity of :type text: list(str) """ return pow(2.0, self.entropy(text))