A corpus may be known as a collection of documents. A corpora is the collection of multiple corpus.
Let's see the following code, which will generate a data
directory inside the home
directory:
>>> import nltk >>> import os,os.path >>> create = os.path.expanduser('~/nltkdoc') >>> if not os.path.exists(create): os.mkdir(create) >>> os.path.exists(create) True >>> import nltk.data >>> create in nltk.data.path True
This code will create a data directory named ~/nltkdoc
inside the home
directory. The last line of this code will return True
and will ensure that the data
directory has been created. If the last line of the code returns False
, then it means that the data
directory has not been created and we need to create it manually. After creating the data
directory manually, we can test the last line and it will then return True
. Within this directory, we can create another directory named nltkcorpora
that will hold the whole corpus. The path will be ~/nltkdoc/nltkcorpora
. Also, we can create a subdirectory named important
that will hold all the necessary files.
The path will be ~/nltkdoc/nltkcorpora/important
.
Let's see the following code to load a text file into the subdirectory:
>>> import nltk.data >>> nltk.data.load('nltkcorpora/important/firstdoc.txt',format='raw') 'nltk '
Here, in the previous code, we have mentioned format='raw'
, since nltk.data.load()
cannot interpret .txt
files.
There is a word list corpus in NLTK known as the Names
corpus. It consists of two files, namely, male.txt
and female.txt
.
Let's see the code to generate the length of male.txt
and female.txt
:
>>> import nltk >>> from nltk.corpus import names >>> names.fileids() ['female.txt', 'male.txt'] >>> len(names.words('male.txt')) 2943 >>> len(names.words('female.txt')) 5001
NLTK also consists of a large collection of English words. Let's see the code that describes the number of words present in the English word file:
>>> import nltk >>> from nltk.corpus import words >>> words.fileids() ['en', 'en-basic'] >>> len(words.words('en')) 235886 >>> len(words.words('en-basic')) 850
Consider the following code used in NLTK for defining the Maxent Treebank POS tagger:
def pos_tag(tok): """
We can use POS tagger given by NLTK to tag a list of tokens:
>>> from nltk.tag import pos_tag >>> from nltk.tokenize import word_tokenize >>> pos_tag(word_tokenize("Papa's favourite hobby is reading.")) [('Papa', 'NNP'), ("'s", 'POS'), ('favourite', 'JJ'), ('hobby', 'NN'), ('is', 'VBZ'), ('reading', 'VB'), ('.', '.')] :param tokens: list of tokens that need to be tagged :type tok: list(str) :return: The tagged tokens :rtype: list(tuple(str, str)) """ tagger = load(_POS_TAGGER) return tagger.tag(tok) def batch_pos_tag(sent): """ We can use part of speech tagger given by NLTK to perform tagging of list of tokens. """ tagger = load(_POS_TAGGER) return tagger.batch_tag(sent)