Chunking is the process used to perform entity detection. It is used for the segmentation and labeling of multiple sequences of tokens in a sentence.
To design a chunker, a chunk grammar should be defined. A chunk grammar holds the rules of how chunking should be done.
Let's consider the example that performs Noun Phrase Chunking by forming the chunk rules:
>>> import nltk
>>> sent=[("A","DT"),("wise", "JJ"), ("small", "JJ"),("girl", "NN"), ("of", "IN"), ("village", "N"), ("became", "VBD"), ("leader", "NN")]
>>> sent=[("A","DT"),("wise", "JJ"), ("small", "JJ"),("girl", "NN"), ("of", "IN"), ("village", "NN"), ("became", "VBD"), ("leader", "NN")]
>>> grammar = "NP: {<DT>?<JJ>*<NN><IN>?<NN>*}"
>>> find = nltk.RegexpParser(grammar)
>>> res = find.parse(sent)
>>> print(res)
(S
(NP A/DT wise/JJ small/JJ girl/NN of/IN village/NN)
became/VBD
(NP leader/NN))
>>> res.draw()The following parse tree is generated:
Here, the chunk rule for Noun Phrase is defined by keeping DT as optional, any number of JJ, followed by NN, optional IN, and any number of NN.
Consider another example in which the Noun Phrase chunk rule is created with any number of nouns:
>>> import nltk
>>> noun1=[("financial","NN"),("year","NN"),("account","NN"),("summary","NN")]
>>> gram="NP:{<NN>+}"
>>> find = nltk.RegexpParser(gram)
>>> print(find.parse(noun1))
(S (NP financial/NN year/NN account/NN summary/NN))
>>> x=find.parse(noun1)
>>> x.draw()The output in the form of the parse tree is given here:
Chunking is the process in which some of the parts of a chunk are eliminated. Either an entire chunk may be used, a part of the chunk may be used from the middle and the remaining parts are eliminated, or a part of chunk may be used either from the beginning of the chunk or from the end of the chunk and the remaining part of the chunk is removed.
Consider the code for UnigramChunker in NLTK, which has been developed to perform chunking and parsing:
class UnigramChunker(nltk.ChunkParserI):
def _init_(self,training):
training_data=[[(x,y) for p,x,y in nltk.chunk.treeconlltags(sent)]
for sent in training]
self.tagger=nltk.UnigramTagger(training_data)
def parsing(self,sent):
postags=[pos1 for (word1,pos1) in sent]
tagged_postags=self.tagger.tag(postags)
chunk_tags=[chunking for (pos1,chunktag) in tagged_postags]
conll_tags=[(word,pos1,chunktag) for ((word,pos1),chunktag)
in zip(sent, chunk_tags)]
return nltk.chunk.conlltaags2tree(conlltags)Consider the following code, which can be used to estimate the accuracy of the chunker after it is trained:
import nltk.corpus, nltk.tag
def ubt_conll_chunk_accuracy(train_sents, test_sents):
chunks_train =conll_tag_chunks(training)
chunks_test =conll_tag_chunks(testing)
chunker1 =nltk.tag.UnigramTagger(chunks_train)
print 'u:', nltk.tag.accuracy(chunker1, chunks_test)
chunker2 =nltk.tag.BigramTagger(chunks_train, backoff=chunker1)
print 'ub:', nltk.tag.accuracy(chunker2, chunks_test)
chunker3 =nltk.tag.TrigramTagger(chunks_train, backoff=chunker2)
print 'ubt:', nltk.tag.accuracy(chunker3, chunks_test)
chunker4 =nltk.tag.TrigramTagger(chunks_train, backoff=chunker1)
print 'ut:', nltk.tag.accuracy(chunker4, chunks_test)
chunker5 =nltk.tag.BigramTagger(chunks_train, backoff=chunker4)
print 'utb:', nltk.tag.accuracy(chunker5, chunks_test)
# accuracy test for conll chunking
conll_train =nltk.corpus.conll2000.chunked_sents('train.txt')
conll_test =nltk.corpus.conll2000.chunked_sents('test.txt')
ubt_conll_chunk_accuracy(conll_train, conll_test)
# accuracy test for treebank chunking
treebank_sents =nltk.corpus.treebank_chunk.chunked_sents()
ubt_conll_chunk_accuracy(treebank_sents[:2000], treebank_sents[2000:])