Swapping verb phrases
by Iti Mathur, Nisheeth Joshi, Deepti Chopra, Jacob Perkins, Nitin Hardeniya
Natural Language Processing: Python and NLTK
- Natural Language Processing: Python and NLTK
- Table of Contents
- Natural Language Processing: Python and NLTK
- Natural Language Processing: Python and NLTK
- Credits
- Preface
- 1. Module 1
- 1. Introduction to Natural Language Processing
- 2. Text Wrangling and Cleansing
- 3. Part of Speech Tagging
- 4. Parsing Structure in Text
- 5. NLP Applications
- Building your first NLP application
- Other NLP applications
- Summary
- 6. Text Classification
- 7. Web Crawling
- 8. Using NLTK with Other Python Libraries
- 9. Social Media Mining in Python
- 10. Text Mining at Scale
- 2. Module 2
- 1. Tokenizing Text and WordNet Basics
- Introduction
- Tokenizing text into sentences
- Tokenizing sentences into words
- Tokenizing sentences using regular expressions
- Training a sentence tokenizer
- Filtering stopwords in a tokenized sentence
- Looking up Synsets for a word in WordNet
- Looking up lemmas and synonyms in WordNet
- Calculating WordNet Synset similarity
- Discovering word collocations
- 2. Replacing and Correcting Words
- 3. Creating Custom Corpora
- Introduction
- Setting up a custom corpus
- Creating a wordlist corpus
- Creating a part-of-speech tagged word corpus
- Creating a chunked phrase corpus
- Creating a categorized text corpus
- Creating a categorized chunk corpus reader
- Lazy corpus loading
- Creating a custom corpus view
- Creating a MongoDB-backed corpus reader
- Corpus editing with file locking
- 4. Part-of-speech Tagging
- Introduction
- Default tagging
- Training a unigram part-of-speech tagger
- Combining taggers with backoff tagging
- Training and combining ngram taggers
- Creating a model of likely word tags
- Tagging with regular expressions
- Affix tagging
- Training a Brill tagger
- Training the TnT tagger
- Using WordNet for tagging
- Tagging proper names
- Classifier-based tagging
- Training a tagger with NLTK-Trainer
- 5. Extracting Chunks
- Introduction
- Chunking and chinking with regular expressions
- Merging and splitting chunks with regular expressions
- Expanding and removing chunks with regular expressions
- Partial parsing with regular expressions
- Training a tagger-based chunker
- Classification-based chunking
- Extracting named entities
- Extracting proper noun chunks
- Extracting location chunks
- Training a named entity chunker
- Training a chunker with NLTK-Trainer
- 6. Transforming Chunks and Trees
- Introduction
- Filtering insignificant words from a sentence
- Correcting verb forms
- Swapping verb phrases
- Swapping noun cardinals
- Swapping infinitive phrases
- Singularizing plural nouns
- Chaining chunk transformations
- Converting a chunk tree to text
- Flattening a deep tree
- Creating a shallow tree
- Converting tree labels
- 7. Text Classification
- Introduction
- Bag of words feature extraction
- Training a Naive Bayes classifier
- Training a decision tree classifier
- Training a maximum entropy classifier
- Training scikit-learn classifiers
- Measuring precision and recall of a classifier
- Calculating high information words
- Combining classifiers with voting
- Classifying with multiple binary classifiers
- Training a classifier with NLTK-Trainer
- 8. Distributed Processing and Handling Large Datasets
- Introduction
- Distributed tagging with execnet
- Distributed chunking with execnet
- Parallel list processing with execnet
- Storing a frequency distribution in Redis
- Storing a conditional frequency distribution in Redis
- Storing an ordered dictionary in Redis
- Distributed word scoring with Redis and execnet
- 9. Parsing Specific Data Types
- A. Penn Treebank Part-of-speech Tags
- 1. Tokenizing Text and WordNet Basics
- 3. Module 3
- 1. Working with Strings
- 2. Statistical Language Modeling
- Understanding word frequency
- Applying smoothing on the MLE model
- Develop a back-off mechanism for MLE
- Applying interpolation on data to get mix and match
- Evaluate a language model through perplexity
- Applying metropolis hastings in modeling languages
- Applying Gibbs sampling in language processing
- Summary
- 3. Morphology – Getting Our Feet Wet
- 4. Parts-of-Speech Tagging – Identifying Words
- 5. Parsing – Analyzing Training Data
- 6. Semantic Analysis – Meaning Matters
- 7. Sentiment Analysis – I Am Happy
- 8. Information Retrieval – Accessing Information
- 9. Discourse Analysis – Knowing Is Believing
- 10. Evaluation of NLP Systems – Analyzing Performance
- B. Bibliography
- Index
Swapping the words around a verb can eliminate the passive voice from particular phrases. For example, the book was great can be transformed into the great book. This kind of normalization can also help with frequency analysis, by counting two apparently different phrases as the same phrase.
In transforms.py is a function called swap_verb_phrase(). It swaps the right-hand side of the chunk with the left-hand side, using the verb as the
pivot point. It uses the first_chunk_index() function defined in the previous recipe to find the verb to pivot around.
def swap_verb_phrase(chunk):
def vbpred(wt):
word, tag = wt
return tag != 'VBG' and tag.startswith('VB') and len(tag) > 2
vbidx = first_chunk_index(chunk, vbpred)
if vbidx is None:
return chunk
return chunk[vbidx+1:] + chunk[:vbidx]Now we can see how it works on the part-of-speech tagged phrase the book was great:
>>> swap_verb_phrase([('the', 'DT'), ('book', 'NN'), ('was', 'VBD'), ('great', 'JJ')])
[('great', 'JJ'), ('the', 'DT'), ('book', 'NN')]And the result is great the book. This phrase clearly isn't grammatically correct, so read on to learn how to fix it.
Using first_chunk_index() from the previous recipe with the vbpred() function defined inline, we start by finding the first matching verb that is not a gerund (a word that ends in ing) tagged with VBG. Once we've found the verb, we return the chunk with the right side before the left, and remove the verb.
The reason we don't want to pivot around a gerund is that gerunds are commonly used to describe nouns, and pivoting around one would remove that description. Here's an example where you can see how not pivoting around a gerund is a good thing:
>>> swap_verb_phrase([('this', 'DT'), ('gripping', 'VBG'), ('book', 'NN'), ('is', 'VBZ'), ('fantastic', 'JJ')])
[('fantastic', 'JJ'), ('this', 'DT'), ('gripping', 'VBG'), ('book', 'NN')]If we had pivoted around the gerund, the result would be book is fantastic this, and we'd lose the gerund gripping.
Filtering insignificant words makes the final result more readable. By filtering either before or after swap_verb_phrase(), we get fantastic gripping book instead of fantastic this gripping book:
>>> from transforms import swap_verb_phrase, filter_insignificant
>>> swap_verb_phrase(filter_insignificant([('this', 'DT'), ('gripping', 'VBG'), ('book', 'NN'), ('is', 'VBZ'), ('fantastic', 'JJ')]))
[('fantastic', 'JJ'), ('gripping', 'VBG'), ('book', 'NN')]
>>> filter_insignificant(swap_verb_phrase([('this', 'DT'), ('gripping', 'VBG'), ('book', 'NN'), ('is', 'VBZ'), ('fantastic', 'JJ')]))
[('fantastic', 'JJ'), ('gripping', 'VBG'), ('book', 'NN')]Either way, we get a shorter grammatical chunk with no loss of meaning.
-
No Comment