Your Turn
by Iti Mathur, Nisheeth Joshi, Deepti Chopra, Jacob Perkins, Nitin Hardeniya
Natural Language Processing: Python and NLTK
- Natural Language Processing: Python and NLTK
- Table of Contents
- Natural Language Processing: Python and NLTK
- Natural Language Processing: Python and NLTK
- Credits
- Preface
- 1. Module 1
- 1. Introduction to Natural Language Processing
- 2. Text Wrangling and Cleansing
- 3. Part of Speech Tagging
- 4. Parsing Structure in Text
- 5. NLP Applications
- Building your first NLP application
- Other NLP applications
- Summary
- 6. Text Classification
- 7. Web Crawling
- 8. Using NLTK with Other Python Libraries
- 9. Social Media Mining in Python
- 10. Text Mining at Scale
- 2. Module 2
- 1. Tokenizing Text and WordNet Basics
- Introduction
- Tokenizing text into sentences
- Tokenizing sentences into words
- Tokenizing sentences using regular expressions
- Training a sentence tokenizer
- Filtering stopwords in a tokenized sentence
- Looking up Synsets for a word in WordNet
- Looking up lemmas and synonyms in WordNet
- Calculating WordNet Synset similarity
- Discovering word collocations
- 2. Replacing and Correcting Words
- 3. Creating Custom Corpora
- Introduction
- Setting up a custom corpus
- Creating a wordlist corpus
- Creating a part-of-speech tagged word corpus
- Creating a chunked phrase corpus
- Creating a categorized text corpus
- Creating a categorized chunk corpus reader
- Lazy corpus loading
- Creating a custom corpus view
- Creating a MongoDB-backed corpus reader
- Corpus editing with file locking
- 4. Part-of-speech Tagging
- Introduction
- Default tagging
- Training a unigram part-of-speech tagger
- Combining taggers with backoff tagging
- Training and combining ngram taggers
- Creating a model of likely word tags
- Tagging with regular expressions
- Affix tagging
- Training a Brill tagger
- Training the TnT tagger
- Using WordNet for tagging
- Tagging proper names
- Classifier-based tagging
- Training a tagger with NLTK-Trainer
- 5. Extracting Chunks
- Introduction
- Chunking and chinking with regular expressions
- Merging and splitting chunks with regular expressions
- Expanding and removing chunks with regular expressions
- Partial parsing with regular expressions
- Training a tagger-based chunker
- Classification-based chunking
- Extracting named entities
- Extracting proper noun chunks
- Extracting location chunks
- Training a named entity chunker
- Training a chunker with NLTK-Trainer
- 6. Transforming Chunks and Trees
- Introduction
- Filtering insignificant words from a sentence
- Correcting verb forms
- Swapping verb phrases
- Swapping noun cardinals
- Swapping infinitive phrases
- Singularizing plural nouns
- Chaining chunk transformations
- Converting a chunk tree to text
- Flattening a deep tree
- Creating a shallow tree
- Converting tree labels
- 7. Text Classification
- Introduction
- Bag of words feature extraction
- Training a Naive Bayes classifier
- Training a decision tree classifier
- Training a maximum entropy classifier
- Training scikit-learn classifiers
- Measuring precision and recall of a classifier
- Calculating high information words
- Combining classifiers with voting
- Classifying with multiple binary classifiers
- Training a classifier with NLTK-Trainer
- 8. Distributed Processing and Handling Large Datasets
- Introduction
- Distributed tagging with execnet
- Distributed chunking with execnet
- Parallel list processing with execnet
- Storing a frequency distribution in Redis
- Storing a conditional frequency distribution in Redis
- Storing an ordered dictionary in Redis
- Distributed word scoring with Redis and execnet
- 9. Parsing Specific Data Types
- A. Penn Treebank Part-of-speech Tags
- 1. Tokenizing Text and WordNet Basics
- 3. Module 3
- 1. Working with Strings
- 2. Statistical Language Modeling
- Understanding word frequency
- Applying smoothing on the MLE model
- Develop a back-off mechanism for MLE
- Applying interpolation on data to get mix and match
- Evaluate a language model through perplexity
- Applying metropolis hastings in modeling languages
- Applying Gibbs sampling in language processing
- Summary
- 3. Morphology – Getting Our Feet Wet
- 4. Parts-of-Speech Tagging – Identifying Words
- 5. Parsing – Analyzing Training Data
- 6. Semantic Analysis – Meaning Matters
- 7. Sentiment Analysis – I Am Happy
- 8. Information Retrieval – Accessing Information
- 9. Discourse Analysis – Knowing Is Believing
- 10. Evaluation of NLP Systems – Analyzing Performance
- B. Bibliography
- Index
Here are the answers to the questions posed in the above sections:
- Can we remove stop words before POS tagging?
No; If we remove the stop words, we will lose the context, and some of the POS taggers (Pre-Trained model) use word context as features to give the POS of the given word.
- How can we get all the verbs in the sentence?
We can get all the verbs in the sentence by using
pos_tag>>>tagged = nltk.pos_tag(word_tokenize(s)) >>>allverbs = [word for word,pos in tagged if pos in ['VB','VBD','VBG'] ]
- Can you modify the code of the hybrid tagger in the N-gram tagger section to work with Regex tagger? Does that improve performance?
Yes. We can modify the code of the hybrid tagger in the N-gram tagger section to work with the Regex tagger:
>>>print unigram_tagger.evaluate(test_data,backoff= regexp_tagger) >>>bigram_tagger = BigramTagger(train_data, backoff=unigram_tagger) >>>print bigram_tagger.evaluate(test_data) >>>trigram_tagger=TrigramTagger(train_data,backoff=bigram_tagger) >>>print trigram_tagger.evaluate(test_data) 0.857122212053 0.866708415627 0.863914446746
The performance improves as we add some basic pattern-based rules, instead of predicting the most frequent tag.
- Can you write a tagger that tags Date and Money expressions?
Yes, we can write a tagger that tags Date and Money expressions. Following is the code:
>>>date_regex = RegexpTagger([(r'(d{2})[/.-](d{2})[/.-](d{4})$','DATE'),(r'$','MONEY')]) >>>test_tokens = "I will be flying on sat 10-02-2014 with around 10M $ ".split() >>>print date_regex.tag(test_tokens)
Can you try a similar word cloud to what we did in Chapter 1, Introduction to Natural Language Processing with only nouns and verbs now?
References:
https://github.com/japerk/nltk-trainer
http://en.wikipedia.org/wiki/Part-of-speech_tagging
http://en.wikipedia.org/wiki/Named-entity_recognition
http://www.inf.ed.ac.uk/teaching/courses/icl/nltk/tagging.pdf
-
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.