We learnt about taggers and parsers that we can use to build a basic information extraction engine. Let's jump directly to a very basic IE engine and how a typical IE engine can be developed using NLTK.
Any sort of meaningful information can be drawn only if the given input stream goes to each of the following NLP steps. We already have enough understanding of sentence tokenization, word tokenization, and POS tagging. Let's discuss NER and relation extraction as well.
A typical information extraction pipeline looks very similar to that shown in the following figure:
We already briefly discussed NER generally in the last chapter. Essentially, NER is a way of extracting some of the most common entities, such as names, organizations, and locations. However, some of the modified NER can be used to extract entities such as product names, biomedical entities, author names, brand names, and so on.
Let's start with a very generic example where we are given a text file of the content and we need to extract some of the most insightful named entities from it:
# NP chunking (NER) >>>f=open(# absolute path for the file of text for which we want NER) >>>text=f.read() >>>sentences = nltk.sent_tokenize(text) >>>tokenized_sentences = [nltk.word_tokenize(sentence) for sentence in sentences] >>>tagged_sentences = [nltk.pos_tag(sentence) for sentence in tokenized_sentences] >>>for sent in tagged_sentences: >>>print nltk.ne_chunk(sent)
In the preceding code, we just followed the same pipeline provided in the preceding figure. We took all the preprocessing steps, such as sentence tokenization, tokenization, POS tagging, and NLTK. NER (pre-trained models) can be used to extract all NERs.
Relation extraction is another commonly used information extraction operation. Relation extraction as it sound is the process of extracting the different relationships between different entities. There are variety of the relationship that exist between the entities. We have seen relationship like inheritance/synonymous/analogous. The definition of the relation can be dependent on the Information need. For example in the case where we want to look from unstructured text data who is the writer of which book then authorship could be a relation between the author name and book name. With NLTK the idea is to use the same IE pipeline that we used till NER and extend it with a relation pattern based on the NER tags.
So, in the following code, we used an inbuilt corpus of ieer, where the sentences are tagged till NER and the only thing we need to specify is the relation pattern we want and the kind of NER we want the relation to define. In the following code, a relationship between an organization and a location has been defined and we want to extract all the combinations of these patterns. This can be applied in various ways, for example, in a large corpus of unstructured text, we will be able to identify some of the organizations of our interest with their corresponding location:
>>>import re >>>IN = re.compile(r'.*in(?!.+ing)') >>>for doc in nltk.corpus.ieer.parsed_docs('NYT_19980315'): >>> for rel in nltk.sem.extract_rels('ORG', 'LOC', doc, corpus='ieer', pattern = IN): >>>print(nltk.sem.rtuple(rel)) [ORG: u'WHYY'] u'in' [LOC: u'Philadelphia'] [ORG: u'McGlashan & Sarrail'] u'firm in' [LOC: u'San Mateo'] [ORG: u'Freedom Forum'] u'in' [LOC: u'Arlington'] [ORG: u'Brookings Institution'] u', the research group in' [LOC: u'Washington'] [ORG: u'Idealab'] u', a self-described business incubator based in' [LOC: u'Los Angeles'] ..