A chatbot is helpful in a scenario where the bot automates some of the more common queries. This is very useful in a practical scenario, especially in cases where you would have to just look up the result from a database or query an API to obtain the result that the query is about.
Given this, there are potentially two ways that you can design a chatbot, as follows:
- Convert the unstructured user query into a structured format:
- Query from the database based on the converted structure
- Generate responses based on the input text
For this exercise, we will be adopting the first approach, as it is more likely to give predictions that can be tweaked further before presenting them to the user. Additionally, we will also understand the reason why we might not want to generate responses based on input text after we go through the machine translation and text summarization case studies.
Converting a user query into a structured format involves the following two steps:
- Assign entities to each word in a query
- Understand the intent of query
Named entity recognition is one of the applications that has multiple use cases across industries. For example, where does the user want to travel to? Which product is a user considering to purchase? And so on. From these examples, it might occur that named entity recognition is a simple lookup from a dictionary of existing city names or product names. However, think of a scenario where a user says I want to travel from Boston to Seattle. In this case, while the machine understands that both Boston and Seattle are city names, we are not in a position to resolve which is the from city and which is the to city.
While we can add some heuristics like a name that has a to before it is the to city and the other is the from city, it is not scalable as we replicate this process across multiple such examples. Neural networks come in handy in that scenario, where there is a less dependency on us to hand-tune the features. We shall let the machine take care of the feature engineering part to give us the output.