The digital world has made everything available at the click of a button. With everything going the online way, online shopping or e-commerce has become a big thing. From groceries to electronics to even cars, everything is available at the Amazon, Flipkart, and eBay of the world. This ever expanding digital market is just the right place for data science to show its magic.
The online revolution of e-commerce has not only empowered the customers, it has also overwhelmed them with too many choices. Choices are not only in terms of products or categories but also between different e-commerce platforms. Being an ecommerce company in this highly competitive market can be really difficult. Standing out is a challenge and that is where data yet again comes to the rescue.
As we saw in Chapter 3, Predicting Customer Shopping Trends with Market Basket Analysis, purchase patterns can provide a lot of insights about shopping behaviors. We utilized such data to find association rules to not only help the customers quickly find the right products but also help the retailers increase revenues (see Chapter 3, Predicting Customer Shopping Trends with Market Basket Analysis). For association rules, the granularity lies at the transaction level. They use transactions as a central entity and hence do not provide user specific insights.
In this chapter as well, we will continue our project work in the e-commerce domain. Here we will tackle the problem of personalization. We will use machine learning algorithms to provide user specific recommendations.
Through this chapter we will learn about:
- Recommendation systems and their types
- Issues with recommendation systems
- Collaborative filters
- Building a recommendation system from scratch based on matrix factorization
- Utilizing highly optimized R packages to build a production ready recommendation engine and evaluate its recommendations
Throughout this chapter we will use the terms recommender engines and recommendation systems interchangeably.
Every individual in unique, the way we do things is what defines us uniquely. We eat, walk, talk, and even shop in a very unique way. Since the focus of this chapter is e-commerce, we will focus mostly on our shopping behaviors. We will utilize each customer's unique behavior to provide a personalized shopping experience.
To accomplish the task of providing a personalized shopping experience, we need a system to understand and model our customers. Recommendation engines are the systems which learn about customer preferences, choices, and so on, to recommend new products which are closer to what the user might have purchased themselves, thus providing a personalized experience. The options presented by such systems would have a high probability of the customer purchasing them.
Let us try to formally define a recommendation system.
Recommendation systems (or recommender engines) are a class of information filtering systems which analyze the input data to predict preferences of a user as they might have done for themselves.
Unlike information filtering systems which remove or filter information, recommender engines add or re-arrange the information flowing towards the user, which is more relevant to the current context.
Recommender engines are not a new concept. They have existed long before the internet was there. They existed in the form of our friends and family who used to recommend us things to buy because they understood our choices. These were and still are a sort of offline-recommender engines. The web is full of online-recommender engines. From recommendation related to Who to follow on Twitter to Other movies you might enjoy on Netflix to Jobs you may be interested in on LinkedIn, recommender engines are everywhere and not just on e-commerce platforms.
Now that we have understood what a recommendation engine is, let us look at their different types:
- User-based recommender engines: As the name suggests, these systems have the user as the central entity. The activities, preferences, or behavior of the users are analyzed to predict what they might like depending upon their similarity with other such users. They are also termed as User Based Collaborative Filters in general due to extensive use of collaborative filters specifically for such recommender engines.
- Content-based recommender engines: As the name suggests, these engines have the content or the items as the central entities. These items are analyzed to extract features; also the user profiles are built to map user preferences to the type of items. The engines then use this information to predict items which are similar to the ones the users have liked in the past. Such recommender engines are also known as item-based collaborative filters and have their roots in information retrieval theory.
- Hybrid recommender engines: These systems take the best of both worlds to improve upon the prediction results. The two pure types can be used simultaneously and then their results can be combined; they can be used by adding collaborative filtering capabilities to content based systems or even by unifying both the approaches into a single model. Multiple studies have been conducted to demonstrate that hybrid approaches are better than the simple ones. Hybrid recommendation engines are also better at tackling the problems which haunt recommender engines in general.
Before we dive deep into the intricacies of these algorithms, let us see the issues that affect the recommender systems.