After the previous Chapter 2, Let's Help Machine Learn, you now know how to make machines learn from observations and data points so that they can find out interesting patterns, trends, and make predictions. In this chapter, we will be dealing with one of the complex problems faced by retailers, stores, and e-commerce marketplaces today. With the advent of modern technology and innovations, shopping has become a relatively pleasant and enjoyable experience which we can enjoy from the comfort of our home, without even venturing to an actual store, using the web or dedicated apps which provide shopping facilities. With a humongous number of retailers, stores, marketplaces, and sellers, competition is pretty stiff, and to attract customers, they have to use all the data they can gather from consumers about their personal traits and shopping patterns, and use machine learning techniques to try and make shopping experiences as personalized as possible based on each customer.
You might be wondering how machine learning can help in making shopping experiences personalized for each user! The answer lies in two things: data and algorithms. Using a combination of both of these, retailers are able to figure out what are the most trending items that the consumers buy, the likes and dislikes of the customers, the peak times when the sales go up and come down, the trending combination of products which people tend to buy, and the product reviews and prices which are being offered by other retailers for the same products. Retailers have their own data science teams which aggregate this data and apply various machine learning algorithms which are used to analyze the trending products and build recommender engines which predict what the customers are most likely to buy, and give recommendations to the customers based on their interests and shopping history.
In this chapter, we will be focusing on product based recommendations where the algorithms focus on customer shopping transactional data, where we observe common patterns of product combinations bought by the customers, to detect and predict what products customers are most likely to buy and what they have bought in the past. The main techniques we will be focusing on in this chapter are as follows:
- Product contingency matrix evaluation
- Frequent itemsets generation
- Association rule mining
Trend analysis using association rules and pattern mining however have their own limitations. They do not provide a more personalized shopping experience for each customer based on attributes like their interests, products they have bought and rated. We will be looking at that in the subsequent chapter where we focus on algorithms such as user-based collaborative filtering, which takes into account both product based and user based features when building recommender engines.
What is most interesting is that all the retailers and e-commerce marketplaces, such as Staples, Macy's, Target, Amazon, Flipkart, Alibaba, Rakuten, Lazada, and many many others, have their own data science teams, which solve a wide variety of problems including the one we discussed earlier. They make use of all the data generated from customer shopping transactions, product stocks, deliveries, SLAs, reviews, advertisements, click-through rates, bounce rates, pricing data, and many other sources. They process this data and feed it into their machine learning algorithm based engines to generate data driven insights to increase sales and profits for the business. Now this is definitely one domain which is hot in the market right now. Let's now look further into some of the machine learning techniques and algorithms which help them in making such great data driven decisions!
In this section, we will talk about what exactly we mean by trends and how the retailers detect and predict these trends. Basically, a trend in the retail context can be defined as a specific pattern or behavior which occurs over a period of time. This may involve a product or a combination of products being sold out in a very short period of time or even the reverse. A simple example would be a best-selling smartphone being prebooked and out of stock before even hitting the shelves on any e-commerce marketplace, or a combination of products like the classic beer and diapers combination which is frequently found in shopping baskets or carts of customers!
How can we even start analyzing shopping carts or start to detect and predict shopping trends. Like I mentioned earlier, we can achieve this with a combination of the right data and algorithms. Let's assume that we are heading a large retail chain. First we will have to keep track of each and every transaction which is taking place from our stores and website. We will need to gather data points relevant to the items being purchased, stockouts, combinations of items purchased together, and customer transactions to begin with.
Once we have this data, we can start processing, normalizing, and aggregating this data to machine readable formats, which can be easily operated on and fed to machine learning algorithms for product recommendations based on the shopping trends. We can achieve this by using the right data structures and constructs which we learned back in Chapter 1, Getting Started with R and Machine Learning. There are several machine learning algorithms which help us in analyzing the shopping transactional data and recommending products based on the shopping trends. The main paradigm under which these algorithms fall is popularly known as market basket analysis. Interestingly, these algorithms use statistical and machine learning concepts such as probability, support, confidence, lift, pattern detection, and many more to determine what are the items being bought together frequently, which helps us in analyzing shopping transactions and detecting and predicting trends. This ultimately helps us in making product recommendations for the customers and also making business decisions wisely, if we were running a retail chain! Do note that the only data we will be using in both these algorithms is pure shopping transactions based data.
Before we start diving into building algorithms to analyze the shopping carts and transactions, let us first see what market basket analysis actually means and the concepts associated with it. This will come in handy later on, when we implement machine learning algorithms using some of these concepts to solve real world problems.