11.1 Introduction

For the realization of smart homes, there is a need to develop smart appliances. Automated lights, fans, fridges, garages, air conditioners, etc., are just a few of these appliances that would make life very easy. A refrigerator is the most used appliance in the kitchen and is important in storing and maintaining the freshness food items at home. According to research conducted by the United Nations Environment Program, each year, more than 1.3 billion tons of food is wasted. The majority of the wastes include expired food items and rotten fruits and vegetables from refrigerators. Thus, it becomes important to make sure that the food items in the fridge are used before they expire or rot.

With the rapid growth of technology, many companies have given ideas for the development of smart refrigerators. But these refrigerators are very costly and are therefore not very popular among the middle-class families. Current smart fridges use a combination of RFID equipment managed through a desktop application that screens the items within the refrigerator and caution the client in case any things are about to run out. The problem of using RFID is that all the products should have an RFID tag attached to it. Tagging cannot be done for all products.

A module that can be added to make a smart fridge based on open-source; low-cost hardware design is developed. The proposed framework can be added to already existing fridges. The framework has a camera that captures an image of the interior of the fridge. From this image, the different products present and their quantity are determined using the YOLO (You Only Look Once) algorithm. The proposed system also contains various sensors that help in determining the presence of different products and their quantity. A food quality monitoring module that helps in reducing the waste generated from the fridge will also be incorporated. In expansion, the framework incorporates an Android application that lets clients effortlessly check a basic need list from their mobile gadgets.

According to surveys conducted by the UN, a lot of food waste comes from the household refrigerator. People can easily forget what they have in the fridge and therefore the product can remain in it for ages and when it is finally taken out it may be spoiled and thus is discarded, especially fruits and vegetables. A fridge that can detect the spoilage of food is required. In today’s society, the percentage of working women has increased. They rarely check what items they have in the refrigerator. They may not know when a product may get over. Thus, a smart fridge that can automatically send a list of required products if they get over becomes a necessity. The smart fridges present in today’s market are very costly and are therefore not easily available to the common man. This is the motivation behind the development of an object detecting system based on IoT in a smart refrigerator.

The existing systems have a stock counting mechanism that determines the number of items in the fridge that is not very efficient. They use high-cost items like RFID to detect every product. They also do not include a technique for measuring the amount of liquids in the bottles. Most of the existing systems need manual entry of this quantity. There are also no systems where they have combined the idea of stock counting mechanisms and a food quality monitoring system that works together.

So, the main challenge here is to develop a cost-efficient system that is really quick in detecting the objects present in the refrigerator. Also, the product quality measuring module has to be simultaneously run along with the product detection. The food monitoring system has to be accurate and must not give false notifications that may be a waste of time to the user.

Another challenge is protecting the entire smart system when it is installed in the refrigerator. It must be able to withstand the cold, and also during power shutdowns, it must be safe from the dripping water from the melting of ice in the fridge. So, it needs a protective casing.

11.2 Literature Survey

Mezgec, S., and Seljak, B. K. [1] have presented a new way to detect and recognize food and drink in images. They have developed a new CNN architecture known as NutriNet. The model had been trained with a dataset that containing 225,953 images, 512 × 512 pixels each, of 520 different items. An image containing a food item or drink acts as the input to the recognition model. The output is a text label that contains the class of the food. The performance of the developed network has been compared with other ones like AlexNet, GoogLeNet, and Deep Residual networks. The CNN that they had developed is a modified version of the AlexNet architecture. The NutriNet uses pre-trained models of some other architectures. Therefore, it has a larger input image size. It also has an additional convolutional layer that sets it apart from AlexNet. There are six layers in total. The additional layer has been used to gain more knowledge about features in images that have high resolutions.

Zhang, J., Shao, K., and Luo, X. [2] have developed a hybrid CNN-GRNN model that can be efficient with a small sample. During the training phase, the model uses a convolutional neural network (CNN) to extract the features of the image. The fully connected layer then carries out the prediction. During the testing phase, feature extraction is done by the CNN similar to training, and the classification is done by the GRNN. The GRNN, which has better function approximation capability, replaces the BP neural network. The GRNN has only one variable which makes it better for training than BP. There is no need for the network to iterate. It is able to work with a small sample database. This increases the identification precision and expands the application scope of CNN.

Radovic, Adarkwa, and Wang [3] have given the set of procedures and parameters that need to be used to train CNNs for the efficient and automatic detection of objects on the ground in aerial images. They have used the YOLO software for adapting and testing the CNN.

Zeng, G. [4] discusses the various non-destructive methods that can be employed to detect the quality of fruits. The methods that have been discussed here include detection using optical properties, detection using sonic vibration, detection using machine vision, detection using nuclear magnetic resonance, detection using electrical properties, detection using computed tomography, and detection using electronic noses.

Gao, H., Zhu, F., and Cai, J. [5] have developed a modular electronic nose that can be fitted inside a refrigerator and detect if there is any food spoilage. The user is notified about the spoilage through an LCD display. It works by detecting gases and chemicals like CO₂, acetone, and ethanol that are produced when food items when they rot. Bluetooth module is used to enable wireless communication with the electronic nose. The sensors that they used include CO₂ sensor, alcohol/organic solvent sensor, and temperature humidity sensor. These sensors work together to provide values on the detection of the chemicals at the particular temperature and humidity setting. These inputs are fed to the microcontroller which processes then and sends signals to the Bluetooth module so that it can be transmitted. They have used an ATMega2561 MCU as the microcontroller.

Muralidaran, S., Patil, S., and Poddar, A. [6] have developed a prototype for the detection of the freshness of fruits. They have used various sensors to determine this. A Raspberry Pi has been used to process the data collected. First, they have used an open CV to detect the shape, size, and color of the fruit. They have combined this with the readings from the sensors to make it more efficient. A load cell has been used to determine weights of the fruits placed on it. A gas sensor has been used to detect the presence of gasses present or those emitted by rotten fruits. They have displayed the freshness percentage on an LCD display.

Jayasankar, Jeyashree, Deepalakshmi, R., and Karthika [7] have proposed an efficient classification system for fruits and vegetables using CNN model which extracts features of the image needed for classification. Saliency is a form of image segmentation technique. Image saliency has been used to select the important regions based on the saliency map, i.e., to segment the images in the picture. Three modules have been used for saliency extraction, namely, particular feature extraction—DKL colors, intensity, and orientations are considered as features to extract; activation—utilizing equilibrium distribution over map locations activation maps are formed; and normalization— normalized activation maps are combined into a single image as the output. The VGG model has been then used to train the items for classification. For the experiment, over 26 categories of fruits and vegetables were used and the classifying system achieved an accuracy rate of 95.6%.

Varun, K. S., Kumar, K. A., Chowdary, V. R., and Raju, C. S. K. have designed a system [8] in which an ultrasonic sensor is used to detect the level of water and is displayed as a percentage. The ultrasonic sensor has two parts: trig and echo. Trig emits a sound wave to an object. When it strikes, the object it bounces back and is called the echo. The echo part detects the reflected sound ray and measures the time taken for the sound to return. The distance can be calculated based on this time.

(11.1)

where v is the velocity of sound, t is the time taken, and d is the distance between the sensor and the water surface.

(11.2)

This sensor is mounted on the roof of the tank. Therefore, based on the distance, the level of water in the tank can be determined.

Sunmonu, R., Aduramo, Sodunke, Abdulai, O., and Agboola, E. [9] have developed a cost-effective automatic pump system that turns on the pump if the level of water in the tank falls below a certain percentage. The liquid level is monitored by an ultrasonic sensor that is strategically placed in the tank. The signals from the sensor are fed to a circuit that controls the working of the pump. They have used an ATMega328P for processing the signal from the ultrasonic sensor. The Arduino Uno is not used as it is fragile. They have placed a LCD display to show the percentage of water present in the tank.

Kaner, A., and Rane, M. [10] have developed a model to automate the process of turning the motor off when the tank is filled. The model contains four probes A, B, C, and D, which are placed at four different levels in the tank. Three LED indicators, red, yellow, and green, connected to transistors have been installed at each level. Red denotes that the tank is empty, yellow denotes that it is half full, and green denotes that the tank is full. Once the green LED indicator is on, a relay switch mechanism automatically switches the motor off to avoid wastage of water.

Vargas, Ceres, Martín, and Caldero L. have developed [11] an automated system for filling of bottles based on the level and correct placement of corks. The laser and IR sensors are discarded as these were more suited for opaque and flat surfaces. Hence, ultrasonic technique is used. The knowledge of the echo signals time is used to check if the measurement is valid or not.

• V_max>Vthreshold
• The rise time T_r, T₁ < T_r < T₂, where T₁, T₂ are empirical values.

Nasikkar, Kulkarni, and Kakuste [12] have developed a state-of-the-art smart refrigerator that intimates the user through an Android app if the quantity of any product is less. They have used various methods of detection of products for the different types of food. IR sensors are used to determine the presence of eggs. Load cells have been used to maintain a certain quantity of food. IR proximity sensors have been used to determine the quantity of milk and juices. The Arduino Uno has been used as the microcontroller. They have also used a GSM module that sends the list of required products to the nearby grocer on the push of a button on the mobile app.

Ringe, Dalavi, and Kabugade [13] have developed a smart refrigerator. A Raspberry Pi module has been used as the main processor. The Raspberry Pi has been used as it has Broadcom BCM2835 SoC full high definition multimedia processors with 512-mb SDRAM memory and it works on the Linux operating system. Gas sensors have been used to determine the freshness of the food. Load sensors have also been used to determine the quantity of the food present. They have used a camera that gives a real-time image of the food items present inside the fridge. An ESP32 Wi-Fi module has been used to connect the fridge to a mobile application.

Wu, H., and Chuang, Y. [14] have built a low-cost smart fridge with Raspberry Pi. They have installed two sensors, namely, the IR proximity sensor and a light sensor. They have used two cameras: one inside and one outside. They are all connected to the Raspberry Pi. The inner camera, which is triggered when the fridge is opened, is used to capture an image of the items inside the fridge. The outer camera, which is triggered when an object is placed near it, captures images of new products that are put into the fridge and adds it to a database created using firebase. An Android application had been created for users to view food items in the fridge from anywhere at any time.

11.3 Materials and Methods

The smart system must be able to calculate the quantity of items like fruit, vegetables, milk and cartons in the refrigerator every time an object is placed or removed from the fridge. It must be able to detect a wide variety of all the commonly used items in our everyday lifestyle. The smart refrigerator system must also detect the presence of any rotten fruits or vegetables inside the refrigerator so that they can be removed by the user. This is to ensure that the other food does not go spoilt and helps to prevent food wastage. It must notify the user if any object has run out of stock or has gone empty. For this, an application connected to the system that can be accessed from the user’s smart mobile phone is needed. The system must be easy to install and must be reliable.

Usability, safety, portability, reliability, extensibility, confidentiality, process integrity, and synchrony are some of the non-functional requirements considered for this proposed system. The hardware and software requirements and the sensors used in the proposed system are listed in Tables 11.1 and 11.2. The system needs two microcontrollers and an array of sensors for functional purposes.

NodeMCU is a microcontroller that was developed by the ESP8266 Opensource Community. MCU stands for micro-controller unit. It acts as an IoT platform. It has an integrated Wi-Fi module (ESP8266) which allows it to directly connect to the internet without other peripherals.

This along with its small size makes it suitable for IoT devices. It has access to 13 GPIO pins of ESP8266 with 30 pins in total [15].

Raspberry Pi is a single board computer that was developed in the UK by Raspberry Pi Foundation. The model used in this proposed system is the RPi 3. It has four Cortex-A53 processors that use the ARMv8-A 32 bit instruction set and has 1 GB memory, 4 USB ports, 1 HDMI output for video, and 1 analog 3.5-mm phone jack for audio output. It has a microSDHC slot, with USB boot mode. It is Wi-Fi and Bluetooth enabled. It has 40 pins. This module is devoted to the sensing and early detection of food spoilage, mainly fruits and vegetables. This module consists of a gas sensor and temperature/ humidity sensor that is controlled by a NodeMCU. These sensors help in measuring some critical values that will help in determining whether the fruits and vegetables are fresh or not. The MQ3 gas sensor detects the presence of any alcohol of ethanol in the air. These are the main gases that fruits and vegetables emit when they start to go rotten or bad. So, when the gas sensor detects the presence of these in the air, its value changes and is notified to the user. The quantity of liquid in bottles is also measured by an ultrasonic sensor fitted in the bottle caps. It detects the level of water or any other liquid inside the refrigerator and updates the info in the database [16].

Sensors placed in the proposed system collect data and they go either to the NodeMCU or Raspberry Pi. The NodeMCU sends the data collected directly to the database. The image data in the Raspberry Pi is transferred to the processor where the objects are identified using the YOLO algorithm. It is then updated to the database. The Android app retrieves data from the database. The flow diagram of the proposed system is shown in Figure 11.1.

The main module has two microcontrollers, namely, the NodeMCU and the Raspberry Pi 3. The ultrasonic sensor, temperature sensor, and gas sensor are connected to the NodeMCU. The PIR motion sensor and the Pi camera are connected to the Raspberry Pi. Firebase is the real-time database that is used to store the data collected by the sensors. The Raspberry Pi is used to provide the 5-V power supply that the sensors require, except the temperature sensor which is powered by the 3.3 V output of the NodeMCU. All the components are connected together using connecting wires and a breadboard. Two B type cables are required to connect the Raspberry Pi and NodeMCU to the processor. The architecture diagram of the proposed system is shown in Figure 11.2. The various phases involved in the proposed system are as follows: image processing, product sensing, quality detection, and Android application.

11.3.2 Product Sensing

This module is used to detect the presence of items in the fridge, identify them, count them, and update it to the database. This module has a Pi camera attached to a Raspberry Pi. The camera takes a snap of the items in the fridge whenever an item is removed or inserted into the fridge. The items are then identified using YOLO and a count is taken. This module also has an ultrasonic sensor that is mounted near the bottle rack. This sensor is used to find the quantity of liquid inside the bottle. This value is then updated to the database.

PIR sensors are normally used to sense motion and to detect the human movement within the allowable range. These sensors are mostly used in home appliances and gadgets because of its characteristics such as low-power, small, less expensive, and do not wear out. “Passive Infrared”, “Pyroelectric”, or “IR motion” sensors are the other names of PIR sensors.

The output of the PIR motion sensor is connected to GPIO pin 23 of the Raspberry Pi. Whenever motion is detected, an image of the items in the fridge is taken and sent to the processor through the socket connection.

YOLO algorithms in a type of convolution neural network that is able to detect the various objects in a captured image by just analyzing the image once [17]. That is why it is called YOLO and its architecture is shown in Figure 11.3. The algorithm is capable of detecting a large number of items in the same image. It detects objects by providing bounding boxes for each item. It uses regression to detect the objects. The algorithm first predicts where the object is inside the image and draws bounding boxes around it. Then, it predicts the class values of the objects inside each bounding box. There are a total of 24 convolutional layers that help in extracting the features of the image. There are two more dense layers that predict the class of the object. A neural network framework called DarkNet is used to implement the YOLO model.

11.3.4 Android Application

The Android app is specifically developed to display what items are present in the refrigerator. The app also allows the user to set up a threshold for each of the products. The application is connected to the firebase database to receive the updated stock of contents in the refrigerator. There are four tabs: Grocery, Bottles, Quality, and Thresholds. The Grocery tab shows the items available in the fridge at that moment. It has to be updated concurrently with the database. The Bottles tab shows the quantity of liquid in the bottle and whether they are empty, full, or half full. The Quality tab shows the details of the temperature, humidity, and presence of any rotten gases inside the fridge (Figure 11.4).

11.4 Results and Discussion

The system consists of two microcontrollers, namely, Raspberry Pi and NodeMCU. It also has various sensors like the PIR sensor, pi camera, MQ3 sensor (gas sensor), and ultrasonic sensor. The object detection module consists of the PIR sensor and Pi camera connected to the Raspberry Pi microcontroller. Whenever a person opens the fridge, the PIR sensor detects motion and triggers the Pi camera to take an image of the contents of the refrigerator. The quality and quantity measuring module consists of the gas sensor and the ultrasonic sensor connected to the NodeMCU. All these are connected to the processor, in this case a computer. The experimental setup of the proposed system is shown in Figure 11.5.

The dataset used to train the YOLO algorithm to detect images was downloaded from Kaggle. It consists of some basic items in a refrigerator used in our daily lifestyle (Figure 11.6). The efficiency of the object detection system can be improved by using a larger dataset for training.

Figure 11.7 is an image of four capsicums. Figure 11.7a is the result got after training the YOLO model using 100 training images. From the image, we see that along with the four capsicums, it has also given a few other extra detections, and also the detection of the fruits is not fully accurate. It shows some capsicums to be labeled as apples. Figures 11.7b and c show the same image but tested on a model trained by using 200 images and 500 images. They have now detected all the four capsicums and have not detected any other fruit or vegetable. So, the accuracy of the proposed system can be increased by including a larger dataset for training. I have trained my model to detect 14 types of items in the refrigerator: apples, oranges, capsicum, banana, juice box, cake, tomatoes, carrots, grapes, potatoes, chili, lemon, egg, and pineapple.

Figure 11.8 is an image of the inside of a refrigerator. Figure 11.8a is the result got after training the YOLO model using 100 training images. From the image, we see that the model has not detected most of the apples and capsicums. It has also wrongly classified capsicum as an orange. Figure 11.8b shows the same image but tested on a model trained by using 200 images. It has now detected all but one apple. It has detected only one capsicum and has done two misclassifications. Figure 11.8c shows the same image but tested on a model trained by using 500 images. It has detected all fruits except an orange. There are also no misclassifications.

Table 11.3 shows the analysis of training the model. The model currently has an accuracy of about 95%. Ten random images were selected and were fed to the model trained using 100, 200, and 500 images. There were 40 items in all 10 images included. The accuracy was calculated using the following equation:

Figure 11.9 shows the graph of the number of epochs vs. the loss after each epoch. The model was trained for 80 epochs.

Figure 11.10 shows the output of the quality and liquid quantity measuring modules. The first image is for an empty bottle (21 cm in length). The second image is for the same bottle but half full of water (10 cm). Thus, the ultrasonic sensor can be used to determine the approximate amount of liquid in the bottle.

11.5 Conclusion

A smart refrigerator module that can help the user to know items present in the refrigerator without being at home has been developed. This intelligence alerts them to buy any missing items on the way home. The smart module has many advantages. The user does not have to go home, open the fridge, see what items are missing, and then go out to buy the things needed. The need for extra travel is avoided. It saves fuel and time. The product quality monitoring module can detect gases from rotten fruits. One important feature is that the user knows the presence of rotten fruits or vegetables in the refrigerator.

Further training can be done to make the system more accurate—more methods for recognizing spoiled food like image recognition can be introduced. The way of measuring liquid quantities using the ultrasonic sensor is also not very accurate. Measurements can be made better by using other methods like using load cells to calculate the weight of the bottles. To make it autonomous, this system can be connected to an e-commerce website so it can order items that are about to get over.