Classification

We have looked at several examples of classification throughout this book. Simply put, the task of classification assigns an image with the class(es) of the object(s) present in the image. It answers the question: “Is there an object of class X in the image?” Classification is one of the most fundamental computer-vision tasks. In an image, there might be objects of more than one class—called multiclass classification. The label in this case for one image would be a list of all object classes that the image contains.

Localization

The downside of classification is that it does not tell us where in the image a particular object is, how big or small it is, or how many objects there are. It only tells us that an object of a particular class exists somewhere within it. Localization, otherwise known as classification with localization, can inform us which classes are in an image and where they are located within the image. The keyword here is classes, as opposed to individual objects, because localization gives us only one bounding box per class. Just like classification, it cannot answer the question “How many objects of class X are in this image?”

As you will see momentarily, localization will work correctly only when it is guaranteed that there is a single instance of each class. If there is more than one instance of a class, one bounding box might encompass some or all objects of that class (depending upon the probabilities). But just one bounding box per class seems rather restrictive, doesn’t it?

Detection

When we have multiple objects belonging to multiple classes in the same image, localization will not suffice. Object detection would give us a bounding rectangle for every instance of a class, for all classes. It also informs us as to the class of the object within each of the detected bounding boxes. For example, in the case of a self-driving car, object detection would return one bounding box per car, person, lamp post, and so on that the car sees. Because of this ability, we also can use it in applications that need to count. For example, counting the number of people in a crowd.

Unlike localization, detection does not have a restriction on the number of instances of a class in an image. This is why it is used in real-world applications.

Segmentation

Object segmentation is the task of assigning a class label to individual pixels throughout an image. A children’s coloring book where we use crayons to color various objects is a real-world analogy. Considering each pixel in the image is being assigned a class, it is a highly compute-intensive task. Although object localization and detection give bounding boxes, segmentation produces groups of pixels—also known as masks. Segmentation is obviously much more precise about the boundaries of an object compared to detection. For example, a cosmetic app that changes a user’s hair color in real-time would use segmentation. Based on the types of masks, segmentation comes in different flavors.

Instance-level segmentation

Given an image, instance-level segmentation identifies the area occupied by each instance of each class. Different instances of the same class will be segmented uniquely.

Table 14-1 lists the different computer-vision tasks.

Table 14-1. Types of computer-vision tasks illustrated using image ID 120524 from the MS COCO dataset

Task	Image	In lay terms	Outputs
Object classification		Is there a sheep in this image?	Class probabilities
Object localization		Is there “a” sheep in the image and where is it?	Bounding box and class probability
Object detection		What and where are “all” the objects in this image?	Bounding boxes, class probabilities, class ID prediction
Semantic segmentation		Which pixels in this image belong to different classes; e.g., the class “sheep,” “dog,” “person”?	One mask per class
Instance-level segmentation		Which pixels in this image belong to each instance of each class; e.g., “sheep,” “dog,” “person”?	One mask per instance of each class

Performance Considerations

From a practitioner’s point of view, the primary concerns tend to be accuracy and speed. These two factors usually have an inverse relationship with each other. We want to find the detector that strikes the ideal balance for our scenario. Table 14-4 summarizes some of those readily available pretrained models on the TensorFlow object detection model zoo. The speed reported was on a 600x600 pixel resolution image input, processed on an NVIDIA GeForce GTX TITAN X card.

A 2017 study done by Google (see Figure 14-10) revealed the following key findings:

• One-stage detectors are faster than two-stage, though less accurate.

• The higher the accuracy of the backbone CNN architecture on classification tasks (such as ImageNet), the higher the precision of the object detector.

• Small-sized objects are challenging for object detectors, which give poor performance (usually under 10% mAP). The effect is more severe on one-stage detectors.

• On large-sized objects, most detectors give similar performance.

• Higher-resolution images give better results because small objects appear larger to the network.

• Two-stage object detectors internally generate a large number of proposals to be considered as the potential locations of objects. The number of proposals can be tuned. Decreasing the number of proposals can lead to speedups with little loss in accuracy (the threshold will depend on the use case).

Figure 14-10. The effect of object detection architecture as well as the backbone architecture (feature extractor) on the percent mean average precision and prediction time (note that the colors might not be clear in grayscale printing; refer to the book’s GitHub website [see http://PracticalDeepLearning.ai] for the color image) (image source: Speed/accuracy trade-offs for modern convolutional object detectors by Jonathan Huang et al.)

Intersection over Union

In an object detection training pipeline, metrics such as Intersection over Union (IoU) are typically used as a threshold value to filter detections of a certain quality. To calculate the IoU, we use both the predicted bounding box and the ground truth bounding box to calculate two different numbers. The first is the area where both the prediction and the ground truth overlap—the intersection. The second is the span covered by both the prediction and the ground truth—the union. As the name suggests, we simply divide the total area of the intersection by the area of the union to get the IoU. Figure 14-11 visually demonstrates the IoU concept for two 2x2 squares that intersect in a single 1x1 square in the middle.

Figure 14-11. A visual representation of the IoU ratio

In the perfect scenario in which the predicted bounding box matches the ground truth exactly, the IoU value would be 1. In the worst-case scenario, the prediction would have no overlap with the ground truth and the IoU value would be 0. As we can see, the IoU value would range between 0 and 1, with higher numbers indicating higher-quality predictions, as illustrated in Figure 14-12.

To help us filter and report results, we would set a minimum IoU threshold value. Setting the threshold to a really aggressive value (such as 0.9) would result in a loss of a lot of predictions that might turn out to be important further down the pipeline. Conversely, setting the threshold value too low would result in too many spurious bounding boxes. A minimum IoU of 0.5 is typically used for reporting the precision of an object detection model.

Figure 14-12. IoU illustrated; predictions from better models tend to have heavier overlap with the ground truth, resulting in a higher IoU

It’s worth reiterating here that the IoU value is calculated per instance of a category, not per image. However, to calculate the quality of the detector over a larger set of images, we would use IoU in conjunction with other metrics such as Average Precision.

Mean Average Precision

While reading research papers on object detection, we often come across numbers such as [email protected]. That term conveys the average precision at IoU=0.5. Another more elaborate representation would be AP@[0.6:0.05:0.75], which is the average precision from IoU=0.6 to IoU=0.75 at increments of 0.05. The Mean Average Precision (or mAP) is simply the average precision across all categories. For the COCO challenge, the MAP metric used is AP@[0.5:0.05:0.95].

Non-Maximum Suppression

Internally, object detection algorithms make a number of proposals for the potential locations of objects that might be in the image. For each object in the image, it is expected to have multiple of these bounding box proposals at varying confidence values. Our task is to find the one proposal that best represents the real location of the object. A naïve way would be to consider only the proposal with the maximum confidence value. This approach might work if there were only one object in the entire image. But this won’t work if there are multiple categories, each with multiple instances in a single image.

Non-Maximum Suppression (NMS) comes to the rescue (Figure 14-13). The key idea behind NMS is that two instances of the same object will not have heavily overlapping bounding boxes; for instance, the IoU of their bounding boxes will be less than a certain IoU threshold (say 0.5). A greedy way to achieve this is by repeating the following steps for each category:

1. Filter out all proposals under a minimum confidence threshold.

2. Accept the proposal with the maximum confidence value.

3. For all remaining proposals sorted in descending order of their confidence values, check whether the IoU of the current box with respect to one of the previously accepted proposals ≥ 0.5. If so, filter it; otherwise, accept it as a proposal.

   

   Figure 14-13. Using NMS to find the bounding box that best represents the location of the object in an image

Data Collection

We know by now that, in the world of deep learning, data reigns supreme. There are a few ways in which we can acquire data for the object that we want to detect:

Using ready-to-use datasets

There are a few public datasets available to train object detection models such as MS COCO (80 categories), ImageNet (200 categories), Pascal VOC (20 categories), and the newer Open Images (600 categories). MS COCO and Pascal VOC are used in most object detection benchmarks, with COCO-based benchmarks being more realistic for real-world scenarios due to more complex imagery.

Downloading from the web

We should already be familiar with this process from Chapter 12 in which we collected images for the “Hot Dog” and “Not Hot Dog” classes. Browser extensions such as Fatkun come in handy for quickly downloading a large number of images from search engine results.

Taking pictures manually

This is the more time consuming (but potentially fun) option. For a robust model, it’s important to train it with pictures taken in a variety of different settings. With the object to detect in hand, we should take at least 100 to 150 images of it against different backgrounds, with a variety of compositions (framing) and angles, and in many different lighting conditions. Figure 14-14 shows some examples of pictures taken to train a Coke and Pepsi object detection model. Considering that a model can potentially learn the spurious correlation that red means Coke and blue means Pepsi, it’s a good idea to mix objects with backgrounds that could potentially confuse it so that eventually we realize a more robust model.

Figure 14-14. Photographs of objects taken in a variety of different settings to train an object detector model

It’s worth taking pictures of the object in environments that it’s unlikely to be used in order to diversify the dataset and improve the robustness of the model. It also has the added bonus of keeping an otherwise boring and tedious process a little entertaining. We can make it interesting by challenging ourselves to come up with unique and innovative photos. Figure 14-15 shows some creative examples of photos taken during the process of building a currency detector.

Figure 14-15. Some creative photographs taken during the process of building a diverse currency dataset

Because we want to make a cat detector, we will be reusing the cat images from the “Cats and Dogs” Kaggle dataset that we used in Chapter 3. We randomly chose images from that set and split them into a train and test set.

To maintain uniformity of input size to our network, we resize all images to a fixed size. We will use the same size as part of our network definition and while converting to a .tflite model. We can use the ImageMagick tool to resize all images at once:

$ apt-get install imagemagick
$ mogrify -resize 800x600 *.jpg

$ find . -type f | awk -F. '!a[$NF]++{print $NF}' |
xargs -I{} mogrify -resize 800x600 *.jpg

Labeling the Data

After we have our data, the next step would be to label it. Unlike classification where simply placing an image in the correct directory would be sufficient, we need to manually draw the bounding rectangles for the objects of interest. Our tool of choice for this task is LabelImg (available for Windows, Linux, and Mac) for the following reasons:

• You can save annotations either as XML files in PASCAL VOC format (also adopted by ImageNet) or the YOLO format.

• It supports single and multiclass labels.

class_for_box1 box1_x box1_y box1_w box1_h
class_for_box2 box2_x box2_y box2_w box2_h

First, download the application from GitHub to a directory on your computer. This directory will have an executable and a data directory containing some sample data. Because we will be using our own custom data, we don’t need the data within the provided data directory. You can start the application by double-clicking the executable file.

At this point, we should have a collection of images that is ready to be used during the training process. We first divide our data randomly into training and test sets and place them in separate directories. We can do this split by simply dragging and dropping the images randomly into either directory. After the train and test directories are created, we load the training directory by clicking Open Dir, as shown in Figure 14-16.

Figure 14-16. Click the Open Dir button and then select the directory that contains the training data

After LabelImg loads the training directory, we can start our labeling process. We must go through each image and manually draw bounding boxes for each object (only cats in our case), as shown in Figure 14-17. After we make the bounding box, we are prompted to provide a label to accompany the bounding box. For this exercise, enter the name of the object, “cat”. After a label is entered, we need only to select the checkbox to specify that label again for subsequent images. For images with multiple objects, we would make multiple bounding boxes and add their corresponding labels. If there are different kinds of objects, we simply add a new label for that object class.

Figure 14-17. Select the Create RectBox from the panel on the left to make a bounding box that covers the cat

Now, repeat this step for all of the training and test images. We want to obtain tight bounding boxes for each object, such that all parts of the object are bounded by the box. Finally, in the train and test directories, we see .xml files accompanying each image file, as depicted in Figure 14-18. We can open the .xml file in a text editor and inspect metadata such as image file name, bounding box coordinates, and label name.

Figure 14-18. Each image is accompanied by an XML file that contains the label information and the bounding box information

Preprocessing the Data

At this point, we have handy XML data that gives the bounding boxes for all of our objects. However, for us to use TensorFlow for training on the data, we must preprocess it into a format that TensorFlow understands, namely TFRecords. Before we can convert our data into TFRecords, we must go through the intermediate step of consolidating the data from all the XML files into a single comma-separated values (CSV) file. TensorFlow provides helper scripts that assist us with these operations.

Now that we have our environment set up, it’s time to begin doing some real work:

1. Convert our cats dataset directory to a single CSV file using the xml_to_csv tool from the racoon_dataset repository by Dat Tran. We have provided a slightly edited copy of this file in our repository at code/chapter-14/xml_to_csv.py:

   $ python xml_to_csv.py -i {path to cats training dataset} -o {path to output
   train_labels.csv}

2. Do the same for the test data:

   $ python xml_to_csv.py -i {path to cats test dataset} -o {path to
   test_labels.csv}

3. Make the label_map.pbtxt file. This file contains the label and identifier mappings for all of our classes. We use this file to convert our text label into an integer identifier, which is what TFRecord expects. Because we have only one class, the file is rather small and looks as follows:

   item {
       id: 1
       name: 'cat'
   }

4. Generate the TFRecord format files that will contain the data to be used for training and testing our model later on. This file is also from the racoon_dataset repository by Dat Tran. We have provided a slightly edited copy of this file in our repository at code/chapter-14/generate_tfrecord.py. (It’s worth noting that the image_dir path in the argument should be the same as the path in the XML files; LabelImg uses absolute paths.)

   $ python generate_tfrecord.py 
   --csv_input={path to train_labels.csv} 
   --output_path={path to train.tfrecord} 
   --image_dir={path to cat training dataset}
   
    
   $ python generate_tfrecord.py 
   --csv_input={path to test_labels.csv} 
   --output_path={path to test.tfrecord} 
   --image_dir={path to cat test dataset}

With our train.tfrecord and test.tfrecord files ready, we can now begin our training process.

Training

Because we are using SSD MobileNetV2, we will need to create a copy of the pipeline.config file (from the TensorFlow repository) and modify it based on our configuration parameters. First, make a copy of the configuration file and search for the string PATH_TO_BE_CONFIGURED within the file. This parameter indicates all of the places that need to be updated with the correct paths (absolute paths preferably) within our filesystem. We will also want to edit some parameters in the configuration file, such as number of classes (num_classes), number of steps (num_steps), number of validation samples (num_examples), and path of label mappings for both the train and test/eval sections. (You’ll find our version of the pipeline.config file on the book’s GitHub website (see http://PracticalDeepLearning.ai).

$ cp object_detection/samples/configs/ssd_mobilenet_v2_coco.config
./pipeline.config
$ vim pipeline.config

So far, we’ve been at the models/research directory within the TensorFlow repository. Let’s now move into the object_detection directory and run the model_main.py script from TensorFlow, which trains our model based on the configuration provided by the pipeline.config file that we just edited:

$ cd object_detection/
$ python model_main.py 
--pipeline_config_path=../pipeline.config 
--logtostderr 
--model_dir=training/

We’ll know the training is happening correctly when we see lines of output that look like the following:

Average Precision (AP) @[ IoU=0.50:0.95 | area=all | maxDets=100 ] = 0.760

Depending on the number of iterations that we are training for, the training will take from a few minutes to a couple of hours, so plan to have a snack, do the laundry, clean the litter box, or better yet, read the chapter ahead. After the training is complete, we should see the latest checkpoint file in the training/ directory. The following files are also created as a result of the training process:

$ ls training/

checkpoint                                   model.ckpt-13960.meta
eval_0                                       model.ckpt-16747.data-00000-of-00001
events.out.tfevents.1554684330.computername  model.ckpt-16747.index
events.out.tfevents.1554684359.computername  model.ckpt-16747.meta
export                                       model.ckpt-19526.data-00000-of-00001
graph.pbtxt                                model.ckpt-19526.index
label_map.pbtxt                            model.ckpt-19526.meta
model.ckpt-11180.data-00000-of-00001       model.ckpt-20000.data-00000-of-00001
model.ckpt-11180.index                     model.ckpt-20000.index
model.ckpt-11180.meta                      model.ckpt-20000.meta
model.ckpt-13960.data-00000-of-00001       pipeline.config
model.ckpt-13960.index

We convert the latest checkpoint file to the .TFLite format in the next section.

$ python export_inference_graph.py 
--input_type=image_tensor 
--pipeline_config_path=training/pipeline.config 
--output_directory=inference_graph 
--trained_checkpoint_prefix=training/model.ckpt-20000

=========================Options=========================
-max_depth                  10000
-step                       -1
...

==================Model Analysis Report==================
...
Doc:
scope: The nodes in the model graph are organized by
their names, which is hierarchical like filesystem.
param: Number of parameters (in the Variable).

Profile:
node name | # parameters
_TFProfRoot (--/3.72m params)
 BoxPredictor_0 (--/93.33k params)
    BoxPredictor_0/BoxEncodingPredictor
    (--/62.22k params)
      BoxPredictor_0/BoxEncodingPredictor/biases
      (12, 12/12 params)
 ...     
FeatureExtractor (--/2.84m params)
...

Model Conversion

Now that we have the latest checkpoint file (the suffix should match the number of epochs in the pipeline.config file), we will feed it into the export_tflite_ssd_graph script like we did earlier in the chapter with our pretrained model:

$ python export_tflite_ssd_graph.py 
--pipeline_config_path=training/pipeline.config 
--trained_checkpoint_prefix=training/model.ckpt-20000 
--output_directory=tflite_model 
--add_postprocessing_op=true

If the preceding script execution was successful, we’ll see the following files in the tflite_model directory:

$ ls tflite_model
tflite_graph.pb
tflite_graph.pbtxt

We have one last step remaining: convert the frozen graph files to the .TFLite format. We can do that using the tflite_convert tool:

$ tflite_convert --graph_def_file=tflite_model/tflite_graph.pb 
--output_file=tflite_model/cats.tflite 
--input_arrays=normalized_input_image_tensor 
--output_arrays='TFLite_Detection_PostProcess','TFLite_Detection_PostProcess:1','TFLi
te_Detection_PostProcess:2','TFLite_Detection_PostProcess:3' 
--input_shape=1,800,600,3 
--allow_custom_ops

In this command, notice that we’ve used a few different arguments that might not be intuitive at first sight:

• For --input_arrays, the argument simply indicates that the images that will be provided during predictions will be normalized float32 tensors.

• The arguments supplied to --output_arrays indicate that there will be four different types of information in each prediction: the number of bounding boxes, detection scores, detection classes, and the coordinates of bounding boxes themselves. This is possible because we used the argument --add_postprocessing_op=true in the export graph script in the previous step.

• For --input_shape, we provide the same dimension values as we did in the pipeline.config file.

The rest are fairly trivial. We should now have a spanking new cats.tflite model file within our tflite_model/ directory. It is now ready to be plugged into an Android, iOS, or edge device to do live detection of cats. This model is well on its way to saving Bob’s garden!

Smart Refrigerator

Smart fridges have been gaining popularity for the past few years because they have started to become more affordable. People seem to like the convenience of knowing what’s in their fridge even when they are not home to look inside. Microsoft teamed up with Swiss manufacturing giant Liebherr to use deep learning in its new generation SmartDeviceBox refrigerator (Figure 14-21). The company used Fast R-CNN to perform object detection within its refrigerators to detect different food items, maintain an inventory, and keep the user abreast of the latest state.

Figure 14-21. Detected objects along with their classes from the SmartDeviceBox refrigerator (image source)

Crowd Counting

Counting people is not just something that only cats do while staring out of windows. It’s useful in a lot of situations including managing security and logistics at large sporting events, political gatherings, and other high-traffic areas. Crowd counting, as its name suggests, can be used to count any kind of object including people and animals. Wildlife conservation is an exceptionally challenging problem because of the lack of labeled data and intense data collection strategies.

Wildlife conservation

Several organizations, including the University of Glasgow, University of Cape Town, Field Museum of Natural History, and Tanzania Wildlife Research Institute, came together to count the largest terrestrial animal migration on earth: 1.3 million blue wildebeest and 250,000 zebra between the Serengeti and the Masaai Mara National Reserve, Kenya (Figure 14-22). They employed data labeling through 2,200 volunteer citizen scientists using a platform called Zooniverse and used automated object detection algorithms like YOLO for counting the wildebeest and zebra population. They observed that the count from the volunteers and from the object detection algorithms were within 1% of each other.

Figure 14-22. An aerial photograph of wildebeest taken from a small survey airplane (image source)

Kumbh Mela

Another example of dense crowd counting is from Prayagraj, India where approximately 250 million people visit the city once every 12 years for the Kumbh Mela festival (Figure 14-23). Crowd control for such large crowds has historically been difficult. In fact, in the year 2013, due to poor crowd management, a stampede broke out tragically resulting in the loss of 42 lives.

In 2019, the local state government contracted with Larsen & Toubro to use AI for a variety of logistical tasks including traffic monitoring, garbage collection, security assessments, and also crowd monitoring. Using more than a thousand CCTV cameras, authorities analyzed crowd density and designed an alerting system based on the density of people in a fixed area. In places of higher density, the monitoring was more intense. It made the Guinness Book of World Records for being the largest crowd management system ever built in the history of humankind!

Figure 14-23. The 2013 Kumbh Mela, as captured by an attendee (image source)

Face Detection in Seeing AI

Microsoft’s Seeing AI app (for the blind and low-vision community) provides a real-time facial detection feature with which it informs the user of the people in front of the phone’s camera, their relative locations, and their distance from the camera. A typical guidance might sound like “One face in the top-left corner, four feet away.” Additionally, the app uses the facial-detection guidance to perform facial recognition against a known list of faces. If the face is a match, it will announce the name of the person, as well. An example of vocal guidance would be “Elizabeth near top edge, three feet away,” as demonstrated in Figure 14-24.

To identify a person’s location in the camera feed, the system is running on a fast mobile-optimized object detector. A crop of this face is then passed for further processing to age, emotion, hair style, and other recognition algorithms on the cloud from Microsoft’s Cognitive Services. For identifying people in a privacy-friendly way, the app asks the user’s friends and family to take three selfies of their faces and generates a feature representation (embedding) of the face that is stored on the device (rather than storing any images). When a face is detected in the camera live feed in the future, an embedding is calculated and compared with the database of embeddings and associated names. This is based on the concept of one-shot learning, similar to Siamese networks that we saw in Chapter 4. Another benefit of not storing images is that the size of the app doesn’t increase drastically even with a large number of faces stored.

Figure 14-24. Face detection feature on Seeing AI

Autonomous Cars

Several self-driving car companies including Waymo, Uber, Tesla, Cruise, NVIDIA, and others use object detection, segmentation, and other deep learning techniques for building their self-driving cars. Advanced Driver Assistance Systems (ADAS) such as pedestrian detection, vehicle type identification, and speed limit sign recognition are important components of a self-driving car.

For the decision-making involved in self-driving cars, NVIDIA uses specialized networks for different tasks. For example, WaitNet is a neural network used for doing fast, high-level detection of traffic lights, intersections, and traffic signs. In addition to being fast, it is also meant to withstand noise and work reliably in tougher conditions such as rain and snow. The bounding boxes detected by WaitNet are then fed to specialized networks for more detailed classification. For instance, if a traffic light is detected in a frame, it is then fed into LightNet—a network used for detecting traffic light shape (circle versus arrow) and state (red, yellow, green). Additionally, if a traffic sign is detected in a frame, it’s fed into SignNet, a network to classify among a few hundred types of traffic signs from the US and Europe, including stop signs, yield signs, directional information, speed limit information, and more. Cascading output from faster networks to specialized networks helps improve performance and modularize development of different networks.

Figure 14-25. Detecting traffic lights and signs on a self-driving car using the NVIDIA Drive Platform (image source)