8.1.3.1 Purpose of Research

This research is concerned with examining how computer vision can be used for vehicle detection. The purpose of this is to serve as an innovative alternative to current traffic flow analysis techniques.

Another goal of this research is to know how innovative technologies may be used to contribute to increase the efficiency of current traffic management approaches in smart cities. This is because in the recent years, computer vision has been noted as an effective and innovative tool in aiding many different areas of general traffic management [5, 6]. Furthermore, a prototype can be designed for capturing traffic flow data, to be evaluated with regard to accuracy and time.

8.1.3.2 Research Questions

The primary research question is as follows:

1. How can computer vision be used to support traffic flow analysis?

Accompanying questions of this research include the following:

2. What is required within the planning, development, and implementation phases of a prototype that performs accurate vehicle detection?
3. How will the performance of a vehicle‐detection prototype be measured?
4. What are the barriers to the development of a vehicle‐detection prototype?

8.1.3.3 Study Aim and Objectives

The central aim of the research is to investigate and discuss a vehicle‐detection prototype. Based on this aim, the three central objectives are as follows:

1. To describe the planning, development, and implementation phases of a modern vehicle‐detection prototype.
2. To propose a modern computer vision‐driven prototype that can perform accurate vehicle detection for traffic flow analysis.
3. To understand how to assess the performance of a prototype based on evaluation criteria.

8.1.3.4 Significance and Structure of the Research

This research offers a unique perspective and serves as a contribution to the small amount of existing literature covering the role of computer vision‐driven traffic solutions within New Zealand and other countries. The significance of this research is the way it informs of how such a traffic monitoring solution might be developed, integrated, and customized. The research also draws attention to a current traffic flow analysis implementation to highlight what changes may be made to strengthen existing processes.

Furthermore, this research explores how incorporating emerging technologies, specifically computer vision‐based approaches within traffic flow analysis in New Zealand, can offer affordances. The significance of this comes from noting how computer vision is explored and employed on a global scale. For instance, successful use cases abroad include the use of computer vision to automate traffic lights to decrease congestion and save costs [7], assess the retroreflectivity of road signs to save time and manual maintenance [8], and estimate traffic flow through automation, minimizing the margin of error and offering higher rates of accuracy [9]. Thus, successful use cases such as the ones highlighted above inform on how computer vision is one of the most popular emerging technologies used within traffic management. However, looking into past usage of computer vision illustrates very few use cases, suggesting that there may be barriers preventing further usage.

This research reports on the investigation, development, and testing and trialing of a computer vision‐driven prototype to perform vehicle detection – with some limitations. The first limitation to note is the lack of information reporting on the specifications of the current traffic flow analysis approaches employed. Although it is apparent what tools are implemented to carry out traffic flow analysis within New Zealand, not having the definite information reporting on factors such as accuracy, time efficiency, and cost makes it difficult to compare the research's prototype against it. The second potential limitation has to do with the current pandemic circumstances. COVID‐19 may impact the research by limiting access to resources required to carry out the research projects. One example of this could be through long wait periods when purchasing required products online from overseas vendors.

Section 8.2 presents a literature review covering relevant academic research within the field of computer vision. Section 8.3 is composed of the methodology portion of the research, addressing the uses of a design‐based research approach and examining the proposed study design and the software and hardware used. This also introduces the methodology including the design and development, testing, analysis, and redesign of the prototype. References are made to the literature, and various insights, benefits, and potential barriers are discussed. Additionally, an outline surrounding the next stages of the research project is presented to be continued in Chapter 9. The last section is the conclusion, which also offers directions for future research.

8.2.2.1 Machine Learning

Machine learning (ML) can be understood as a field of computer science that focuses on the study of computer algorithms to carry out intelligent predictions based on a given data input [10]. As ML algorithms are given more data, they are able to perform better and change or alter how they process data over time by providing feedback on previous performances. This, in turn, serves as an extension off of traditional statistical modeling approaches [11]. Additional reasons for the growing interest in ML include advancements in big data, significant increase in computational power, and large progress in the general development and design of ML algorithms [12].

Some of the most popular and widely used ML algorithms fall under either the categories of supervised learning or unsupervised learning. Supervised learning techniques focus on either classification to predict a data label by mapping inputs to outputs or regression, with the objective to predict a data quantity [13]. Conversely, unsupervised learning techniques focus on providing either clustering, which concentrates on finding any fundamental groupings among data, or dimension reduction, which focuses on reducing the number of variables within data to localize the necessary information [14, 15].

8.2.2.2 Deep Learning

Deep learning is a subset of ML and is based on the communication nodes present within the biological systems of humans [16]. It is important to understand the general concept of deep learning architectures and the structure of an artificial neural network (ANN). The architecture of the ANN is composed of three layers. As illustrated in Figure 8.1, the first layer is the input layer, where data is taken into the system for further processing within the succeeding layers [17]. Once data is taken into the input layer, it is transferred to the hidden layer where the artificial neurons take in a set of weight data inputs and generate a data output through completing an activation function. Lastly, the output layer takes the data and produces the given outputs.

Subsequently, deep learning serves as an extension off of a typical ANN, offering an additional two or more layers, hence gaining the term “deep” learning [18]. There are many different deep learning neural network (DNN) algorithms, with some of the most common being convolutional neural networks (CNNs), recurrent neural networks, recursive neural networks, and unsupervised pretrained networks (UPNs). The common characteristics of deep learning approaches include having strong learning ability and being extremely powerful and efficient when utilizing significantly large datasets [18]. Deep learning requires high‐end graphical processing units and hardware that offer incredibly high performance to process workloads. Nevertheless, overall time requirements are larger [19]. Although these factors can be considered constraints, a higher level of accuracy is often attributed to deep learning approaches [20].

8.2.2.3 Computer Vision

Some examples of computer vision techniques can be demonstrated in subdomains such as image classification, object detection, object tracking, semantic segmentation, and instance segmentation [21]. Taking these subdomains into account, and depending on which technique is used, computer vision supports capabilities that are illustrated through software such as facial recognition, surveillance, biometrics, and self‐driving cars. If developing computer system's objective was to offer speed detection, such an approach would be implemented to achieve this. One example of this approach can be shown within recent research conducted into obstacle detection and classification for high‐speed autonomous driving [22]. To ensure a car is able to achieve full detection and classification of objects, the PASCAL VOC image dataset was selected, chosen as it is composed of many objects common within traffic contexts such as vehicles, pedestrians, and animals. Once the dataset was selected, it was then used to train a region‐based CNN, with a Titan X GPU running on it, processing 10 frames per second (FPS). Due to such a high frame rate, the system was considered suitable for high‐speed driving of autonomous vehicles, and the results from object classification and detection demonstrated full invariance when tested in different conditions such as changes in lighting or climate [22].

8.2.3.1 Object Recognition

Object recognition is a computer vision technique that enables software solutions to recognize object classes within images, video files, or livestream video sequence recordings. Once object classes have been recognized within data, they are then assigned a label that serves as a title [23]. The process of object recognition is composed of the following steps. Data is first collected from images or video files. Once obtained, features can then be extracted from the data through the use of a feature extraction algorithm. This algorithm may concern itself with acquiring certain features within each piece of data it is feed and is performed to differentiate between subsequent object classes later on in the learning process [24]. Once this process has been completed, the features that have been extracted can then be added to the selected object recognition model that will classify the features into different groupings. These groupings will then be used to help assist in analyzing newly input data and, in turn, perform recognition of the object classes that exist within the data. Overall, there are many different types of feature extraction algorithms and object classification models that work together to achieve object recognition.

8.2.3.2 Object Detection

Object detection can be understood as a computer vision technique that enables software solutions to detect and locate objects of certain classes within images, video files, or livestream video sequence recordings. Sometimes, object detection is used interchangeably with the technique of object recognition. However, there is a difference between the two. For instance, object recognition is concerned with recognizing the object within image data, whereas object detection focuses on focuses understanding what the object is but also locating its position within the given data [25]. To further make sense of how object detection functions, it is important to differentiate between some of the common terms commonly used when performing this approach. Below, object classification and object localization will be briefly examined in accordance to their role and function within object detection.

Object classification is one of the first steps necessary in completing object detection, used to predict the class within an image. This is also the core step in completing object recognition [25]. The first step of object classification is to feed a computer vision algorithm input data. Generally, the input data includes images composed of differing objects that the algorithm uses to learn and predict the objects in future data. Once the algorithm is given the input image data, it then outputs the same image data with a class label, such as one or more integers assigned to objects within the image data [26].

Once the computer vision algorithm has completed object classification, it is then able to move on to object localization. Object localization is a computer vision approach used to determine where an object is situated within an image or video file. For the algorithm to be able to correctly identify the location of a specific object, differing approaches are employed, with one of the most common being bounding box regression that is used to refine or predict localization boxes within data. Usually, the bounding box regressors are trained to identify the predefined object classes through regressing either fixed anchor boxes or region proposal networks [27]. Hence, the function of object detection serves to fulfill two main objectives: to comprehend each object class, also known as object classification, and to understand where objects are located within an image, also referred to as object localization [28].

8.2.3.3 Object Tracking

Object tracking is related to how an application can maintain the ability to stay focused on specific objects that have been detected and determining the placement of the object over various video frames. This can be employed on batched video files or in real time. The process of object tracking can be broken down into three main components. The first step includes taking a set of bounding box coordinates from object detection approaches. Once these coordinates have been obtained, an individual ID for each of the initial detections must be created. The final step is then to track each of the detected objects as they change location through each video frame while simultaneously upholding the assignment of each of the IDs [29]. Overall, the process of assigning IDs to each object allows the ability to track them across the video frames.

8.2.4.1 Edge Computing

The first architecture may be evidenced by edge computing. The term edge computing comes about when addressing its main objective, being that it is concerned with processing data at the edge of a network [30]. Edge computing can also be defined as a distributed computing paradigm, where the management of data is completed closer to enterprise applications, such as local servers or IoT devices [31]. Taking this into account, computer vision approaches greatly rely on the speed at which data may be handled to offer real‐time results. Thus, by enabling data processing at different points and allowing data to be processed wherever it is required the most, edge computing offers enterprises the ability to achieve immediate insights that can be transformed into significant processes and practices.

8.2.4.2 Fog Computing

Fog computing is another approach that can be taken to deploying computer vision applications. This, also referred to as fogging or fog networking, can be defined as a scenario where a large amount of heterogeneous, ubiquitous, and decentralized devices connect with one another to undertake processing and storage tasks [32]. These tasks can vary from supporting new services and applications, to assisting with basic networking functions. Furthermore, users that rent out these services are able to receive incentives as compensation. Thus, fog computing is incredibly similar to edge computing in the way that data is managed and processed closer to where it is originally created. Due to this, the terms edge computing and fog computing are often used interchangeably. However, one of the key differences between the two is the location of where the processing of the data takes place [32]. For instance, edge computing focuses on processing data directly on the devices close to where sensors are located, whereas fog computing ordinarily sends the computing power to a local area network (LAN). Once the data has been sent to an LAN, it is then processed either through a node, hub, router, or gateway, and the results of the processing are then sent through to the elected devices [33].

8.2.4.3 Cloud Computing

Lastly, another option may be demonstrated by cloud computing that can be defined as the on‐demand delivery of computer services, including networking, software, storage, servers, and databases. The delivery of these services is offered through the Internet, also recognized by the term the “cloud” [34]. Many of these services are given acronyms, with some of the most popular being SaaS meaning software as a service, IaaS meaning infrastructure as a service, and PaaS meaning platform as a service. Furthermore, these service models are offered for remote use, allowing users utilizing applications the ability to employ them remotely and only needing to pay for as much of the service as they require [35].

8.2.6.1 Big Data Issues

Big data can be understood as information that exceeds processing, storage, and computing capacity of traditional databases and data analysis approaches [36]. Taking this into account, computer vision approaches depend on big data to fulfill their objectives. For instance, one example can be evidenced by a scenario concerned with employing an object detection algorithm for real‐time detection. In this context, not only does object detection require initial training data, but also it must analyze the data presented in real time. Thus, such a reliance on big data often opens itself up to different vulnerabilities.

To prevent some of these barriers, Qiu et al. [37] undertook research into the use of ML processes for big data processing, finding that there are five critical issues that ML may be met with when handling big data. These issues include the (i) volume of data, (ii) variety of data, (iii) velocity of data, (iv) veracity of data, and, lastly, (v) the value of data. For instance, volume of data represents the quantity of data that is available. Although large scales of readily available data are often advantageous, there is also a drawback. For instance, as enterprises regularly amass terabytes to petabytes of information, the volume of big data can be considered insurmountable [38]. Because of information overload, the volume characteristic is considered one of the most significant and distinctive components of big data analysis, prompting the need for specific requirements that must be met to assist in managing it [38]. Thus, the role of computer vision approaches in handling large volumes of big data can be considered an issue not only in its management but also in the way that it must deal with the scale of differing data sources.

8.2.6.2 Privacy Issues

As evidenced above, computer vision greatly relies on big data to function. Hence, another point to consider are the precautions that must be taken to uphold and preserve privacy in certain aspects of data usage. These precautions must be taken to negate any negative consequences that may stem from poor differential privacy [39]. Overall, differential privacy can be defined as an instrument that is able to output information of an underlying dataset and withhold specific data so that it is able to serve its intended purpose. Thus, this instrument completes its objective by offering the required information, such as patterns or insights in the underlying data, but ensuring that sensitive information existing within that same dataset is not compromised [39]. Furthermore, it should be noted that differential privacy is not regarded as a binary notion, but rather a matter of accumulative risk [40]. What this means is that the privacy of an individual is not considered either breached or anonymous, but instead is given parameters that quantify data loss through different mathematical functions.

Computer vision adoption has raised privacy concerns within the facial recognition, facial expression recognition, and scene recognition domains in the recent years. This is largely due to the reason that cameras have become ubiquitous and users not being aware of their presence [41]. This reality not only calls for users to be granted the right to be aware of where cameras are situated but also acknowledges that computer vision‐driven solutions must have adequate control over the real‐time collection and usage of this data.

8.2.6.3 Technical Barriers

There are some current technical barriers that computer vision faces upon its implementation, such as class imbalance problems, limitation of model generalization, absence of scene understanding, and issues faced during image synthesis. Below, the significance and the true meaning of these technical issues will be considered.

One of the most common issues surrounding the implementation of computer vision approaches can be evidenced when addressing class imbalance problems. Class imbalance can be defined as a classification issue, where the distribution of instances across a given dataset is either one sided or skewed. Class imbalance problems can largely vary; however more severe instances such as where high‐class imbalance occurs may call for specialized techniques and approaches to be efficient [42]. Hence, it is evident that effective classification using imbalanced data is an exceptionally important area of research due to the reason that high‐class imbalance inherently exists within many real‐world applications, such as cancer detection and fraud detection [43]. However, it has been noted that only a small amount of empirical research exists surrounding the use of deep learning with existing class imbalance. As a result, present research focuses its efforts on computer vision tasks with CNNs and neglects to consider the effects of big data [43].

A second technical issue to consider when implementing computer vision approaches can be evidenced by model generalization limitations. Model generalization can be defined as a model's ability to adapt properly to previously unseen or used data and draw the same distributions as the ones drawn from the one used to create the initial model with the original data [44]. Furthermore, the act of data collection, data cleaning, data preparation, and building models can all bring different challenges; however, one of the most significant of these challenges is determining how to distinguish if the model will predict future distributions [45]. Taking this into account, computer vision functions by exploiting a dataset to train and assess a model, achieved by splitting up datasets into a training set and a test set [46]. However, the downside to this approach is that the trained model then has the same data distribution across the training and test set, meaning that there is a likelihood that the data distribution will vary from the data used throughout training and testing, such as different camera angles or object scales.

8.3.2.1 Selection Rationale

The primary motivation for selecting the design‐based research methodology can be demonstrated through the objectives both the methodology and the research share in common. As the objective of the research is related to the creation of a prototype through the implementation of a phase‐based approach, this complements the phases also laid out within the design‐based research methodology (Table 8.1).

8.3.2.2 Potential Challenges

Upon selecting the design‐based research methodology, it is essential to identify any potential challenges that may be presented when aligning it with the research to mitigate them. Below, some of the potential challenges that come with electing the design‐based research methodology will be outlined.

It must be noted first that one of the most significant challenges that come with utilizing the design‐based research methodology is how it operates in different research contexts. For instance, the design‐based research methodology places significant emphasis on objectivity, reliability, and validity, illustrating that these features are vital in ensuring that scientifically sound research is effectively produced [48]. However, when these features are managed in a different context, such as in a controlled experimentation environment, the way they are prioritized may also vary. This in itself places the researcher in a difficult position. On the one hand, it is vital that objectivity is advocated for when facilitating meaningful interventions. However, on the other hand, the researcher may find themselves in both of the roles of advocate and critic [48].

A second challenge that may come about may be demonstrated by uncertainty arising when implementing this approach [49]. For instance, it has been posited that the efficacy of the design‐based learning methodology is strongly dependent upon who conducts it. This bears a significant contrast to other approaches such as grounded theory or further dedicated experiments that commonly explain and outline their processes in particular detail. Although this flexibility gives researchers the ability to adopt a range of differing design processes that may fall under the design‐based learning methodology, it is reasoned that some of these processes fail to wholly define the advanced phases of design. Similarly, an additional issue to address is that some of these design frameworks also fail to properly convey how they accompany research components [49].

Taking these challenges into account, there are a few provisions that may be taken to help mitigate them. First, this research may place significant focus on the approach to the design‐based learning methodology for the research objectives and ensuring they fall in line with the defining key phases. Next, the way in which the design‐based methodology complements its research counterpart may be realized through ensuring that a clear definition of its key phases is offered. Through presenting a well‐defined classification of each of the phases, Easterday et al. [49] reasons that it resolves any uncertainty, informs new researchers through illustrating best practices, and, overall, strengthens communication among the research community.

8.3.4.1 Hardware: The NVIDIA Jetson Nano Developer Kit and Accompanying Items

The NVIDIA Jetson Nano Developer Kit can be understood as a small yet strong computer that enables users to run multiple neural networks in parallel and, in turn, allows the ability to harness capabilities such as image classification, segmentation, object detection, recognition, and tracking and implement them into applications in real time [51]. The rationale behind selecting the NVIDIA Jetson Nano Developer Kit over alternative modern day embedded system options is as follows. First, analyzing the current literature concerned with the use of embedded systems for computer vision in traffic management illustrates that one of the most popular choices are the range of Raspberry Pi‐embedded devices [52, 53]. Due to this, there is a gap in the literature surrounding the employment of the NVIDIA Jetson range for traffic management and, in particular, the newest model NVIDIA Jetson Nano Developer Kit, released in March 2019. Additionally, further choice behind the selection of the NVIDIA Jetson Nano Developer Kit can be evidenced by price (NVIDIA^® [54]). One final reason for the choice in the NVIDIA Jetson Nano Developer Kit can be evidenced by the objectives set out by the research, as well as analyzing the requirements necessary to design and develop the vehicle detection prototype. For instance, as the research focuses on specifically vehicle detection approaches, it requires a significant amount of processing power. Hence, composed of a Quad‐core ARM Cortex ‐A57 MPCore central processing unit (CPU), the NVIDIA Jetson Nano Developer Kit offers a sufficient processing power suitable for modern computer vision applications [55].

To use the NVIDIA Jetson Nano Developer Kit, a microSD card, micro‐USB power supply, WiFi supply, a computer display using either HDMI or DP, and a USB keyboard and mouse are required. Taking this into account, it is recommended to utilize a microSD card that holds 16 gigabytes (GB) minimum [51]. Thus, a 64 GB microSD card was obtained for the prototype. Moreover, it is also recommended to use a 5v2A micro‐USB power supply and so this was also obtained.

The NVIDIA Jetson Nano Developer Kit also has camera capabilities, and to utilize this feature for the development of the prototype, a Raspberry Pi V2 camera module was obtained due to its cost‐effective price and its compatibility across a wide range of differing edge computing devices [56] (Table 8.6).

8.3.5.1 Software Stack: NVIDIA Jetpack SDK and Accompanying Requirements (All Iterations)

The underlying software considered foundational is the NVIDIA Jetpack Software Development Kit (SDK). This is one of the most comprehensive solutions for building applications and solutions leveraging deep learning on the NVIDIA Jetson Nano Developer Kit as it is composed of many different components that may be used for design and development, including a range of software libraries and application programming interfaces (APIs) [57]. Furthermore, the NVIDIA Jetpack SDK also includes the latest operating system (OS) images for the NVIDIA Jetson range and sample applications and demos, developer tools, and documentation [57]. Each time the Jetpack SDK is updated to a newer version, so too are the versions of the libraries included within the Jetpack SDK version. Both versions of Jetpack 4.2 and 4.4 have been obtained and utilized for the purpose of design and development of the prototype by being written to two identical microSD cards to test out approaches that utilize both older and newer versions of the software (Table 8.7).