Chapter 3
Theory of HiTLCPSs
So, what exactly is a HiTLCPS? Understanding the principles and theory behind these systems is the main objective of the current chapter. First, in Section 3.1, we will establish a taxonomy that will be used throughout the book. Then, in Sections 3.2 through 3.4, we will address the human role in the three basic processes of HiTLCPSs, namely data acquisition, state inference, and actuation, respectively.
3.1 Taxonomies for HiTLCPSs
As HiTLCPSs have a wide spectrum of applicability, it is difficult to cover all possible examples of HiTL solutions for a multitude of domains. Thus, we will begin by presenting the most basic, common processes in HiTL control, shown in Figure 3.1. The first phase is known as “data acquisition”. Data related to the human individual is gathered from the available sensors. This data is then processed in the “state inference” stage with the objective of inferring the human's physical and/or psychological state. Some approaches may also attempt to predict future states based on historical data and the current state. Finally, in the “actuation” stage, the system may or may not perform certain actions based on the observed state. Some “open-loop” systems do not affect the system per se, that is their results are merely informative, without direct actuation. However, “closed-loop” systems actuate directly on the environment or the human, in order to influence the loop and achieve a given target.
Figure 3.1 Basic processes of human-in-the-loop control.
From now on we will call this reference model the Internet of all (IoA), meaning that it includes not only (traditional) IoT but also humans as fundamental elements. In this way, we emphasize that this Internet is made by humans, for humans, and with humans.
IoA is built from spatially distributed devices that are considered by standard IoT, like laptops, mobile phones, computers, sensors, actuators, “classic” network elements (we mean all passive elements like routers, switches, access points, etc.), RFID tags, readers, cars, intelligent clothes, wearable devices, furniture, and home appliances. As can be inferred from the general model in Figure 3.1, IoA also includes robotics and its interaction with intelligent devices and sensors. However, on top of these man-made devices, we also consider human beings themselves as part of the system: their actions, drives, desires, and emotions.
How can we distinguish between different types of HiTLCPSs? In order to better comprehend the dimension of such an expansive field as HiTLCPSs, it is important to resort to taxonomies that allow us to better structure our ideas and concepts.
Some authors have previously proposed taxonomic distinctions and classifications for HiTL systems based on the type of exerted control. In an attempt to attain a greater understanding of the spectrum of HiTL applications, and of their underlying principles, requirements, and models, Stankovic et al. [45] began to establish a taxonomic foundation for HiTLCPSs applications. According to Stankovic et al. [45], it is possible to organize existing HiTL applications into three types: (1) applications where humans directly control the system, (2) applications where the system passively monitors humans and takes appropriate actions, and (3) a hybrid of (1) and (2). These three basic types are represented in Figure 3.2, and are described below.
Figure 3.2 Taxonomy of human control.
Human Control: there are two main scenarios where humans directly control CPSs. In supervisory control scenarios, human operators oversee an otherwise mainly autonomous process. The operators are responsible for adjusting certain set points that may influence the system. This is the case of, for example, industrial scenarios where operators mainly set or adjust certain target parameter values that are then enforced by an autonomous robotic CPS. If humans have a more direct command over the process, we are in the presence of direct control scenarios. These are typical master slave scenarios where humans issue commands to the CPS, which then carries the necessary actions, and reports back the results. An example of such a system can be seen in [74], where a wheelchair-mounted robotic arm is controlled by a disabled person to retrieve objects.
Human Monitoring: applications that passively monitor humans, also known as people-centric sensing applications, use their monitoring data to take appropriate actions. In the scope of CPSs, these can be of two types: open-loop and closed-loop systems. Open-loop systems monitor information about humans regarding several aspects (e.g. sleep quality, physical activity, attention-level) and report these results. One example is Look4MySounds, a remote monitoring platform for auscultation of cardiac sounds and automatic detection of pathologies [75]. The platform uses an integrated stethoscope with which auscultation sounds are recorded and processed for automatically detecting pathologies. The sound samples and obtained diagnosis are thereafter remotely sent to a clinician. Despite the human being in the loop, the system does not take any proactive actions and simply relays the results to a specialized medical practitioner. On the other hand, closed-loop systems use their sensory data and processing results in order to actively contribute to a specific goal. For example, a smartshirt may monitor a human's exercise levels at the gym, while a sensor placed on the wall monitors the room temperature. When the human is exercising, the HiTL control may signal the heating, ventilation, and cooling (HVAC) systems to reduce the room's temperature in order to make the exercise more pleasurable.
Hybrid Systems: hybrid systems take people-centric sensing information as feedback to their control-loops while also taking direct human inputs into consideration. Let us expand our smartshirt example to include a smartphone application that allows the user to keep track of his/her exercise and also set a desired room temperature. The hybrid system could take the user's desired temperature as input while using the activity information to fine-tune the absolute temperature value, or to control the rate of temperature change.
Stankovich et. al. [45] only classified the different types of HiTL applications according to how humans interact with them. In this book, we intend to go a little further and provide an alternative point of view of the HiTL process. We will expand this taxonomic exercise to also consider the possible roles of humans in these systems. We believe such a distinction is important, since it will allow us to better cope with some of the existing challenges, such as determining how to incorporate human behavior into the methodology of feedback control. Thus, in the following text, we establish a classification of human roles within future IoT.
How can humans contribute to CPSs? Human presence can manifest itself in different ways: humans may acquire data by themselves, may provide assessment of situations, and may also actuate when necessary. Thus, we would like to reflect not simply on where to place the model of human behavior within the control loop but also understand what roles a human may play within an HiTLCPS and how to best explore this resource; not as an external and unpredictable element but as an inherent part of the system. Figure 3.3 summarizes our understanding of the possible human roles within HiTLCPSs.
Figure 3.3 Taxonomy of human roles.
The next subsections detail the human role in each of the identified categories.
3.2 Data Acquisition
3.2.1 Humans as Sets of Sensors
There are several ways a human can act as a sensor, or even as a set of sensors. Whenever the human is capable of directly feeding the system with information, we can say that human is acting as a sensor. For example, each time a user indicates that he likes something on a social network, gives feedback for machine learning, or provides collaborative information for crowd sourcing (e.g. indicates that a road is blocked on a collaborative navigation app), that user is acting as a human sensor. Human-provided information has several advantages. It tends to be of a more abstract/complex type, and it may be easier and less expensive to obtain than the one provided by sensor machines, as most data is provided voluntarily and without the need for additional hardware. Taking our road example, it is difficult to reliably detect a blockage without a considerable amount of sensors (infrared proximity, cameras, etc.) and signal processing, which can end up being rather expensive even for very short sections of the road. However, a human with a smartphone can easily take pictures, comment, and report such blockages in a way that is useful for the rest of the HiTLCPS. On the other hand, this information is also more difficult to parse: rarely do people communicate in a machine-readable way. At the same time, this source of information is also more unreliable: unlike people, machines do not tend to lie or misinform (unless they are broken). Hence, most of the effort and cost of using human-derived information comes from its parsing and validation: depending on the use-case, it may be useful to use it or not.
Another way in which humans become an integral part of sensor networks is through the sensors that they carry. Wearable devices, such as smartphones, smartwatches, or intelligent clothing, can also become important elements in the future Internet. Some years from now, nano-technology might also become an important element in this regard; some researchers point out how this technology can bring intra-body elements into the IoA [76]. Nano-networks have been receiving a lot of attention from the scientific community and, in the near future, new studies and prototypes will begin to emerge that might result in very advanced applications in the biomedical area. HiTL concepts will certainly apply to these types of scenarios. The source of information is definitely human-influenced but, at the same time, machine-derived, as this mitigates many of the limitations associated with information derived exclusively from humans. Still, depending on the use-case, the usage of these sensors in a useful way may require a considerable amount of integration and processing.
The concept of humans as sensors extends beyond the use of direct human feedback and sensor devices. Social-networking, for example, also serves as a rich information source. In the not-too-distant future, this information could be combined from both sources, with sensor nodes placed in major shopping centers to, for instance, help and support the shopping of human beings. Smart glasses could overlay price-tags on the products of its user's interests (i.e. on things that they “liked”).
We can talk about two types of sensing in these scenarios: direct sensing and indirect sensing. Direct sensing involves using sensors or human feedback that is directly related to the sensing target. On our shopping mall example, using GPS localization or a questionnaire asking the user which stores he visited are direct ways of determining his shopping habits. On the other hand, indirect sensing refers to a case where we infer desired information from other responses. For example, by using the shopping mall building's vibration sensors [77], or even by aggregating information from the shopping mall's information terminals, one can infer which floors or shops tend to attract most interest and customers.
While these sorts of applications are not unfeasible, concerns over the intrusiveness of such practices are more than justified. Thus, as we will see in later chapters, it is also the responsibility of HiTLCPSs to ensure the privacy of their users, enforcing strict control over the sharing of personal information.
3.2.2 Humans as Communication Nodes
As social creatures, humans are masters of communication. Many of the technologies that we see all around us were built specifically to improve this faculty: televisions, radios, telephones, the Internet; they all have the purpose of conveying messages. Thus, this ability of spreading information is not to be undervalued within HiTLCPS: messages, photographs, tweets, comments, and posts are all perfectly valid subjects of communication.
In fact, some researchers have pointed out that these data items may enable faster and more reliable communication than traditional “news” media outlets. Wang et. al. have explored these ideas in their 2015 book “Social Sensing: Building Reliable Systems on Unreliable Data” [78]. According to the authors, human beings are “sensors engaging directly with the mobile Internet”, emphasizing the key problem of extracting reliable information from data collected from largely unknown and possibly unreliable sources. The book explains how myriad societal applications can be derived from the massive amount of data collected and shared by average individuals. In other words, how can we know if this human communication is reliable? In the authors' opinion, the rate of human information generation has long outclassed humans' own cognitive ability for processing it. Thus, new algorithms are needed to preserve the quality of information as much as possible. Ascertaining the correctness of reported information is referred to as the “truth estimation problem”, and it affects the ability of humans to act as both sensors and communication nodes.
Nevertheless, much like when acting as sensors, the ability of communicating information is greatly increased when humans and machines work together. Multi-hop is a very common technique used by tiny devices to save energy. Intermediate nodes can be used in a communication process between a sender node and a receiver node to reduce the required signal power. In this context, human devices such as smartphones and body-area sensors may also be used as intermediate nodes in the “hopping” process, taking advantage of human mobility and intelligence for distributing information more effectively in the network. This may be particularly useful in, for example, metropolitan-wide collaboration systems, where human presence and mobility may be crucial in re-passing non-critical information about the environment. Instead of using multi-hopping or long-distance communication between sensor nodes to, for example, monitor temperature, this information might be aggregated and stored by human-carried devices as people move around the city, opportunistically forwarding it when appropriate, thus, reducing the amount of energy required for communications.
Either way, be it through their own means or supported by machines, the human ability for transmitting information within HiTLCPS is undeniably important. Despite having discussed this communication ability under Data Acquisition, it remains important through all phases of HiTL control, particularly in Actuation, as discussed in Section 3.4.
3.3 State Inference
3.3.1 Human Nature
Human nature is a mysterious thing, and tremendously difficult for current machines to understand. Our best efforts at understanding what a person wants still reside in simply asking him/her directly. In fact, researchers are still trying to correlate smartphone sensing data features with human behavior, through sampling questions and surveys [11]. How can we hope to create machines that are capable of decoding such elusive but important aspects of existence that are so difficult to understand even for humans themselves?
The combination of sensors through body-area-networks may be able to alleviate this difficulty in gathering human information. This human body-area-network is composed of a variety of sensors (accelerometers, smartshirts, smartshoes, bracelets, watches, etc.) and is capable of measuring several different aspects of human activity, including vital signs (heart rate, ECG, EEG, movement, etc). More interestingly, we are continuously learning how to use this information for characterizing actions, and for detecting activities and even psychological states and emotions. Current research indicates promising leads to new powerful and complex machine learning solutions that are becoming increasingly more cognizant of “human nature” phenomena, making them an integral part of the control loop in IoA scenarios. For example, the attention level of a driver affects the cruise control mechanisms of an automobile, the user's exercise levels affect the air conditioning of his/her house, or a human's emotional state may affect the user interface of his/her smartphone application. Humans are no longer external entities that simply benefit from the system. Their presence, actions, and emotional states strongly affect how IoT things react. We will discuss some of these new research lines in greater detail in Sections 4.1.2 and 4.2.
3.3.2 Humans as Processing Nodes
As we have discussed in Section 3.2.1 (Humans as Sets of Sensors) and 3.2.2 (Humans as Communication Nodes), humans can also directly contribute to the processing of information. No machine has yet been capable of matching human capability for pattern recognition; thus, as previously discussed, human information should not be undervalued.
Still, there are other aspects of processing that are not directly related to human cognition and yet still greatly depend on human behavior. Since single individuals are now becoming equipped with a considerable number of mobile devices (smartphones, tablets, smart wearables, etc.), human behavior begins to have a significant impact on the availability of resources within an HiTLCPS. These resources contribute to the overall computation capability of the system: each of these individual devices represents an untapped computational resource that is available on site; by taking advantage of these devices, it is possible to reduce the need for distant service providers: direct communication with neighboring devices becomes key for handling local tasks and information. Thus, the traditional cloud is descending to the network edge and becoming diffused among the client devices in both mobile and wired networks. This concept has come to be known as the “fog of things” [79].
Although most devices carried by humans are very simple and have limited processing capabilities, the use of distributed algorithms can take advantage of the huge number of processing elements, and enable collaborative tasks that could not be fulfilled by any particular individual node. Smartphones can be major participants in this processing, but other, simpler, wearables and appliances can also become useful processing sub-nodes in the new IoA.
3.4 Actuation
3.4.1 Humans and Robots as Actuators
Humans already act as actuators and as a function of the medium. If a gas leak is detected in a factory, the responsible employee quickly goes to the control room and closes the respective valve. If in a hospital the blood pressure of a patient reaches a prohibited value, the nurse on duty, hearing the alarm signal, goes directly to the patient's room to administer a new drug. Unfortunately, current IoT systems are still mostly unprepared for handling human actuation as an inherent component of the system. In HiTLCPSs, human actions remain extremely important, since human conceptualization will continue to be unmatched by artificial intelligence (AI) for, most likely, many years to come. However, unlike most current CPSs, the IoA paradigm takes human action into consideration in the control-loop, in the sense that these systems are made for humans, with humans. Examples of this human role are industrial systems that may use WSNs and robots to monitor and detect problems, and then require specialized actuation of humans to fix the problem. On our social-networking shopping mall example, users may consider product suggestions from other clients with similar interests and psychological states, and collaboratively suggest products of their own interest.
In HiTLCPSs, human and machine actuation go hand-in-hand and can often complement each other. In this way, IoA systems are not “devoid of human soul” but make human actuation as an integral part of their functioning. As we will see in later chapters, particularly in Part III, the ability to work in combination with actuation machines, such as robots, will become increasingly important in the future.
3.5 In Summary..
In this chapter, we defined a general reference model, the Internet of all, where human actions become a fundamental part of the control loop of CPSs. We then organized the major ideas behind HiTLCPSs, starting with a previously proposed taxonomic classification, based on the type of exerted control.
In this classification, we saw that HiTL control can be classified into three types: (1) applications where humans directly control the system (human control), (2) applications where the system passively monitors humans and takes appropriate actions (human monitoring), and (3) hybrids of (1) and (2).
We then proposed an alternative taxonomic point of view of the HiTL process that highlights the roles a human may play within HiTLCPSs:
- Data Acquisition
- Humans as sets of sensors: Humans can feed the system with information, either collaboratively or through the sensors they carry.
- Humans as communication nodes: Humans are masterful communicators, able to quickly share information through social networks. Their body-area devices can also store and pass data as part of a “hopping” process.
- State Inference
- Human nature: Understanding human nature is tremendously difficult, but a combination of body-area sensors and powerful machine learning solutions may alleviate the problem of recognizing human-centric states.
- Humans as processing nodes: Machines have yet to match humans in their capability for pattern recognition. Additionally, human-carried devices (e.g. smartphones, smartwearables) have considerable amounts of processing power and may diminish the need for cloud-centric solutions in the near future.
- Actuation
- Humans and robots as actuators: There is much to be said about human actuation in HiTLCPSs. Issues such as human motivation, robotic collaboration, and AI will have an important impact on future deployments. As we will see in later chapters, it is likely that human-machine collaboration will play an important role in future technologies.
Now that we have described HiTLCPSs from a theoretical perspective, we will focus on providing practical examples of existing deployments and technologies.