1.1 The Rise of Cyber-Physical Systems

Interestingly, once the Internet was in place, we quickly achieved the power to extend it to our traditional tools and appliances, which then became “interconnected”. One of the first “tools” ever connected to the Internet was the Carnegie Mellon University Computer Science Department's Coke Machine, in the early 1980s [19], which was able to report its stock and label it as “cold” or not, depending on how much time it had been inside the machine. An idea began to spread: a vision of an interconnected world where information on most everyday objects was accessible.

Since then, scientists and engineers have developed this idea into a concept that is known as the “Internet of Things” (IoT). The idea started small, considering scenarios where radio-frequency identification allowed the “tagging” and managing of objects by computers. Each object would carry a radio-frequency identification (RFID) tag, a small, traceable chip which could be wirelessly scanned by a nearby RFID reader. The RFID tag enabled the automatic identification of the object and allowed it to be traced/managed through the Internet.

The continued advances in miniaturization allowed us to go beyond the simple tagging and identification of everyday objects. As predicted by Gordon Moore, back in 1965, the amount of computing power in integrated circuits has been doubling every 18 months for the last 50 years [20]. The remarkable work of computer industry engineers and scientists has led to many new technologies. The continuous integration of computational resources into all kinds of objects made our tools “intelligent”. Everything from light bulbs to refrigerators, microwaves, and coffee machines will soon be connected to the Internet. In fact, some studies estimate that we will have an IoT with 26 billion connected devices by 2020 [21].

We can see evidence of this trend all around us. The Internet now interconnects a large number of highly heterogeneous devices, from traditional desktop PCs to laptops, tablets, and smartphones.

For example, the area of sensing technologies and wireless sensor networks (WSNs) is becoming increasingly prominent. WSNs are composed of dozens or even hundreds of autonomous “sensor nodes”, small computerized devices that are capable of collecting physical world data and forwarding it by means of wireless communication. They can be used to monitor environmental luminosity, temperature, pressure, sound, and many other parameters, and can be spatially distributed in an ad hoc fashion. These technologies have been receiving a great deal of attention from the research community due to their potential in almost every application area. In fact, WSN deployments can now be found in many industrial, medical, and domestic environments. Recent studies in WSNs have brought great advancements in this area, namely in terms of energy efficiency and integration capabilities, with sensors being provided as services [22 23], accessible through the Internet [24]. Sensors are now indispensable devices, for they allow us to collect data from real-world phenomena, handle this data in digital form, and ultimately extend the Internet to the physical world.

In fact, the number of sensors that nowadays can be deployed on humans can turn them into walking sensor networks. Humans can use smart-shirts; carry a smartphone with several sensors and networking capabilities (e.g. global system for mobile communications (GSM), Bluetooth, long-term evolution (LTE)); and use Google glasses, iPods, smart watches, and shoes with sensors. In terms of sensing applied to individual users, Bosch Sensory Swarms and the Qualcomm Swarm Lab at UC Berkeley estimate that the number of sensors in personal devices can add up to 1000 wireless sensors per person, to be deployed over the next 10 to 15 years [25], resulting in large amounts of data being available for processing, and allowing a wide range of sensing applications to be deployed. This reality depends, of course, on the drastic reduction of sensor production costs, which are expected to come down to negligible values over time, as with most silicon-based hardware [26].

As for automated actuation, the world has seen a gradual increase in the number of installed robots per year. The 2015 World Robot Statistics study, issued by the International Federation of Robotics (IFR) [27], indicates that the total number of professional service robots sold in 2014 rose by 11.5% compared to 2013, from 21,712 to 24,207 units. IFR expects that, for the 2015–2018 period, sales of service robots for professional use will increase to about 152,375 units, while sales of robots for personal use will reach about 35 million units, with a total estimated value of about $40 billion. Global sales of industrial robots, on the other hand, will experience a yearly growth of 15% until 2018, while the number of sold units will double to around 400,000.

Interwoven with the concept of IoT is the concept of cyber-physical systems (CPSs), which consist in the sensing and control of physical phenomena through networks of devices that work together to achieve common goals. These CPSs represent a confluence of robotics, wireless sensor networks, mobile computing, and the IoT, to achieve highly monitored, easily controlled, and adaptable environments.

The IoT and CPS concepts have been pushed by two distinct communities. IoT was initially developed using a computer science perspective, mostly supported by the European Commission. The goal was to develop a network of smart objects with self-configuration capabilities on top of the current Internet. This development effort included hardware, software, standards, and interoperable communication protocols and languages that describe these intelligent devices [28]. IoT builds on several requirements, namely the development of intelligence in devices, interfaces and services; the assurance of security and privacy; systems integration; communication interoperability; and data “semantization” and management [29].

On the other hand, the concept of CPSs was initially supported by the US National Science Foundation (NSF). CPSs stem from an engineering perspective and concern the control and monitoring of physical environments and phenomena through sensing and actuation systems consisting of several distributed computing devices [30]. These systems are mostly interdisciplinary, requiring expertise and skills in mathematical abstractions (algorithms, processes) that model physical phenomena, smart devices and services, effective actuation, security and privacy, systems integration, communication, and data processing [31].

Thus, IoT tended to focus more on openness and the networking of intelligent devices, while CPSs were more concerned with applicability, modeling of physical processes, and problem solving, often through closed-looped systems. While their core philosophy and focus were initially different, their many similarities, such as intensive information processing, comprehensive intelligent services, and efficient interconnection and data exchange, have led to both terms being used interchangeably [32] without clearly identified borders [30].

CPSs combine elements from robotics, wireless sensor networks, and mobile computing, among others, to achieve specific goals. From industrial applications that monitor and actuate on several industrial processes, to social applications that aggregate data from various users in order to achieve goals, such as reducing pollution and traffic in metropolitan areas, CPSs can encompass a multitude of domains. For example, improvement of personal health can be achieved through body networks that integrate the user's vital signs and activity levels with environmental information on pollutants or noise to suggest healthier and more pleasant walking routes, restaurants, and leisure activities. CPSs can also be used in transportation, as many modern vehicles feature cruise control systems that maintain the automobile's speed or perform parking maneuvers, not to mention autonomous driving. All these systems combine sensors, actuators, and the computational capabilities of the devices to achieve the desired results. In fact, these sensors and actuators can be used not only in individual objects but also in structures and buildings in order to monitor, for example, their structural health.

While IoT and CPS technologies do exist, current systems are still designed with a specific scientific, industrial, or engineering application in mind. They are, for example, typically responsible for collecting data from sensors and analyzing it for a certain task. This objective-driven approach results in academic or industrial systems that may be effective for their targeted scenarios but are very constrained in applicability and, therefore, narrow in their usability.

Nevertheless, we know from previous experience that this happens during the beginning of any new paradigm-changing technology, as was the case with most information technologies. The most striking example is the Internet itself, which initially only connected the University of California at Los Angeles, the Augmentation Research Center at Stanford Research Institute, the University of California at Santa Barbara, and the University of Utah's Computer Science Department. This and other initial computer networks continued to grow and merge, giving birth to the Internet as we know it today.

Much like the Internet, it is very likely that existing disconnected and restricted CPS deployments, whose primary beneficiaries are privileged users who already benefit from and explore their capabilities, are just the initial steps towards a future where the vast majority of intelligent devices are interconnected in massive, non-centralized networks. In fact, some researchers argue that future CPSs will become ubiquitous and distributed, with many data streams overlaying the network, provided by large amounts of sensors. They also argue that these streams should be open for use, without centrally controlled authorization, through self-advertising and discovery by nearby users. Thus, data acquisition, processing, and visualization should be focused on users, not administrators or scientists [33].

1.2 Humans as Elements of Cyber-Physical Systems

The reduction of production costs of silicon-based hardware [34] continues to fuel this increasingly pervasive technological world, endowing people with the ability to access extremely rich and dynamic pools of data pertaining to their surrounding environment. The epitome of these ideas was first put forward by the renowned computer scientist Mark Weiser, in his famous 1991 article “The computer for the twenty-first century” [35]. Weiser maintained that, as devices became smaller in size, more powerful, and efficient, they would eventually disappear. Technology would become so intrinsic to everyday life that we would no longer perceive it as an isolated concept but as an inherent part of our existence. This idea came to be known as “ubiquitous computing”, and the concept of “calm” technology arose. This concept is a direct antitheses of the stressful use of technology, which is still prevalent. Each time we have to navigate menus, errors, bugs, or unintuitive setups, we become stressed by our computers and appliances. On the other hand, Weiser suggested that the true purpose of computers was to help us in a way similar to intuition. He propounded the view that the ideal computer would be something invisible that could truly understand human nature and interpret people's unconscious actions and desires. Instead of humans adapting to technology and learning how to use it, it would be technology that would adapt to the disposition of human beings.

Weiser also predicted that these “calm” interactions would be informative but not intrusive, not demanding the user's attention, and would make use of human intuitive clues. He was right, since we can see his vision materializing with every passing day. Computers no longer require people to sit in front of them; machines now enter the human environment embedded in all kinds of objects, integrating computing in the course of everyday human activities. In fact, current technology is quickly evolving towards these principles predicted 25 years ago: current mobile devices replaced traditional buttons with much more intuitive touchscreens, and software developers give an ever-increasing importance to usability and non-intrusiveness. The number of portable mobile devices has also grown exponentially, and the number of communication interfaces used by them has also grown. It is not hard to imagine a near future where we get up in the morning and our home also “wakes up” and automatically launches and executes many of the routines corresponding to that particular day of the week, under the control of several computing devices. In fact, as computation evolves, humans will most likely stop “using” computer devices, that is human–computer interaction will no longer require direct user attention and will become intuitive, as if it is second nature. In the words of Weiser, “The more you can do by intuition, the smarter you are; the computer should extend your unconscious.”

It is not sufficient for interconnected and intelligent tools to communicate with each other without any human involvement. Human technology is made by humans, for humans. In order to promote the creation of systems that are useful to the average person, it is necessary to consider efficient and intuitive operation. Therefore, in addition to providing basic functionality, openness, heterogeneity, and integration capabilities, it is equally important to discern how systems or tools can be used within a certain context.

These ideas have been previously explored as context awareness, or context adaptation, for mobile and wireless networking [36] and IoT [37]. Actually, increasing context awareness is a cross-cutting challenge for the design of highly optimized networking systems that support distributed autonomic decision making and reconfigurable aspects [36]. However, current trends on context awareness research encompass a broad definition of context. “Context” can be defined as any information that can be used to characterize an entity, that is a person, place, or object [38]. Thus, several works in the area attempt to assess context [39] and use this information to achieve several goals, such as mobility management [40] or energy efficiency in ubiquitous environments [41]. There are also remarkable proposals for frameworks that manage and distribute this contextual data [42 37].

Yet, outside of the area of e-health, whose primary objective is the monitoring of patients [43], there is still scarce knowledge on the actual effects of this human “context” on the CPS control loop. Indeed, one important element often left out of current CPS research is the human user [33]. Most current CPSs that involve control loops still keep humans as external elements to the control system. This is apparent if we think on the technology that we currently find around us. For example, aircraft pilots decide for themselves when to engage the autopilot or when to take manual control of the plane, and cruise control systems for automobiles simply maintain the desired speed without taking the driver's behavior into consideration.

Systems that consider the human context will become increasingly more important, and most future technologies will be human-aware. Future CPSs will most likely bolster a much stronger tie between humans and control loops. This involves using a large variety of sensors and mobile devices to monitor and evaluate human nature, making humans an integral part of the CPS.

We are now in the realm of human-in-the-loop cyber-physical systems (HiTLCPSs), that is cyber-physical systems that take human response into consideration. Human presence and behavior are no longer seen as external and unknown factors but have become a key part of the system instead. HiTLCPSs infer the user's intentions, psychological states, emotions, and actions through sensors, integrating this information into the control loop as feedback to determine the actions of the CPS. By considering humans an integral part of the system, the control loop's performance and accuracy can be vastly improved. For example, cruise control HiTLCPS systems will be able take the driver's psychological state into consideration (e.g. fatigue, attention-levels, etc.) in order to generate alarms and suggest the activation of cruise control [13]. In fact, previous research has already proposed image-based processing of facial expressions to detect irritation in drivers, and use this information to improve driving safety [44]. HiTLCPSs also include brain–computer interfaces, controlled assistive robots, intelligent prostheses, and monitoring systems, among others [16]. In order to improve the accuracy and timeliness of the system by considering the human element, it is essential to develop and integrate reliable and accurate human behavior modeling techniques that attempt to learn and predict human behavior. Since human behavior is quite hard to predict, making humans part of CPSs is a colossal challenge, as it requires modeling of complex behavioral, psychological, and physiological aspects of human nature. Nevertheless, a multitude of variables regarding a person's state may be measured, including movement, vital signs, and attention level, among many other things, which may be crucial to control the task at hand.

The maturing of HiTLCPS's design has yet to be achieved. For the most part, we have not reached a consensus or even a general understanding regarding the underlying requirements, principles, and theory. This drives us to ask questions, such as why do current IoT solutions still leave behind the human factor? Why have we yet to integrate the human component into CPSs? What challenges do we still face in order to achieve true HiTLCPS deployments? How can we take advantage of new ubiquitous sensing platforms such as smartphones and personal devices used massively by people throughout the day?

In the paper written by Stankovic et al. [45] three main challenges were identified for HiTLCPSs.

• First, there is a need for a comprehensive understanding of the spectrum of HiTL applications, which requires a study of existing, emerging, and potential solutions so that common underlying principles, requirements, and models may be found. As HiTLCPSs have a wide spectrum of applicability, it is necessary to amass examples of HiTL solutions from multiple domains before such an understanding may be achieved [13].
• Second, it is necessary to improve the techniques that derive models of human psychological states, emotions, physiological responses, and actions. In other words, we need reliable mechanisms for modeling, detecting, and possibly predicting human behavior, such as advanced mathematical models or machine learning techniques. Current state-of-the-art techniques are either very coarse and general or too application-specific. The development of dynamic human behavior models that are both accurate and general enough remains an enormous challenge.
• Finally, human behavior models need to be incorporated into the formal methodology of feedback control, either outside or inside the loop, within the system model or at any other hierarchical control level.

As a consequence of our research activity, we have come to believe that current research is close to providing answers to many of these obstacles, and that HiTL concepts will become increasingly more common. Despite being in its infancy, we have found promising research that indicates we may be reaching a tipping point in our technological evolution. More than having intelligent IoT and CPS systems that autonomously control our environment, these systems will, more importantly, adapt to the human context and needs. In fact, with HiTLCPSs we may be on the verge of achieving an unprecedented control over our environment, one that we could only conceive of in our wildest dreams.

1.3 Objectives and Structure

The next few years will most likely converge into not an Internet of things, but an “Internet of all” (IoA): an Internet that includes the emotions, psychological states, actions, and drives of the ordinary user, the human user, as part of larger-scale systems.

In this context, this book has two main objectives. First, it is intended to be a primer on HiTLCPSs, providing some insights into the research being done on this topic, current challenges, and requirements. As such, throughout the book we will lead the reader on a journey through this new and exciting area of research and technological development. The book's second objective is to initiate the reader in the development of HiTLCPS systems, using a hands-on approach. We will guide the reader through a comprehensive tutorial where the major theoretical concepts behind HiTLCPS are applied to a sample application and explained from a practical perspective. To cope with these objectives, this book is divided into three major parts.

Part I provides an overview of HiTLCPSs, encompassing their evolution, theory, technologies, and some applications. Chapter 2 presents the evolution of these systems, beginning with the scope of simple “things”. From there, we evolve to whole environments and, finally, we consider the monitoring of human beings. HiTLCPSs have a wide spectrum of applicability. As such, in an attempt to cover as many solutions and domains as possible, Chapter 3 presents a general taxonomy of human roles within HiTLCPS. Concluding Part I, Chapter 4 presents several pieces of research work and several technologies that can be or have been used in HiTLCPS deployments. The purpose of this chapter is to provide the reader with an idea of current, real-world HiTL implementations.

Part II addresses our second objective, that is to provide a hands-on tutorial that will consolidate the previously presented theoretical concepts. To do so, we guide the reader through the creation of a simple, smartphone-based HiTLCPS. In Chapter 5, we present a sample Android application that we will endow with HiTL control. The objective is to begin with a bare-bones map application to a system capable of rough estimations of the user's current mood, providing suggestions to improve their physical and mental well-being. After explaining how to set up the necessary development environment in Chapter 6, we delve into actual HiTL development, from Chapters 7 through 9.

Part III discusses topics that will affect future and emerging HiTLCPS applications, providing the reader with pointers on several aspects that must be taken into consideration when implementing HiTLCPSs. As such, we first discuss existing requirements and challenges for HiTL applications in Chapter 10. Subsequently, we conclude our book in Chapter 11, with some remarks and conclusions on the covered material, identifying the main technical and ethical limitations that may be expected in future endeavors.