Past and Present

OpenCV was launched in August 1999 at the Computer Vision and Pattern Recognition conference (and so turns 17 years old at the publication of this book). Gary Bradski founded OpenCV at Intel with the intention to accelerate both the research and use of real applications of computer vision in society. Few things in life go according to their original plan, but OpenCV is one of those few. As of this writing, OpenCV has nearly 3,000 functions, has had 14 million downloads, is trending well above 200,000 downloads per month, and is used daily in millions of cell phones, recognizing bar codes, stitching panoramas together, and improving images through computational photography. OpenCV is at work in robotics systems—picking lettuce, recognizing items on conveyor belts, helping self-driving cars see, flying quad-rotors, doing tracking and mapping in virtual and augmented reality systems, helping unload trucks and pallets in distribution centers, and more—and is built into the Robotics Operating System (ROS). It is used in applications that promote mine safety, prevent swimming pool drownings, process Google Maps and streetview imagery, and implement Google X robotics, to name a few examples.

Since the previous edition of this book, OpenCV has been re-architected from C to modern, modular C++ compatible with STL and Boost. The library has been brought up to modern software development standards with distributed development on Git, continuous build bots, Google unit tests, comprehensive documentation, and tutorials. OpenCV was intended to be cross-platform from the beginning when it spanned Windows, Linux, and Mac OS X. It continues active support for these desktop OSes, but now also covers mobile with Android and iOS versions. It has optimized versions for Intel architectures, ARM, GPU, NVidia GPUs, and Movidius chips but also works with Xylinx Zync FPGAs. In addition to efficient C++ source code, it has extensive interfaces in Python (compatible with NumPy), Java, and MATLAB.

OpenCV has also added a new, independent section maintained by users. In that repository,  all routines are standalone and follow OpenCV style and documentation, as well as pass the Buildbot tests. With opencv_contrib, OpenCV keeps up with the latest algorithms and applications in computer vision; see Appendix B for a snapshot of the directory’s contents.

How Well Did Our Predictions Go Last Time?

In the previous edition of this book, we made some predictions about OpenCV’s future. How did we do? We said that OpenCV would support robotics and 3D; this clearly came true. One of the authors, Gary, launched a robotics company, Industrial Perception Inc., that used OpenCV and 3D vision routines to allow robots to handle boxes in distribution centers. Google bought that company in 2013. At the same time, the other author, Adrian, ran many industry, government, and military robotics contract projects incorporating OpenCV while he was working at Applied Minds.

Calibration was forecast to be expanded and to include passive and active sensing. True to form, OpenCV now includes ArUco augmented reality markers and the combination of checkerboard and ArUco patterns so you no longer need to see the whole board, and multiple cameras can see different pieces of the same calibration pattern (see Appendix C). All these routines now exist to solve more challenging calibration and multicamera pose problems.

We predicted new 3D object recognition and pose recognition capabilities, and these were also integrated—from human-defined features in linemod to deep network 3D object recognition and pose. Indeed, opencv_contrib was itself predicted as a modular repository that would make user contribution much easier.

Most of the applications predicted in the previous book, from much better stereo vision algorithms to dense optical flow, have come true. Back then, we said that 2D features would be expanded and supported by an engine, all of which happened in features2D(), which covers a large percentage of the hand-crafted 2D point detectors and descriptors. Improved functionality with Google data structures is also under way. We also said that better support for approximate nearest neighbor techniques would be added, and it was with the incorporation of FLANN (Fast Library for Approximate Nearest Neighbor) into OpenCV. We have long since run developer workshops at computer vision conferences as outlined in the previous book. Finally, better documentation did finally show up (http://docs.opencv.org).

What were we wrong about? We did not yet get a more general camera interface for higher-bit or multispectral cameras. SLAM (Simultaneous Localization And Mapping) support is in, but not as a robust complete implementation. Bayesian networks were not pursued because deep networks outpaced them. We did not yet implement anything special for artists, but artists nevertheless have continued to expand the use of OpenCV.

Some AI Speculation

We are clearly at a turning point in the development of artificial intelligence (AI). As of this writing, AlphaGo from Google’s Deep Mind group has beat the world champion, Lee Sedol, at the very difficult “spatial strategy” game Go. Using AI, robots are learning to drive, fly, walk, and manipulate objects. Meanwhile, AI technology is making speech, sound, music, and image recognition natural in our devices and across the Web. Silicon Valley has seen many “gold rushes” since the original one for real gold in 1848. The winners and losers in this new AI gold rush remain to be seen, but it is clear that the world will never be the same. In its function of accelerating progress in perception, OpenCV plays a role in this historical movement toward sentient (self-aware) machines.

It’s clear that deep neural networks have essentially solved the problem of feed-forward recognition of patterns (one can say that they are superb function approximators), but such networks are nowhere near sentient or “alive.” First, there is the problem of experience itself. We humans don’t just see, say, a color; we experience it subjectively. How this subjective experience arises is called the problem of “qualia.” Second, machines also don’t seem to ever really be autonomously creative. They can generate new things within an explicit domain, but they don’t invent new domains, nor actively drive experimentation and open-loop discovery.

What may be missing is “embodiment.” Humans and many robots have a model of their own being, their “self,” acting in the world. This self can be simulated in isolation for planning actions, but this simulated model of self is more often coupled to the world by sensors. Using this coupled model, the embodied mind gives causal meaning to the world (choosing where to walk, avoiding danger, observing consequences to its plan), and this gives the embodied mind a sense of meaning in relation to its model of itself.

We believe that such a world-coupled model of itself allows the AI to make metaphors [Lakoff08] that are used to generalize to later experience. When young, for example, humans experience putting things into and taking things out of containers. Later in life, this embodied experience informs what it means to be, say, “in” a garden; in other words, the early experience of playing with containers is used to generalize what it means to be in a garden. Causal experience of the model of self, coupled to the world, allows an entity to attach meaning to things. Such meaning stabilizes perception since categories don’t just come and go—they have causal and time-stable consequences to our simulated model of self within the world. It is complete speculation, but qualia, or subjective experience, may arise from simulating how our model of the world affects our simulated model of the self; that is, we experience our model’s simulated reaction to the world, not the world itself.

Stanley, the robot that won the $2 million prize for the 2005 DARPA Grand Challenge robot race across the desert, used many sensors such as GPS, accelerometers, gyros, laser range finders, and vision to sense the world and fused these sensed results into a computationally efficient “bird’s-eye view” world model. The model consisted of a tilted plane reflecting the general angle of the terrain that was then marked with drivable, un-drivable, and unknown regions derived from the sensor readings. In this world model, Stanley ran physics simulations of itself driving in the general direction of the next few GPS waypoints. The resulting paths were rated to find the most efficient path that would not tip the robot over. Stanley’s brain was sufficient to win the DARPA Grand Challenge, but consider what it wasn’t sufficient to do: it could not represent love, politics, astrophysics, or Shakespeare very well. If Stanley could ask us what a Shakespeare play meant, at best we could say it was something like the boundaries between the drivable and unknown areas in a difficult map. Stanley’s model of itself and its interaction with the world are too sparse for understanding most of the things in the world. It seems obvious that we ape-like beings that live mostly in low-lying temperate watersheds are similarly limited in our ultimate ability to even detect what we don’t know about the universe. In this way, “we are all Stanley.”

We humans find it pretty easy to understand something such as the need for food (a natural part of our model), but we find it extremely difficult to figure out how to create a more intelligent AI or to fathom what qualia is. As another example, if we raised a kitten and had it listen to Shakespeare all day until it was grown, we wouldn’t expect our grown cat to understand a sonnet. If we want to explain to the cat what a sonnet means, the best we could do is use a metaphor from the cat’s natural models, such as, “Shakespeare’s sonnet is like a kitten that inevitably gets lost in bad places.” The cat might think, “Now I understand,” but it has no means by which to even understand what it does not understand! Again, we humans must also be similarly limited. Perhaps we can build more powerful machines to which the problem of qualia is simple. But when we ask the machine to explain it to us, it might get flustered and then finally say, “Qualia is like a kitten that inevitably gets lost in bad places.”

In Stanley the robot, the nature of its perception is entirely in terms of its model. Stanley doesn’t perceive the world; instead, its cameras and sensors transduce signals that populate a causal model of the robot in the world. But the model is only like the real world in terms of the navigational needs of the robot car. By way of another example: in a laptop computer, you might see a GUI and conclude, for example, that there’s a trash can inside the computer since you see one on the screen. A more clever physicist might look closely at the screen and cry out, “Everything is made up of quanta” (pixels)! But, in fact, the GUI is only a causal model to a linear Von Neumann machine reality inside. What is real is that things dragged into the trash are erased—the causal consequences of the model are real. Again, we humans must be similarly limited in what we can know of our own universe, since we’ve inherited a causal model mainly directed at our direct physical and social experience. But our machines may see further.

Today, there is a lot of debate about the dangers of AI. People confuse their metaphors around this. They think the “AI,” the intelligence, is what drives the behaviors of the larger system. But look to ourselves. Our “programming language” as humans isn’t our intellect, but our moods and drives—our emotions! Stanley’s goal was to safely traverse GPS points as fast as possible in the correct order. It found an orderly following of GPS waypoints to be attractive and so that’s what it employed its intelligence to do. Our programming isn’t “thought,” it’s emotion. The emotions guide “what” to do; the intellect guides “how.” The same will be true of our future machines. Design the motives well, and the machines will pursue them.

Ah, but the reader may worry that perhaps those machines will alter the goals given to the next generation of machines? First of all, it will be no easier for a greater machine intelligence to understand and create the minds of a yet greater machine than it is for us to create the first generation. The problem just shifts upward. The intelligent machines also will face the same dangers to themselves from their next evolution of AI that we do from the first evolution, and they will tend to program goals and emotions accordingly. In the end, AIs will be consumed with their own goals, which have evolved from our goals as embodied beings. The danger from AIs is not from the possible malevolence of their goals, but from the nonhuman differences in their goals, which to us may seem like indifference or even hostility. This is what made H. P. Lovecraft’s alien monsters so interesting and frightening in his fiction—they were not so much malevolent as driven by wholly different motives and so wholly indifferent to our fate. We, for example, give little thought to the lives of ants and thereby sometimes bring them to harm.

However, and for the record, the authors don’t fear AI, but rather see it as absolutely essential to solve many of humanity’s vexing problems, such as providing reliable health care for all people, ending hunger, curing diseases, providing and maintaining new energy generation and storage techniques while protecting the environment, and helping run our ever-more-complex world. Rather than a threat, in AI we see purpose. We would not want the spark of self-aware intelligent life to die out with our world, but instead would rather see intelligence grow outward in space and in time. This, in a way, may be (or could be chosen to be¹) humanity’s ultimate purpose, and so is also, hereby, an indirect purpose of OpenCV!

Afterword

We’ve covered a lot of theory and practice in this book, and we’ve described some of the plans for what comes next. Of course, as we’re developing the software, the hardware is also changing. Cameras are now increasingly cheaper and more capable and have proliferated from cell phones to traffic lights and into factory and home monitoring. A group of manufacturers are aiming to develop cell phone projectors—perfect for robots, because most cell phones are lightweight, low-energy devices whose circuits already include an embedded camera. This opens the way for close-range portable structured light and thereby accurate detailed depth maps, which are just what we need, together with the development of light field cameras for robot manipulation and 3D object scanning.

Both authors participated in creating the vision system for Stanley, Stanford’s robot racer that won the 2005 DARPA Grand Challenge. In that effort, a vision system coupled with a laser range scanner worked flawlessly for the seven-hour desert road race [Dahlkamp06]. For us, this drove home the power of combining vision with other perception systems: we converted the previously unsolved problem of reliable road perception into a solvable engineering challenge by merging vision with other forms of perception. It is our hope that—by making vision easier to use and more accessible through this book—others can add vision to their own problem-solving tool kits and thus find new ways to solve important problems. That is, with commodity camera hardware, cheap embedded processors, and OpenCV, people can start solving real problems, such as using stereo vision as an automobile backup safety system (or to make automotive improvements in general), monitoring all rail lines for people and vehicles on the tracks, implementing swimming safety measures, building new game controls, developing new security systems, and so on. Be sure to keep an eye on tiny-dnn, a fully featured deep net library with a focus on embedded computing in opencv_contrib. Finally: get hacking!

Computer vision has a rich future ahead, and it seems likely to be one of the key enabling technologies for the 21st century. OpenCV seems likely to be (at least in part) one of the key enabling technologies for computer vision. Endless opportunities for creativity and profound contribution lie ahead. We hope that this book encourages, excites, and enables all who are interested in joining the vibrant computer vision community!