Object Fabrication and Camera Capabilities

As a most basic question, we might begin by asking: is it possible for the camera to capture the adversarial aspects of a printed object sufficiently in its digital rendition? To establish the answer to this question, a first step would be to print the adversarial perturbed image and see whether a camera can feasibly extract the perturbation and generate a digital representation that is rendered adversarial. This basic experiment is depicted in Figure 8-1. Notice that at this stage we’re not worrying about complexities such as camera angle or lighting; we’re simply interested in whether the adversarial information can be successfully transferred via printing and then a camera sensor.

Figure 8-1. Highly constrained physical-world adversarial attack using a printed image

Although this is a fairly simple experiment, it’s an important first step and has been successfully proven to be possible.¹ So, yes, adversarial examples generated digitally can still remain adversarial after passing through the extra printing–camera step. There are several considerations, however, including the following:

Object fabrication (2D or 3D printing)

The attacker must have the tools to print the perturbation or patch at the level of detail required. This could present a problem as the range of colors that a printer is capable of producing (its color gamut) is a subset of all the possible colors that can be represented by RGB digital data rendition. If the adversarial perturbation or patch is captured in RGB values that cannot be reproduced by the printer, the attack will not be successful. Similarly, the printer must be capable of printing to the pixel granularity required for the adversarial attack.

The colors also require reliable reproduction for adversarial perturbation to work effectively. This is surprisingly difficult due to inconsistencies and inaccuracies of image color reproduction by printers. One approach to solving this challenge is to map possible RGB pixel values to their actual printed color. It’s then possible to quantify a “printing error” over a complete picture by taking the difference between the correct and actual values. A requirement to minimize the printing error is then incorporated as an additional constraint within the adversarial cost function, with the result that the adversarial perturbation produced is optimized to use colors that are more accurately rendered by the printer.² In practice, this approach is not particularly robust as printing errors are inconsistent, varying not only between printers and but also between printouts from a single device. A more effective approach is to create the adversarial example that places less reliance on any specific colors, ensuring robustness of the adversarial example to color inaccuracies. This method is discussed further in the following section,³ as it also caters to other problems such as the effect of lighting.

Camera capabilities

The precision with which the adversarial information is captured will always be limited by the sensitivity and accuracy attainable by the camera. Perturbation encoded within single pixels on a printed image, for example, will be lost unless the camera is capable of capturing that pixel precision at the distance at which it is placed from the object.

Data precision and the treatment of image noise and distortion are a consideration in the preprocessing chain for both physical-world and digital attacks, so we’ll consider these challenges in further detail in “Preprocessing in the Broader Processing Chain”.

Viewing Angles and Environment

Now let’s up the game and consider a less constrained and more realistic environment, perhaps with the goal of adding some adversarial perturbation to a road sign in the hope that it’s misclassified, as illustrated in Figure 8-2. This time the adversary decides that they want to go further and alter or create an object in the real 3D world, where camera angles and conditions are no longer guaranteed.

Figure 8-2. Unconstrained physical-world adversarial attack

The first obvious challenge with adversarial objects in the physical world is that they, or the camera capturing them, may move. For example, the images in Figure 8-3 were all taken within a few minutes of each other, at different camera angles. Notice the significant changes in angles, lighting, and exposure. If adversarial perturbation was added to the sign using the methods described so far, it would fail to transfer between these images. Initially, this challenge of transferability was assumed to indicate that creating robust physical-world examples would not be feasible.⁴

Figure 8-3. The effect of camera angles and light on objects in the physical world

You’ll notice a similarity to the problem facing the adversary in generating image-agnostic reusable adversarial perturbation, as discussed in “Reusable Patches and Reusable Perturbation”. The adversarial change must be sufficiently flexible to work across different images. Broadly speaking, the challenges encompass:

Viewing angles and distances

Let’s consider the relative camera position and its settings first, and the many potential resulting transformations to the image. Considerations here include:

Zoom

The image may be magnified (due to varying relative distance between the camera and the target object or as a result of camera magnification). As a result, the object may occupy a large or smaller area of the overall camera view.

Translation and rotation

The object may move with respect to the camera, or the camera may move with respect to the object. The result is that the object moves within the camera frame and the angle at which it is viewed varies.

Skew (or shear)

The position of the camera with respect to the object and the camera’s focal length may result in image distortion.

Lighting and environment

For an algorithm to be robust across viewpoints, it must also consider the position of the object in relation to any light sources, other objects, and environmental conditions.

The angle of the light combined with the object’s colors and textures will affect how the image is captured by the camera. Even the contrast between stark sunlight and a cloudy day can have an immense effect on the clarity and colors in an image captured on camera.

Objects scatter light differently. The way that the light bounces of an object’s surface depends on the material of the surface—its texture, color, diffusion, and reflectance. On nonreflective surfaces, light rays are diffused and bounce off in different directions, giving a more “matte” appearance. Conversely, on reflective surfaces (such as metals), light rays bounce off at the opposite angle to that at which they hit the object. This mirror-like behavior is known as specular reflection.

Light will also be affected by the position of other objects and environmental conditions. The adversarial object may become partially hidden or its reflective properties may change (for example, due to rain).

Photographic capture also incurs the risk of introducing noise. Photographs captured in low light may be susceptible to unwanted noise across the image—random variation of color or brightness—manifesting as “specks” in the image. Software in the sensor or data processing steps prior to the neural network processing stage may remove noise introduced during the data collection stage. However, this is unlikely to be helpful to the adversary as noise cleaning steps will not reintroduce subtle adversarial perturbation that was originally masked by the noise.

Attack constraints

Finally, the placement of the adversarial perturbation or patch might be constrained by what the attacker is able to change in the physical environment. Depending on the scenario, the adversarial change may be required on a specific object that already exists, or alternatively may be fabricated in a new object that is placed in the environment. Although the aim may not be to entirely hide the perturbation, the level to which any change can be made while still appearing nonadversarial to onlookers will be a key consideration of the attacker. The adversary may consider camouflaging the attack, for example, as a logo. Alterations to specific areas, colors, or textures may be restricted, and there may be restrictions as to what is physically accessible to change.

With all these challenges, you might assume it impossible to create adversarial objects in the real world. However, in 2018, researchers proved that it is possible to create adversarial examples that are robust over a realistic distribution of viewing and lighting conditions.⁵ To achieve this, they used a combination of techniques. Firstly, they exploited wire-mesh 3D images rather than 2D images. These 3D models are wire-frame descriptions of a 3D object that can be digitally manipulated and rotated in 3D space. They also have associated colors and textures, so it’s possible to digitally synthesize the effects of varying lighting conditions on the object.

Armed with a far richer representation of objects in the real world, the researchers then created functions that could be applied to the object to simulate how it might be transformed. This included rendering from 3D to 2D, lighting, rotation, and translation. The cost function used to calculate adversarial perturbation was then updated to consider these transformations. Using white box methods, rather than establishing a cost gradient for the target prediction with respect to changes in a single 2D image, changes were considered over a distribution of transformations of 3D textured models. This technique is referred to by the authors as Expectation over Transformation (EOT).

The images in Figure 8-4 are taken directly from the paper by Athalye et al. and depict the results of photographing and classifying a 3D-printed adversarial turtle.

Figure 8-4. 3D-printed turtles misclassified as rifles

It’s worth noting that this research also has significance to digital adversarial examples. Creating transformation-resilient adversarial perturbations using 3D modeling and then rendering to 2D would be an effective approach to creating adversarial examples that are more robust to preprocessing and defenses (which we’ll discuss in Chapter 10).

As discussed in Chapter 1, Eykholt et al.⁶ use a similar approach of incorporating transformations into a 2D image to cause a DNN to misinterpret its meaning (for example, misinterpreting a stop sign as a speed limit sign thanks to the use of adversarial stickers, as shown in Figure 8-5). The researchers also go a step further in capturing physical-world changes to the road sign on camera and including these in the adversarial calculation.

The two examples presented (the 3D-printed turtles and the stickers on the stop sign) highlight the differing constraints placed on the attacker when introducing an adversarial perturbation or patch into the environment. The first approach introduces a complete object over which the adversary has complete control. There is flexibility in where adversarial change is added to this object and the shape of the object itself. Of course, an attacker will aim to create an object that appears benign within the environment in which it will be placed, as described in the hypothetical example presented in the following sidebar.

Figure 8-5. Physical perturbation applied to a stop sign (image from Eykholt et al. 2018)

With the stop sign example, alterations were restricted to the existing sign itself. To generate adversarial alterations that appear innocuous, the researchers limited the perturbation calculation to only some parts of the image, making them more likely to be overlooked by a human observer as dirt or graffiti.

Audio Reproduction and Microphone Capabilities

As with image, let’s begin by considering the simplest possible scenario: a microphone within close proximity of the sound coming from a single speaker (Figure 8-6). We’ll assume a “perfect” sound environment with no other noises.

Figure 8-6. Physical-world adversarial attack using sound

Limitations to be aware of include:

Reproduction of digital audio (speakers)

As with printing adversarial objects, the most basic requirement is to reproduce the digital audio as sound waves at the precision required for adversarial distortion to be effective. This will depend on the precision and capabilities of the speakers.

An interesting aspect of this threat is that, in many scenarios, the attacker does not control the speakers used to render the digital audio to sound. The quality of the sound of an audio adversarial attack shared online, for example, will be at the mercy of the speakers that the playing device is using. This could vary from high-quality speakers for playing audio in a room to low-quality speakers on a basic smartphone.

Speakers translate digital audio to sound waves through vibration of some flexible material (the speaker membrane). In the same way that it is not possible for printers to produce every possible digital RGB combination, the pitch of the sounds that can be produced by a speaker is restricted by the speed at which the speaker membrane can physically move. Digital adversarial audio that requires pitches outside this limit will not be reproduced.

Microphone capabilities

Data capture through the microphone incurs the risk of introducing noise or distortion. Similar to the “specks” introduced when an image is captured, audio may include “static” introduced during the initial rendering of the sound to data by unwanted electronic fluctuations. The microphone or preprocessing chain is also very likely to perform preprocessing (such as MFCC—see “Audio”) to extract the relevant features and remove data that is not pertinent to the task.

It has been proven possible to reproduce unrecognizable adversarial commands over the air and successfully capture them using a smartphone,⁷ so this basic ability to generate adversarial distortion and capture it digitally is feasible.

Audio Positioning and Environment

Now let’s consider the additional complications brought about by the environment and relative positioning of the speaker/audio:

Environment

A fundamental challenge to launching a physical adversarial audio attack is that adversarial sounds will need to compete with other sounds in the environment. In many threat scenarios, the attacker will have no control over the amount of other noise in the environment; however, there are some cases where they may have some control, or at least some prior knowledge. Consider, for example, generating voice commands intended to be adversarial to a voice control system incorporated in a car. The attacker may not have knowledge of all the possible competing sounds, but would be able to consider the relatively predictable impact of the car’s engine noise during the generation of the attack.

We are all aware of how sound changes in timbre and character in different environments. There’s significant difference between sounds when heard outside, for example, versus when they are heard inside. Inside, waves reflect and reverberate off the walls and indoor surfaces. Objects such as furnishings and room content potentially dampen and spread sound waves. The sound will have been reflected, resonated, and attenuated by nearby objects prior to reaching the microphone. Clearly, the position of the sound source relative to the target device microphone will also have a massive effect on the sound and its quality. Creating adversarial sound that is robust to different environments and speaker/microphone positioning will be challenging, but the attacker may focus on a subset of scenarios (such as a small room) or, if creating the sound in a public space, may even control the speaker and its positioning.

Feedback and confirmation

Digital assistant technology usually supports some form of audio or visual feedback as an assurance of user intent—especially when commands have greater security risk (such as an online purchase). The assistant, for example, might light up or might generate a voice response to request verification. While the actual verification could be included within the adversarial command (such as a carefully timed, inaudible “yes”), the attacker has the problem of ensuring that the audio feedback (such as “are you sure you wish to go ahead with that evil plan?”) is not overheard by a bystander.

If the attacker has constructed the adversarial distortion in audio over which they have complete control (music in a video shared online, for example), they could simply increase the volume of the benign sound at the expected point in time of the digital assistant’s response. More likely, it may not actually matter if the attack does not always work; sometimes it’s thwarted because someone happens to hear it, but occasionally when no one is around or noticing, it might succeed.

Constraints

As with adversarial objects, the success of an adversarial sound attack may depend on the extent to which the attacker can make changes to the audio while the distortion remains undetected. It may be more difficult for the attacker to hide adversarial commands within existing audio than in audio that they control. For example, if the attacker adds audio distortion in some music or speech that they also create, they have far greater control over the audio that masks the attack. The other question is: does the adversarial sound need to be completely hidden? So long as people ignore or overlook the sound as unrecognizable, it may not need to be disguised as normal music or speech.

Attacks with inaudible voice commands are possible even in more challenging environmental conditions. A likely approach would be to repurpose the EOT method for generating adversarial 3D models (as introduced in “Adversarial Objects”) in the audio domain, considering audio transformations and restrictions of speaker reproduction of sound.