10. Creating Layer and Channel Masks

Introduction

When you work on documents in Adobe Photoshop, the application does its best to make the experience as similar to the real world as possible. When you select the Brush tool and drag within the document, you expect to see a swath of color using the preselected size and brush tip. That’s what you expect to see when you drag a brush on a canvas, and that’s what you see in Photoshop. Creating a realistic experience is what Photoshop is all about. Yet, as realistic as the Photoshop experience is, there are elements of digital design that go way beyond the real world. For example, Photoshop has an undo button; the real world does not. Layer masks, for example, give you the ability to remove elements of a layer without actually erasing the image pixels. Layer masks give you ultimate control over your Photoshop design by deciding what elements of an image are visible, and making changes to the image without destroying any pixels. In addition, layer masks are editable, which means you can change your mind at any time during the creative process.

Imagine creating a complicated selection in Photoshop. Selections are temporary; they last only as long as your document is open. But what if you want to save this selection for use later in the design? What you need is a channel mask. Channel masks hold simple, or complicated selections, and can be saved with the document. Channel masks are created from preexisting selections, or can be created from scratch by painting the mask with black, white, or shades of gray. The process of creating a Channel mask is simple, but the results are powerful.

Understanding the Role of Layer Masks

Layer masks are not new; however, Photoshop designers find new ways to use them every day. A layer mask is an attachment to a layer that defines the visible elements of the layer. Each layer type in Photoshop, with the exception of the Background, has the ability to hold a layer mask. Imagine a layer mask as a piece of paper on top of the image. Then take a trimming blade and cut holes in the paper. The holes in the paper represent the visible area of the image underneath the mask, while the rest of the paper (layer mask) hides the rest of the image. Each layer in a multi-layered document can have its own mask; the mask only influences the image elements of the layer to which it’s attached. Once a mask is created, it can be modified using any of Photoshop’s painting or drawing tools. Black represents transparent (invisible) elements, and white represents visible elements. Painting with a shade of gray introduces a variable level of transparency. For example, painting with 50% gray makes the image pixels appear 50% transparent.

Creating a Layer Mask

The creation of a layer mask requires two things: an open document and any type of Photoshop layer with the exception of the Background. When you add a layer mask, you will be working with two layer elements—the image and the mask. It’s important to know which element you’re working on or you might wind up painting on the image instead of the mask. When that occurs, it’s good to remember the Undo button. Once the mask is created, you can selectively control, without erasing, the visible portions of the image.

Create a Layer Mask

Working with the Paint Tools

Unless a mask is created using a selection, layer masks begin their lives as completely white. The reason a newly created layer mask does not appear to have any visible effect on the image is that the color white indicates visible areas of the image. It’s only when you begin painting on the mask that the image changes. You create transparency in the image by painting in shades of gray. The deeper the shade of gray, the more transparent the image. Painting the mask with pure black produces 100% image transparency. Any of Photoshop’s drawing or painting tools can be used to create a mask. In fact, you could even use the Custom Shape drawing tools and create a mask in the shape of a rubber duck. The trick is to use the right tools to create the right effect. For example, using a hard-edged brush creates an image with sharp edges and using a soft-edged brush creates an image where the visible edges of the image blend smoothly with transparent elements.

Work with the Paint Tools

Using Selections to Generate Complex Layer Masks

Layer masks are easy to create; you select a layer and then click the Add Layer Mask button. Unfortunately, when you create a mask this way, it’s up to you to define the transparent areas, using drawing or painting tools. There is another way to generate a mask, and that’s by making a selection first. When you click the Add Layer Mask button, Photoshop searches the document for any selected areas. If it doesn’t find any, it creates a blank (all white) mask. However, if you first select an area of the image, Photoshop interprets the selection as the area you want to remain visible.

Use Selections to Generate Masks

Using Layer Masks to Generate Soft Transparency

When you create a layer mask using Photoshop’s selection tools, the edges of the image where transparency occurs appear as if cut out with a knife. This occurs because selection tools create hard-edged selections, and then, when you create the layer mask, the edges appear as ragged as the mask. To soften the edge of the selection boundary, you can use the Feather option in the Refine Edge dialog box, which can also be accessed from the Masks panel by clicking on the Mask Edge button (New!). As you adjust the Feather value, the feather effect appears in the Document window with a preview of the image. When you create the layer mask, the edges appear soft and feathered.

Use Layer Masks for Transparency

Blending Images with Layer Masks

You can use layer masks to make areas of an image appear transparent by painting with black. Then just as quickly, you can make those areas reappear by painting with white. There are obvious applications to the use of layer masks—changing a sky, removing a tree, or even removing a person from the photo. For example, you have a line art image and you want to change it to a normal photograph. To accomplish this, you’ll need a copy of the image in a separate layer, a layer mask, and the linear gradient tool.

Blend Images with Layer Masks

Using a Layer Mask to Create a Vignette

When you open a document in Photoshop, the image is displayed in a standard bounding-box format—a square or rectangle with 90-degree corners. Bounding boxes serve a purpose, and many times a nice square or rectangle box is exactly what you want. But let’s face it, bounding boxes can be a bit boring, especially when you want to spice up that image with a nice soft vignette. To create a soft vignette, you need four things—an image, a selection, a layer mask, and the Gaussian Blur filter.

Create a Vignette

Creating Unique Layer Mask Borders

Layer masks can create more than simple vignettes around an image. In fact, with the right filters you can create some very interesting and fun-looking borders. For example, when you create a selection using one of Photoshop’s standard selection tools—rectangle, ellipse, or lasso—the selection has a sharp, definable border. The secret to creating unique borders is to create a general selection around a portion of the image you want to preserve, create the layer mask, and then use some of Photoshop’s creative filter effects, such as the Artistic or Distort filters on the mask. Using filters on the image mask creates eye-catching borders and it’s only a filter away.

Create a Layer Mask Border

Understanding Channel Masks

The Channels panel serves three purposes—to hold color information, to hold spot color information, and to hold selections (channel masks). Creating channel masks can be as easy as clicking the Create New Channel button and then using any painting or drawing tools to create the mask, or by making a selection and converting the selection into a mask by clicking the Save Selection As Channel button. When you paint the channel mask, the defaults are—black for masked areas, white for selected areas, and shades of gray for percentages of selection.

Use Channel Masks

Creating Channel Masks from Scratch

Channel masks are easy to create and once created, are just as easy to modify. All you need is an open document, and access to the Channels panel. After selecting a painting or drawing tool, you paint on the mask to define the selection area. The problem is you can’t see the image; you can only see the mask. What you need is the ability to view the mask and the image at the same time, as if you were using tracing paper, and then use the drawing tools to paint (trace) the portions of the image you want to select. The secret to viewing the image as you create the mask is to temporarily enable or show, the composite channel. In fact, the composite channel acts like a toggle switch—when it’s visible, you see the image and the mask (tracing paper); when it’s hidden, you only see the mask.

Create Channel Masks

Modifying Channel Mask Options

Channel masks have default options that control how the mask looks and functions. Photoshop gives you the ability to change the default options of a channel mask. For example, you could change the default color from red to blue, or you could change the function of the channel mask from Selection to Spot Color. Knowing you can change the mask options gives you more control over the final results.

Modify Channel Mask Options

Creating Channel Masks from Selections

Creating channel masks from scratch (using brush and drawing tools) is a useful feature, and gives you the ability to create a mask in any desired size or shape. However, there are times when it would be easier to first define the areas you want to protect, and then create the mask. When you create a channel mask from an existing selection, Photoshop uses the selected areas to create the mask. For example, you have an image of a woman wearing a red dress, and you want to change the color of the dress to green. The first step would be to select the red dress. Rather than create a new mask, and paint out the area representing the dress, it would be easier to first use a tool like the Magic Wand, select the dress, and then convert the selected area (the dress) into a channel mask. Once the mask is created, you could fine-tune the mask using Photoshop’s painting tools, and then change the dress color. Whether you create a mask from scratch or choose to create one through a predefined selection depends on the image and what you’re trying to accomplish.

Create Channel Masks from Selections

Making Channel Masks from Native Color Channels

Creating a channel mask from a native color channel takes a few steps; however, if you’re successful, the effort spent making the selection is well worth it. The trick to creating a mask with a native color channel is to use the shifts of gray in one specific color channel to create the black and white areas characteristic of a typical channel mask. For example, you have an image of a model, and you want to remove the model from the background. Unfortunately, that requires selecting around the model’s hair, a difficult thing to accomplish, even in the best of circumstances. To make the channel mask, open the Channels panel and examine the native color channels, one at a time. You’re looking for a color channel that displays a significant shift of gray (amount of contrast) between the model’s hair and the background. For example, you click on the red channel of an RGB image and the model’s hair appears dark gray, while the background appears light gray. The difference is so pronounced you can actually see individual strands of hair standing out against the background. If you can find such a contrast, you can quickly make a channel mask.

Make Channel Masks from Native Color Channels

Loading Channel Masks

Once you create a channel mask (and you can have up to 28 separate masks in one document), the next step is to use the masks. To save the channel mask all you have to do is save the document in a format that supports channels, such as Photoshop’s native PSD format, or even the TIFF format. The next time you open the document, your channel masks will be there. Deleting a channel mask is simple; just drag the channel mask over the Delete button located at the bottom of the Channels panel, or select the channel mask you want to delete, and click the Delete button. However, sooner or later, you’re going to want to do more than just save or delete—you’re going to want to use a channel mask. Using a channel mask involves a process of converting the mask back into a selection. It’s an easy step, but necessary to complete the process from mask to selection.

Load Channel Masks

Moving Channel Masks Between Documents

Once you create a channel mask in one document, it is possible to move that channel mask to another document. While most channel masks are so specific to a particular document it wouldn’t be practical to move them—a channel mask defining a selection of a specific tree against a blue sky, for example—many channel masks can be used over and over again. For example, you might have a series of channel masks creating unique selection borders around an image. You spent a lot of time creating the borders, and you would like to apply those same border selections to other images. If that’s the case, then increase your efficiency by saving them as channel masks and moving them between documents. Not only will it save you a lot of time, but using selections more than once can add a sense of cohesiveness to a design.

Move Channel Masks Between Documents

Combining Channel Masks

Channel masks are simply selections defined by black, white, and shades of gray. Once a channel mask is placed in the Channels panel, you can use Photoshop’s vast array of drawing and painting tools or filters. The Gaussian Blur filter can make a great enhancement to a channel mask. It’s even possible to combine the selection elements of two or more channels together, and in doing so, create an even more complicated mask.

Combine Channel Masks

Using the Quick Mask Mode

Quick Mask mode gives you the ability to create a selection using painting and drawing tools without creating a channel mask. For example, you’re creating a selection using traditional selection tools and there’s a portion of the image you’re having difficulty selecting. Since this is a one-time selection, you don’t want to go to the trouble of creating a channel mask. The solution is to move into the Quick Mask mode. Quick Mask mode toggles between a normal (Standard) selection view and a Quick Mask view. When you enter Quick Mask mode, any preexisting selections are converted into a red mask, and changes to the mask are performed using painting tools. When you return to Standard mode, the masked (painted) areas are converted into a selection. While Quick Masks are created the same way as channel masks, they’re temporary. It’s a quick way to create a one-time selection.

Use the Quick Mask Mode

Working with Quick Mask Options

When you work in the Quick Mask mode, the color for the mask is red, the opacity of the mask is 50%, and the red mask represents the masked areas of the document. Photoshop uses these Quick Mask options as the default, but they can be modified. For example, it would be very difficult to view a red mask if you were working on a primarily red image, or you might want to increase or decrease the opacity of the mask. Photoshop lets you do this through Quick Mask options.

Work with Quick Mask Options

Modifying Selections with Quick Mask Mode

Photoshop represents an active selection using an animated, single-pixel wide marquee, sometimes referred to as a “marching ant” marquee. Typically, the enclosed or “marquee” area represents the working area of the document. Unfortunately, when selections become complicated, you could wind up with ants marching all over the screen. While complicated selections are a part of the Photoshop designer’s life, they shouldn’t have to be hard to visualize or modify. Photoshop knows this and created the Quick Mask option. When you’re using Quick Mask, Photoshop displays the selected areas with a user-defined color and opacity. Then, by using your painting tools, you can make quick work of modifying the selection.

Modify Selections with Quick Mask Mode