Making Complex Selections in Photoshop

Making precise selections in Photoshop is an essential skill that every designer needs. As a print designer, you will find yourself using selections to remove objects from their background to place within ads. As a web designer, you might extract an image and place it on a website with no background. If you are a photographer, you might make a selection to remove blemishes or other unwanted features from an image. You can’t get around it; everyone uses selections, and if you make excellent selections, you’ll end up with excellent work.

So, what do you do when you have something that is extremely difficult to select with normal selection tools? As basic selection tools, you have the marquee tool, the lasso tools, the magic wand tool, and the quick selection tool at your disposal. These don’t always do the trick; you don’t have a truly precise tool to make intricate selections.

Even with the masking and channels, you can’t make a precise selection consistently. Sure, you can refine your selection, feather it, and spend a lot of time on it, but that method is neither easy nor consistent. Notice in the image below (which you can download from here), there is a woman on a subtle background. This might not be considered a terribly complex selection, but the hair is fairly difficult to extract onto its own layer.

You can try the quick selection tool, which does a great job of making a selection of the bulk of the image, but making a more refined selection is difficult with this method. The problem is the hair. The wisps of hair are too fine and tedious to select with standard selection tools.

In comes Refine Mask to the rescue. In the example, I made the selection using the Quick Selection Tool. This does a good job of selecting most of the image, but now we are going to refine our selection. In Photoshop, when you make a selection, the options menu for your selection tool should offer a button that says “Refine Mask.” Click that button to bring up the Refine Mask Dialog Box.

The Refine Mask menu is one of the best tools for refining your selection. You have many options for viewing your mask:

• Marching Ants, which is your basic selection view.
• Overlay, which is the same view as quick mask mode.
• On Black, which gives you a lot of contrast in your view and looks just like a normal layer mask.
• On White, which offers the same kind of contrast with the opposite background.
• Black & White, which helps you distinguish your selection from the background when you are trying to see dark and light areas of an image that you want to add and subtract from your selection.
• On Layers, which shows your underlying layers.
• Reveal layer, which shows the normal background.

Each view has its own purpose, but the best views for contrast would have to be On Black, On White, and Black & White. This dialog box is handy, because you can easily toggle between each view by hitting the “F” key. You can see in the example below that our selection is good, but the wisps of hair are not included in the selection.

We can fix this by using the Smart Radius Slider. Just by using this slider, we can a more refined and precise selection. I bumped it all the way up to 250px and in doing so, have the result shown below:

This slider does a great job, but there are still a couple of hair wisps that aren’t included in the selection, making our image incomplete. We want our images to look as natural as possible, so we’ll need to include those. The Refine Mask dialog has a brush on the left that you can use to add to and subtract from the selection. Hold shift to add to the selection, and hold alt/​option to subtract from the selection. If you need a different view, use the “F” key to toggle through the different views.

Our selection looks great and is much more detailed than before. As we cycle through the different views, we can see the hair that we wanted to include in our selection. Many would say that this is a very good selection, but now, if you go to the view called On Black, you can see a light outline around the woman’s body and in her hair.

This is called fringe. You could go through and erase it all or try to contract your selection, but it could cut into your image, which makes your selection worse instead of better. The best option in the Refine Edge menu is the bottom one called Output. If you select Decontaminate colors, you can move the slider to the right and remove the fringe.

You can also select how this is output. You can choose New layer with layer mask, new layer, new document, or new document with layer mask. If you still need to keep this in the current document, then choose to create a new layer with a layer mask. This gives you the ability to refine the layer mask even more. When you are done with your refinements, right-​​click on the layer mask icon and select Apply Mask.

With the image extracted, now you can place it over any background that you want. With your complex and professional level selections, you can convert them to masks and mask out and extract any object from its background. I added this background texture below the extracted image, and the result is shown below.

You might think that you need a lot of contrast for this to work, but in reality, you don’t. Here is a sample image of a dandelion, which you can find here. The image doesn’t have a lot of contrast, and the tones of the background and subject are very similar.

Make a rough selection of the dandelion. This doesn’t have to be exact, but try to select as much of the main image as you can. I used the quick selection tool, which took about 3 seconds to make this selection.

Then, I simply turned on smart radius and bumped the value all the way up.

To get rid of the extra content in the background, such as the green from the rest of the field, I used Decontaminate Colors and bumped the value all the way up. I exported the selection as a mask on its own layer and voila!

The Refine Edge and Refine Mask Settings can handle easy or tough selections, and it makes the process of making complex selections much easier. These techniques can make your Photoshop selections quicker, easier, and better.