It could be the era of deep learning and big data, where complex algorithms analyze images by showing millions, but color spaces are still surprisingly useful for image analysis. Simple methods can still be powerful.
In this article you will learn how to simply segment an object from a color-based image in Python using OpenCV. A popular artificial vision library written in C / C ++ with links to Python, OpenCV offers simple ways to manipulate color spaces.
Although it is not necessary to be familiar with OpenCV or the other helper packages used in this article, it is assumed that you have at least some basic knowledge of the encoding in Python.
What are the color spaces?
In the most common color space, RGB (red green blue), colors are represented in terms of red, green and blue components. In more technical terms, RGB describes a color as a tuple of three components. Each component can assume a value between 0 and 255, where the tuple
(0, 0, 0) represents black and
(255, 255, 255) it represents white.
RGB is considered an "additive" color spaceand colors can be imagined as produced by brilliant amounts of red, blue and green light on a black background.
Here are some examples of RGB colors:
|Red||255, 0, 0|
|Orange||255, 128, 0|
|Pink||255, 153, 255|
RGB is one of the five main color space models, each of which has many offshoots. There are so many color spaces because the different color spaces are useful for different purposes.
In the world of printing, CMYK It is useful because it describes the color combinations required to produce a color from a white background. While the tuple 0 in RGB is black, in CMYK the tuple 0 is white. Our printers contain cyan, magenta, yellow and black ink containers.
In some types of medical fields, slides mounted with colored tissue samples are scanned and saved as images. They can be analyzed HED space, a representation of the saturations of stain types: hematoxylin, eosin and DAB applied to the original tissue.
HSV and HSL are descriptions of hue, saturation and luminance / luminance, which are particularly useful for identifying contrast in images. These color spaces are frequently used in color selection tools in software and for web design.
In reality, color is a continuous phenomenon, in the sense that there is an infinite number of colors. Color spaces, however, represent color through discrete structures (a fixed number of integer values of integers), which is acceptable because even the human eye and perception are limited. The color spaces are fully capable of representing all the colors that we are able to distinguish.
Now that we understand the concept of color spaces, we can continue to use them in OpenCV.
Simple segmentation using color spaces
To demonstrate the technique of color space segmentation, we provided a small set of clown fish image data in the Real Python material repository here for download and play. Clownfish are easily identifiable by their bright orange color, so they are a good candidate for segmentation. Let's see how we can find Nemo in an image.
The key Python packages to follow are NumPy, the most advanced scientific calculation package in Python, Matplotlib, a print library and, of course, OpenCV. This article uses OpenCV 3.2.0, NumPy 1.12.1 and Matplotlib 2.0.2. Slightly different versions will not make a significant difference in terms of following and grasping the concepts.
If you are not familiar with NumPy or Matplotlib, you can read about them in the official NumPy guide and the excellent article by Brad Solomon on Matplotlib.
Color spaces and reading images in OpenCV
First of all, you need to configure your environment. This article will assume that you have installed Python 3.x on your system. Note that while the current version of OpenCV is 3.x, the package name to be imported is still
If you have not previously installed OpenCV on your computer, the import will fail until you do it before. You can find an easy-to-use tutorial for installation on different operating systems here, as well as the OpenCV installation guide. Once you have imported OpenCV, you can look at all the color space conversions provided by OpenCV and you can save them all in one variable:
>>> flags = [[[[I for I in dir(CV2) Self I.begins with(& # 39; COLOR _ & # 39;)]
The list and the number of flags may vary slightly depending on the version of OpenCV, but regardless, there will be a lot! See how many flags you have available:
>>> len(flags) 258 >>> flags[[[ & # 39; COLOR_BGR2RGB & # 39;
The first characters later
COLOR_ indicates the source color space and the characters after the
2 they are the target chromatic space. This flag represents a conversion from BGR (blue, green, red) to RGB. As you can see, the two color spaces are very similar, with only the first and last channel traded.
You will need
matplotlib.pyplot for displaying images and NumPy for some image manipulations. If you have not already installed Matplotlib or NumPy, you will have to do it
pip3 installs matplotlib is
pip3 installs numpy before attempting imports:
>>> import matplotlib.pyplot as plt >>> import numpy as np
You are now ready to upload and review an image. Note that if you work from the command line or terminal, your images will appear in a pop-up window. If you are working on a Jupyter notebook or something similar, they will simply be shown below. Regardless of your setup, you should see the image generated by the
to exhibit() command:
>>> nemo = CV2.imread(& # 39; ./ images / nemo0.jpg & # 39;) >>> plt.imshow(nemo) >>> plt.to exhibit()
Hey, Nemo … or Dory? You will notice that it appears that the blue and red channels have been confused. Indeed, OpenCV reads images in BGR format by default. You can use the
cvtColor (image, flag) and the flag we saw above to solve this problem:
>>> nemo = CV2.cvtColor(nemo, CV2.COLOR_BGR2RGB) >>> plt.imshow(nemo) >>> plt.to exhibit()
Now Nemo looks a lot more like himself.
Display of Nemo in RGB Color Space
HSV is a good choice of color space for segmentation by color, but to understand why, we compare the image in RGB and HSV color spaces by displaying the color distribution of its pixels. A 3D chart shows it quite well, with each axis representing one of the channels in the color space. If you want to know how to create a 3D chart, look at the compressed section:
To create the texture, you will need other Matplotlib libraries:
>>> from mpl_toolkits.mplot3d import Axes3D >>> from matplotlib import centimeter >>> from matplotlib import colors
These libraries provide the features necessary for the plot. You want to position each pixel in its position based on its components and color it according to color. OpenCV
Divided() it is very useful here; splits an image into its component channels. These few lines of code divide the image and set the 3D plot:
>>> r, g, B = CV2.Divided(nemo) >>> Figure = plt.figure() >>> axis = Figure.add_subplot(1, 1, 1, projection="3d")
Now that you have set the plot, you need to set the pixel colors. To color each pixel based on its true color, a little bit of remodeling and normalization is needed. It seems chaotic, but in essence it is necessary that the colors corresponding to each pixel of the image are flattened in a list and normalized, so that they can be passed to the
facecolors Matplotlib parameter
will disperse ().
Normalizing simply means condensing the range of colors from
0-1 as required for the
facecolors parameter. Lastly,
facecolors wants a list, not a NumPy array:
>>> pixel_colors = nemo.remould((np.form(nemo)[[[*np.form(nemo)[[[, 3)) >>> norm = colors.Normalize(Vmin= -1.,vmax=1.) >>> norm.ladder trucks(pixel_colors) >>> pixel_colors = norm(pixel_colors).to list()
Now we have all the components ready for printing: the positions of the pixels for each axis and the corresponding colors, in the format
facecolors he waits. You can build the scatterplot and view it:
>>> axis.scatter(r.flatten() g.flatten() B.flatten() facecolors=colors, marker="") >>> axis.set_xlabel("Red") >>> axis.set_ylabel("Green") >>> axis.set_zlabel("Blue") >>> plt.to exhibit()
Here is the colored scatter plot for the Nemo image in RGB:
From this plot, you can see that the orange parts of the image cover almost the entire range of red, green, and blue values. Since parts of Nemo extend across the plot, Nemo's segmentation in RGB space based on ranges of RGB values would not be easy.
Display of Nemo in HSV Color Space
We saw Nemo in the RGB space, so now let's look at it in the HSV space and compare it.
As mentioned briefly above, HSV stands for hue, saturation and value (or brightness)and it is a cylindrical color space. The colors, or hues, are modeled as an angular dimension that rotates around a central vertical axis, which represents the value channel. Values range from darkness (0 below) to illuminate at the top. The third axis, saturation, defines the shades of hue from the less saturated ones, on the vertical axis, to the most saturated ones farthest from the center:
To convert an image from RGB to HSV, you can use
>>> hsv_nemo = CV2.cvtColor(nemo, CV2.COLOR_RGB2HSV)
hsv_nemo memorizes the representation of Nemo in HSV. Using the same technique as above, we can look at an image graph in HSV, generated by the following compressed section:
The code to show the image in HSV is the same as RGB. Note that you use the same
pixel_colors variable to color the pixels, because Matplotlib expects the values to be in RGB:
>>> h, S, v = CV2.Divided(hsv_nemo) >>> Figure = plt.figure() >>> axis = Figure.add_subplot(1, 1, 1, projection="3d") >>> axis.scatter(h.flatten() S.flatten() v.flatten() facecolors=pixel_colors, marker="") >>> axis.set_xlabel("Hue") >>> axis.set_ylabel("Saturation") >>> axis.set_zlabel("Value") >>> plt.to exhibit()
In HSV space, Nemo oranges are much more localized and visually separable. The saturation and the value of oranges vary, but are mainly found within a small interval along the axis of the hue. This is the key point that can be exploited for segmentation.
Choose a range
We remove Nemo's threshold based on a simple range of oranges. You can choose the range by looking up the chart above or using an online color selection app like this RGB to HSV tool. The samples chosen here are a light orange and a darker orange which is almost red:
>>> Light orange = (1, 190, 200) >>> dark orange = (18, 255, 255)
If you want to use Python to display the colors you've chosen, click on the collapsed section:
An easy way to display colors in Python is to create small square images of the desired color and plot them in Matplotlib. Matplotlib only interprets the colors in RGB, but useful conversion functions are provided for the main color spaces, so that we can draw the images in other color spaces:
>>> from matplotlib.colors import hsv_to_rgb
So, build the baby
10x10x3 squares, filled with the respective color. You can use NumPy to easily fill the squares with the color:
>>> lo_square = np.full((10, 10, 3) Light orange, DTYPE=np.uint8) / 255.0 >>> do_square = np.full((10, 10, 3) dark orange, DTYPE=np.uint8) / 255.0
Finally, you can create them together by converting them to RGB for viewing:
>>> plt.subplot(1, 2, 1) >>> plt.imshow(hsv_to_rgb(do_square)) >>> plt.subplot(1, 2, 2) >>> plt.imshow(hsv_to_rgb(lo_square)) >>> plt.to exhibit()
This produces these images, filled with the chosen colors:
Once you have a decent range of colors, you can use it
cv2.inRange () try to overcome Nemo.
inRange () takes three parameters: the image, the lower range and the highest range. Returns a binary mask (a
ndarray of 1s and 0s) the size of the image where values of
1 indicates values within the interval and zero values indicate external values:
>>> mask = CV2.InRange(hsv_nemo, Light orange, dark orange)
To impose the mask over the original image, you can use
cv2.bitwise_and (), which retains every pixel in the given image if the corresponding value in the mask is
>>> result = CV2.bitwise_and(nemo, nemo, mask=mask)
To see what he did exactly, let's see both the mask and the original image with the mask at the top:
>>> plt.subplot(1, 2, 1) >>> plt.imshow(mask, CMAP="Grey") >>> plt.subplot(1, 2, 2) >>> plt.imshow(result) >>> plt.to exhibit()
Here it is! This has already done a good job in capturing the orange parts of the fish. The only problem is that Nemo also has white stripes … Fortunately, adding a second mask that looks for whites is very similar to what you've already done with oranges:
>>> light_white = (0, 0, 200) >>> dark_white = (145, 60, 255)
Once you've specified a range of colors, you can look at the colors you've chosen:
To view whites, you can follow the same approach as before with oranges:
>>> lw_square = np.full((10, 10, 3) light_white, DTYPE=np.uint8) / 255.0 >>> dw_square = np.full((10, 10, 3) dark_white, DTYPE=np.uint8) / 255.0 >>> plt.subplot(1, 2, 1) >>> plt.imshow(hsv_to_rgb(lw_square)) >>> plt.subplot(1, 2, 2) >>> plt.imshow(hsv_to_rgb(dw_square)) >>> plt.to exhibit()
The upper range I have chosen here is a very blue white, because white has shades of blue in the shade. Let's create a second mask and see if it captures Nemo strips. You can build a second mask the same way you did the first one:
>>> mask_white = CV2.InRange(hsv_nemo, light_white, dark_white) >>> result_white = CV2.bitwise_and(nemo, nemo, mask=mask_white) >>> plt.subplot(1, 2, 1) >>> plt.imshow(mask_white, CMAP="Grey") >>> plt.subplot(1, 2, 2) >>> plt.imshow(result_white) >>> plt.to exhibit()
Not bad! Now you can combine masks. The addition of the two masks together results in
1 values wherever there is orange or white, which is exactly what is needed. We add the masks together and plot the results:
>>> final_mask = mask + mask_white >>> final results = CV2.bitwise_and(nemo, nemo, mask=final_mask) >>> plt.subplot(1, 2, 1) >>> plt.imshow(final_mask, CMAP="Grey") >>> plt.subplot(1, 2, 2) >>> plt.imshow(final results) >>> plt.to exhibit()
Basically, you have an approximate segmentation of Nemo into the HSV color space. You will notice that there are a few wandering pixels along the edge of the segmentation and, if you wish, you can use a Gaussian blur to reorder the small erroneous readings.
A Gaussian blur is an image filter that uses a type of function called Gaussian to transform each pixel in the image. It has the result of attenuating image noise and reducing details. Here's how to apply the blur for our image:
>>> blur = CV2.Gaussian blur(final results, (7, 7) 0) >>> plt.imshow(blur) >>> plt.to exhibit()
Is this segmentation generalized to the relatives of Nemo?
Just for fun, let's see how this segmentation technique is generalized to other images of clown fish. In the repository, there is a selection of six images of Google's clownfish, licensed for public use. The images are in a subdirectory and an indexed nemoI.jpg, where I is the index from
First, load all Nemo's relatives into a list:
pathway = "./images/nemo" nemos_friends =  for I in range(6): friend = CV2.cvtColor(CV2.imread(pathway + str(I) + ".jpg") CV2.COLOR_BGR2RGB) nemos_friends.to add(friend)
You can combine all the code used above to segment a single fish into a function that will take an image as input and return the segmented image. Expand this section to see what it looks like:
here is the
segment_fish () function:
DEF segment_fish(Image): & # 39; & # 39; & # 39; Attempts to segment the clownfish with the image provided & # 39; & # 39; & # 39; # Convert the image to HSV hsv_image = CV2.cvtColor(Image, CV2.COLOR_RGB2HSV) # Set the orange range Light orange = (1, 190, 200) dark orange = (18, 255, 255) # Apply the orange mask mask = CV2.InRange(hsv_image, Light orange, dark orange) # Set a white range light_white = (0, 0, 200) dark_white = (145, 60, 255) # Apply the white mask mask_white = CV2.InRange(hsv_image, light_white, dark_white) # Combine the two masks final_mask = mask + mask_white result = CV2.bitwise_and(Image, Image, mask=final_mask) # Clean up the segmentation using a blur blur = CV2.Gaussian blur(result, (7, 7) 0) return blur
With this useful function, you can segment all fish:
results = [[[[segment_fish(friend) for friend in nemos_friends]
We see all the results tracing them in a cycle:
for I in range(1, 6): plt.subplot(1, 2, 1) plt.imshow(nemos_friends[[[[I]) plt.subplot(1, 2, 2) plt.imshow(results[[[[I]) plt.to exhibit()
The clownfish in the foreground has darker shades of orange in our range.
The shaded lower half of Nemo's nephew is completely ruled out, but the pieces of purple anemone in the background look awfully like Nemo's azure stripes …
Overall, this simple segmentation method successfully located the majority of Nemo relatives. It is clear, however, that segmenting a clownfish with particular lighting and background may not necessarily be generalized to the segmentation of all clown fish.
In this tutorial you have seen which are different color spaces, how an image is distributed between RGB and HSV color spaces and how to use OpenCV to convert between color spaces and segmentation intervals.
Overall, you have learned as a basic understanding of how color spaces in OpenCV can be used to perform object segmentation in images, and hopefully it has seen its potential to perform other tasks as well. Where lighting and background are controlled, as in an experimental setup or with a more homogeneous data set, this segmentation technique is simple, fast and reliable.