We are back this week for part 2 to last week’s post. When I was working on this post, it reminded me of apps that allow you to decorate and furnish the rooms and walls of your house to help you better visualize your ideas. The ones that let you see if this color or that painting or those chairs would look great in your living room.
So, those three empty frames keep staring back at me. How can you use OpenCV and Python to see what paintings would look great on that wall, you ask? Good Question!
This week we are going to talk about bounding boxes, show how you can select regions of images (ROI), and combine them with scaling and color transformations for a very practical application: interior design.
For this tutorial, I used OpenCV 3.4.1 and Python 3.6.
Why are you showing me a picture of foxes when you want to decorate a wall? Those boxes in the image are called bounding boxes and are used for many practical applications in computer vision tasks, from face detection, to object detection and localization, and a bunch of other tasks. They are very useful for locating objects in images and tracking objects in videos.
In computer vision, a bounding box is used to represent a possible region of image (ROI) (sometimes written as region of interest). Feature and region detection algorithms will return the ROI in the form of pixel coordinates and the width and height of the box. But for this lesson, we are going to manually select our own region of interest, rather than use a detection algorithm to locate our ROIs.
Select ROIs
Our goal is to fill in those three frames with images.
Therefore, what we want to do is select the areas in the frames, these being our ROIs, and then return the x and y coordinates of the top left corner of the area, followed by the size of the rectangle we draw on the image. We can then use this information to edit the images we want and then copy them onto the image of the wall.
To select the ROIs, we are going to use a function in OpenCV called selectROI( ). Open a new file, and call it select_roi.py.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
import cv2 # import opencv # load image image = cv2.imread('gray_wall.jpg') # press 'q' to exit while loop while True: # display the image and wait for key press cv2.imshow("Original Image", image) key = cv2.waitKey() # if 'a' is pressed, draw roi on copy image if key == ord('a'): (x,y,w,h) = cv2.selectROI("Image", image, fromCenter=False, showCrosshair=False) # make sure an area is selected before pressing 'enter', 'space' or 'q' # crop the image and display cropped image cropped_roi = image[y:y+h, x:x+w] cv2.imshow("Cropped", cropped_roi) # print y, x coordinates for roi, and width and height of selected roi print("Coordinates: ({}, {}), Width: {}, Height: {}".format(y, x, w, h)) elif key == ord('q'): break # close all the windows cv2.destroyAllWindows() |
To run the program in terminal:
|
1 |
$ python select_roi.py |
When you start the program, we load the image in line 4 and then enter the while loop. You will see the original wall image displayed, line 9. The program is waiting for a key, either ‘a’ or ‘q’, to be pressed in line 10.
Pressing ‘a’ will open a copy of the original window, where we can draw our ROIs. Select the top left corner of the area that you want to draw a rectangle, drag the mouse, and when you finish, release the mouse and press the ‘enter’ key or ‘space’ key.
In lines 14 – 15, shows the selectROI( ) function. It takes as arguments the title of the window, the image, fromCenter (if fromCenter = True, then we start drawing the box from the center), and showCrosshair (if True, then it separates the box into four sections). This returns the top left x and y coordinates of the bounding box, and the width and height.
NOTE: There is still an error I am trying to figure out, and will update the code once I do. Basically, if you press ‘a’ and then press ‘enter’ before selecting an ROI you will get an error. I believe it is because we are loading an empty image with imshow( ) in line 19.
When then crop the image in line 18 and display the cropped section to make sure we captured the region that we wanted.
In the terminal window, you should see messages related to the selectROI ( ) function. After we press ‘enter’, the ROI information will be displayed (line 22). If you make an error, you can always press the ‘a’ key again, and draw another ROI.
These are the coordinates for the 1st ROI. (NOTE: The coordinates are displayed as (y, x), not (x, y).)
|
1 2 3 |
Select a ROI and then press SPACE or ENTER button! Cancel the selection process by pressing c button! Coordinates: (104, 75), Width: 99, Height: 126 |
And the 2nd and 3rd ROIs.
|
1 |
Coordinates: (104, 264), Width: 99, Height: 126 |
|
1 |
Coordinates: (104, 446), Width: 99, Height: 126 |
Now that we have the locations and sizes, we can move onto editing the images however we would like.
Image Processing
For our picture to test on the wall we are going to use a single image of a redwood tree and process it multiple times.
Refer back to the image processing tutorial for help if you get stuck. I will discuss the image transformations that I used in this tutorial.
Open a new file, and save it as add_images.py.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
import cv2 # import opencv # load images of test images and wall image = cv2.imread('redwood.jpg') wall_image = cv2.imread('gray_wall.jpg') # get the size of the image print("Image size: {}".format(image.shape)) # Adjust the size of the images based on the height of the ROI height = 126 width = int(image.shape[1] * height / image.shape[0]) resize_dim = (width, height) # resize image and display image and size resized_img = cv2.resize(image, resize_dim, interpolation = cv2.INTER_AREA) print("Resized image size: {}".format(resized_img.shape)) cv2.imshow('Resized image', resized_img) cv2.waitKey(0) # crop images to fit the ROI width and height, and change color cropped_img = resized_img[:, 13:112] cie_img = cv2.cvtColor(cropped_img, cv2.COLOR_RGB2XYZ) bgr_img = cv2.cvtColor(cropped_img, cv2.COLOR_RGB2BGR) |
All of this can be done a number of different ways. Here we resize the images first, because we want to retain the entire tree in the image, and then crop them so they better fit the ROI sizes (the inside of the frame). However, depending upon your image and which part you want to display, the order and transformations will be different. Definitely experiment with what you think looks best!
We load the images (line 4 – 5), get the size (line 8), adjust the dimensions based on height not width since these picture frames are taller than wider (lines 11 – 13), resize the image and look at the new dimensions (lines 16 – 17), and finally crop (line 22) to fit the ROI height and width, and change the color (lines 23 – 24).
For the next part, since we are simply using ROIs and they are of rectangular shape, we can simply play around with the index values and copy images to other parts of images. If we wanted to extract parts of the images before copying them to a new image, we would actually need to use a few bitwise operations. Luckily, that is not the case.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# add smaller image to larger image, create an ROI in the location of larger image rows, cols, channels = cropped_img.shape wall_image[104:104 + rows, 75:75 + cols] = small_img # left wall_image[104:104 + rows, 264:264 + cols] = cropped_img # middle wall_image[104:104 + rows, 446:446 + cols] = bgr_img # right cv2.imshow('New wall', wall_image) cv2.waitKey(0) # save final image cv2.imwrite('new_wall.jpg', wall_image) cv2.destroyAllWindows() |
In line 2, we get the number of rows and cols (width and height) of the cropped image. For this image, all of the frames are the same. However, if we had different images and frame sizes we would need to check the shape of those images, too.
Since an image is a big matrix of pixels, we can play with those index values and replace them with other values. Lines 3 – 5 set index values in wall_image[y top left corner : y left corner + height of image, x top left corner : x left corner + width of image] equal to the values of the cropped and recolored images. We use the x and y coordinates that we found earlier when drawings ROIs.
For the middle image, we will just use the original colors, and the right-most image we will use the BGR color scheme.
After that, we display the image, save it, and close all windows.
And here is another image I did:
Summary
In today’s tutorial, we looked at how we can use OpenCV and Python for creativity and for practical reasons. We learned about ROIs and how that can be used to select regions of an image for detection. Today’s post also covered a way that image processing and manipulation can be useful for everyday purposes.
There are a number of ways that this code could be improved, or other ways to achieve the same goal. If you have an ideas, please leave a comment and let me know what you would like to see in the future.
References
OpenCV Tutorials for image processing
