How do I find several key pixel points in a picture using Python?

Question

I know that by Opencv-python's matchtemplate function can match picture 2 from picture 1.

import cv2


template = cv2.imread("1.bmp")
template = cv2.cvtColor(template, cv2.COLOR_BGR2GRAY)
template = cv2.Canny(template, 50, 200)


edged = cv2.imread("2.bmp")
edged = cv2.cvtColor(edged, cv2.COLOR_BGR2GRAY)
edged = cv2.Canny(edged, 50, 200)


result = cv2.matchTemplate(edged, template, cv2.TM_CCOEFF_NORMED)
(_, maxVal, _, maxLoc) = cv2.minMaxLoc(result)

But if I only have pixels like picture 3, how do I match from picture 1?

picture 3: Some pixels in picture two are extracted and the rest is filled in red.

I didn't find a opencv-python or a PIL solution. I think it can be solved by traversing pixels, but that doesn't guarantee performance. So is there a better way?

Possible duplicate of matchTemplate with non-rectangular templ. — WhoIsJack
– WhoIsJack, Commented Jun 28, 2018 at 16:47
So I have to turn the picture 3 into alpha? I don't want to do this, I hope the algorithm is able to automatically ignore those simple pixels, detect only those pixels that are prominent. @WhoIsJack — Martin Zhu
– Martin Zhu, Commented Jun 28, 2018 at 16:59
Unfortunately, "the algorithm" doesn't automagically ignore "simple" pixels, so you will have to provide a mask. As the answer in the linked question suggests, matchTemplate supports the mask parameter for this purpose (as of OpenCV 3). — WhoIsJack
– WhoIsJack, Commented Jun 28, 2018 at 17:15
OK, Can I generate a mask layer from the code in picture 3? to ignore the red part.@WhoIsJack — Martin Zhu
– Martin Zhu, Commented Jun 28, 2018 at 17:57
@MartinZhu try using keypoint feature extraction like ORB, SIFT, etc.. — Jeru Luke
– Jeru Luke, Commented Jun 29, 2018 at 5:53

WhoIsJack · Accepted Answer · 2018-06-29 12:09:53Z

1

According to the answer by Neeraj Komuravalli to this question, matchTemplate supports the mask argument as a way of excluding certain pixels in the template from being considered in the matching (docs).

To generate a mask based on the red pixels, a simple solution is to use a Boolean expression to select those pixels that are 0 in blue and green but 255 in red:

mask = (template[:,:,0]==0) & (template[:,:,1]==0) & (template[:,:,2]==255)
mask = ~mask
mask = mask.astype(np.uint8)

Note that the conversion to uint8 is necessary since the mask must be of the same datatype as the template.

Edit: ~mask inverts the mask (0 becomes 1 and vice versa), which is necessary since 0 indicates the pixels to be masked, at least when using the method cv2.TM_CCORR_NORMED.

Whilst this addresses your question in principle, it will not yield a working solution in this case.

This is because of the Canny edge filter that is being applied to the images. Since there is no way of masking the red pixels in the template when applying Canny, the boundaries of the red pixel regions will affect the result of the edge detection and thus change the template to look quite different from the original.

In this example the matching fails as a consequence, returning a position that is completely wrong.

Removing the Canny steps solves this problem... but it also makes the approach a little less robust/precise. In this case, the matching actually seems to end up a few pixels off from the 100% correct match. Unfortunately, I can't think of any means of improving this.

Here's the complete code that works for me (with the aforementioned caveat in terms of precision):

import cv2
import numpy as np

template = cv2.imread("masked_template.png")

mask = (template[:,:,0]==0) & (template[:,:,1]==0) & (template[:,:,2]==255)
mask = mask.astype(np.uint8)

template = cv2.cvtColor(template, cv2.COLOR_BGR2GRAY)
#template = cv2.Canny(template, 50, 200)

edged = cv2.imread("edged.png")
edged = cv2.cvtColor(edged, cv2.COLOR_BGR2GRAY)
#edged = cv2.Canny(edged, 50, 200)

result = cv2.matchTemplate(edged, template, cv2.TM_CCORR_NORMED, mask=mask)
(_, maxVal, _, maxLoc) = cv2.minMaxLoc(result)

edited Jun 29, 2018 at 12:09

answered Jun 28, 2018 at 23:05

WhoIsJack

1,5082 gold badges18 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Martin Zhu Over a year ago

Thank you for your answer. I have a question that (template[:,:,0]==0) & (template[:,:,1]==0) & (template[:,:,2]==255) will make the red pixels true (uint8 == 1), however, 0 in the mask indicates occlusion.

WhoIsJack Over a year ago

Where did you get this information? I wasn't sure and so I just tested both (you can invert the mask by adding mask = ~mask before the astype line); assuming that 1 occludes gives a decent match, assuming that 0 occludes doesn't.

WhoIsJack Over a year ago

Hm... apparently that also depends on the method used. In the answer, I am using cv2.TM_SQDIFF, which gives a good result with this mask and a poor result with the inverted mask. However, when using cv2.TM_CCORR_NORMED it seems to be the other way around. Perhaps you could use some additional test cases to check whether this observation is consistent or spurious?

Martin Zhu Over a year ago

I used TM_CCORR_NORMED, because it can get a fractional match rate, it is more intuitive. After testing, It seems to indicate 0 in the mask indicates occlusion . . .

Martin Zhu Over a year ago

I found a new problem. when using GRAY, the match degree is 0.9, and when use BGR, the match is only 0.4. This means that when using GRAY, the grayscale will make the error a lot more. ...@WhoIsJack

|

Collectives™ on Stack Overflow

How do I find several key pixel points in a picture using Python?

1 Answer 1

7 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related