How to improve image segmentation using the watershed?

Question

I am developing an application to detect the lesion area, for this I am using the grabcut to detect the ROI and remove the background from the image. However in some images it is not working well. He ends up not identifying the borders of the region of interest well. The watershed can better identify the edges for this type of work, however I am having difficulties making this transition from grabcut to watershed. Before processing the grabcut, the user uses touchevent to mark a rectangle around the image of interest (wound area) to facilitate the work of the algorithm. As the image below.

However, using other wound images, segmentation is not good, showing flaws in ROI detection.

Image using grabcut in app

Image using watershed in desktop

this is the code:

private fun extractForegroundFromBackground(coordinates: Coordinates, currentPhotoPath: String): String {
    // TODO: Provide complex object that has both path and extension

    val width = bitmap?.getWidth()!!
    val height = bitmap?.getHeight()!!
    val rgba = Mat()
    val gray_mat = Mat()
    val threeChannel = Mat()
    Utils.bitmapToMat(bitmap, gray_mat)
    cvtColor(gray_mat, rgba, COLOR_RGBA2RGB)
    cvtColor(rgba, threeChannel, COLOR_RGB2GRAY)
    threshold(threeChannel, threeChannel, 100.0, 255.0, THRESH_OTSU)

    val rect = Rect(coordinates.first, coordinates.second)
    val fg = Mat(rect.size(), CvType.CV_8U)
    erode(threeChannel, fg, Mat(), Point(-1.0, -1.0), 10)
    val bg = Mat(rect.size(), CvType.CV_8U)
    dilate(threeChannel, bg, Mat(), Point(-1.0, -1.0), 5)
    threshold(bg, bg, 1.0, 128.0, THRESH_BINARY_INV)
    val markers = Mat(rgba.size(), CvType.CV_8U, Scalar(0.0))
    Core.add(fg, bg, markers)

    val marker_tempo = Mat()
    markers.convertTo(marker_tempo, CvType.CV_32S)

    watershed(rgba, marker_tempo)
    marker_tempo.convertTo(markers, CvType.CV_8U)

    val imgBmpExit = Bitmap.createBitmap(width, height, Bitmap.Config.RGB_565)
    Utils.matToBitmap(markers, imgBmpExit)

    image.setImageBitmap(imgBmpExit)


    // Run the grab cut algorithm with a rectangle (for subsequent iterations with touch-up strokes,
    // flag should be Imgproc.GC_INIT_WITH_MASK)
    //Imgproc.grabCut(srcImage, firstMask, rect, bg, fg, iterations, Imgproc.GC_INIT_WITH_RECT)

    // Create a matrix of 0s and 1s, indicating whether individual pixels are equal
    // or different between "firstMask" and "source" objects
    // Result is stored back to "firstMask"
    //Core.compare(mark, source, mark, Core.CMP_EQ)

    // Create a matrix to represent the foreground, filled with white color
    val foreground = Mat(srcImage.size(), CvType.CV_8UC3, Scalar(255.0, 255.0, 255.0))

    // Copy the foreground matrix to the first mask
    srcImage.copyTo(foreground, mark)

    // Create a red color
    val color = Scalar(255.0, 0.0, 0.0, 255.0)
    // Draw a rectangle using the coordinates of the bounding box that surrounds the foreground
    rectangle(srcImage, coordinates.first, coordinates.second, color)

    // Create a new matrix to represent the background, filled with black color
    val background = Mat(srcImage.size(), CvType.CV_8UC3, Scalar(0.0, 0.0, 0.0))

    val mask = Mat(foreground.size(), CvType.CV_8UC1, Scalar(255.0, 255.0, 255.0))
    // Convert the foreground's color space from BGR to gray scale
    cvtColor(foreground, mask, Imgproc.COLOR_BGR2GRAY)

    // Separate out regions of the mask by comparing the pixel intensity with respect to a threshold value
    threshold(mask, mask, 254.0, 255.0, Imgproc.THRESH_BINARY_INV)

    // Create a matrix to hold the final image
    val dst = Mat()
    // copy the background matrix onto the matrix that represents the final result
    background.copyTo(dst)

    val vals = Mat(1, 1, CvType.CV_8UC3, Scalar(0.0))
    // Replace all 0 values in the background matrix given the foreground mask
    background.setTo(vals, mask)

    // Add the sum of the background and foreground matrices by applying the mask
    Core.add(background, foreground, dst, mask)

    // Save the final image to storage
    Imgcodecs.imwrite(currentPhotoPath + "_tmp.png", dst)

    // Clean up used resources
    firstMask.release()
    source.release()
    //bg.release()
    //fg.release()
    vals.release()
    dst.release()

    return currentPhotoPath
}

Exit:

How do I update the code to use watershed instead of grabcut?

It's so annoying to read posts this long. You can improve the formatting by resizing the images using markdown tags, like: <img src="https://i.sstatic.net/nmzwj.png" width="210" height="150"> — karlphillip
– karlphillip, Commented Apr 28, 2020 at 17:57

desertnaut · Accepted Answer · 2024-03-06 04:10:05Z

2

A description of how to apply the watershed algorithm in OpenCV is here, although it is in Python. The documentation also contains some potentially useful examples. Since you already have a binary image, all that's left is to apply the Euclidean Distance Transform (EDT) and the watershed function. So instead of Imgproc.grabCut(srcImage, firstMask, rect, bg, fg, iterations, Imgproc.GC_INIT_WITH_RECT), you would have:

Mat dist = new Mat();
Imgproc.distanceTransform(srcImage, dist, Imgproc.DIST_L2, Imgproc.DIST_MASK_3); // use L2 for Euclidean Distance 
Mat markers = Mat.zeros(dist.size(), CvType.CV_32S);
Imgproc.watershed(dist, markers); # apply watershed to resultant image from EDT
Mat mark = Mat.zeros(markers.size(), CvType.CV_8U);
markers.convertTo(mark, CvType.CV_8UC1);
Imgproc.threshold(mark, firstMask, 0, 255, Imgproc.THRESH_BINARY + Imgproc.THRESH_OTSU); # threshold results to get binary image

The thresholding step is described here. Also, optionally, before you apply Imgproc.watershed, you may want to apply some morphological operations to the result of EDT i.e; dilation, erosion:

Imgproc.dilate(dist, dist, Mat.ones(3, 3, CvType.CV_8U));

If you're not familiar with morphological operations when it comes to processing binary images, the OpenCV documentation contains some good, quick examples.

edited Mar 6, 2024 at 4:10

desertnaut

60.8k32 gold badges155 silver badges183 bronze badges

answered Apr 17, 2020 at 3:12

danielcahall

2,92711 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

Carlos Diego Over a year ago

Daniel, I did as you suggested, but I had a mistake. For better viewing I updated the code and the error in the question.

danielcahall Over a year ago

Ah sorry - the distanceTransform function expects a grayscale image, so that line should change from Imgproc.distanceTransform(srcImage, dist, Imgproc.DIST_L2, Imgproc.DIST_MASK_3); to Imgproc.distanceTransform(threeChannel, dist, Imgproc.DIST_L2, Imgproc.DIST_MASK_3);, as it looks like threeChannel is the RGB image converted to grayscale in your code (Imgproc.cvtColor(srcImage, threeChannel, Imgproc.COLOR_RGB2GRAY))

Carlos Diego Over a year ago

Daniel, now it was another error in the watershed line (dist, markers), I updated the code and the error in the question. Another doubt Daniel, the rest of the code I just comment on the grabcut line and the rest is like this?

danielcahall Over a year ago

The error occurs here: github.com/opencv/opencv/blob/master/modules/imgproc/src/…. It appears that dist isn't the correct type - try adding dist.convertTo(dist, CvType.CV_8UC3); after Imgproc.distanceTransform(threeChannel, dist, Imgproc.DIST_L2, Imgproc.DIST_MASK_3);

Carlos Diego Over a year ago

Daniel, I managed to solve the problem. She can already recognize some regions of the wound, but how do I use the coordinates I got from the user so that the watershed works only in the coordinate area? I updated the code and the output.

|

Collectives™ on Stack Overflow

How to improve image segmentation using the watershed?

1 Answer 1

9 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

9 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related