OpenGL render to a OpenCV mat as texture and reuse it in OpenCV

Question

I have a OpenCV mat that I want to render to with OpenGL. So I try to load the openCV mat as a texture via a code that I found online and render to that texture:

void BindCVMat2GLTexture(cv::Mat& image, GLuint& imageTexture)
{
    if (image.empty()) {
        std::cout << "image empty" << std::endl;
    }
    else {
        glEnable(GL_TEXTURE_2D);
        glTexEnvi(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_MODULATE);
        //glTexEnvi(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_REPLACE);
        glGenTextures(1, &imageTexture);
        glBindTexture(GL_TEXTURE_2D, imageTexture);

        glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
        glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);

        // Set texture clamping method
        glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP);
        glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP);

        cv::cvtColor(image, image, CV_RGB2BGR);

        glTexImage2D(GL_TEXTURE_2D,         // Type of texture
            0,                   // Pyramid level (for mip-mapping) - 0 is the top level
            GL_RGB,              // Internal colour format to convert to
            image.cols,          // Image width  i.e. 640 for Kinect in standard mode
            image.rows,          // Image height i.e. 480 for Kinect in standard mode
            0,                   // Border width in pixels (can either be 1 or 0)
            GL_RGB,              // Input image format (i.e. GL_RGB, GL_RGBA, GL_BGR etc.)
            GL_UNSIGNED_BYTE,    // Image data type
            image.ptr());        // The actual image data itself
            //NULL);
    }
}

Then I create a FBO:

//init the texture
    GLuint imageTexture;
    BindCVMat2GLTexture(target, imageTexture);

    //init the frame buffer
    GLuint fbo = 0;
    glGenFramebuffers(1, &fbo);
    glBindFramebuffer(GL_FRAMEBUFFER, fbo);
    // Set "renderedTexture" as our colour attachement #0
    glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, imageTexture, 0);

Then I'm calling:

glViewport(0, 0, target.cols, target.rows);
MyDraw();

After this calls I use another function I found to copy the texture data back to an openCV mat:

cv::Mat GetOcvImgFromOglImg(GLuint ogl_texture_id)
{
    glBindTexture(GL_TEXTURE_2D, ogl_texture_id);
    GLenum gl_texture_width, gl_texture_height;

    glGetTexLevelParameteriv(GL_TEXTURE_2D, 0, GL_TEXTURE_WIDTH, (GLint*)&gl_texture_width);
    glGetTexLevelParameteriv(GL_TEXTURE_2D, 0, GL_TEXTURE_HEIGHT, (GLint*)&gl_texture_height);

    unsigned char* gl_texture_bytes = (unsigned char*)malloc(sizeof(unsigned char)*gl_texture_width*gl_texture_height * 3);
    glGetTexImage(GL_TEXTURE_2D, 0 /* mipmap level */, GL_BGR, GL_UNSIGNED_BYTE, gl_texture_bytes);

    return cv::Mat(gl_texture_height, gl_texture_width, CV_8UC3, gl_texture_bytes);
}

and call this by:

target = GetOcvImgFromOglImg(imageTexture);

Also I create triangle vertices and a vertex shader that simply leaves them untouched and a fragment shader that just puts out the color red. In theory I just want a red triangle drawn over this texture as a first step. However it is not working since I get no rendered image at all. I'm a very beginner with openGL and I can't find any solution by myself right now. Any ideas what I'm doing wrong?

Sergey K. · Accepted Answer · 2018-12-01 07:10:51Z

2

OpenCV store image as rows data tightly packed but might align them to certain byte boundaries may be 4 or 8 but we don't know that If you're lucky it will use 4-byte alignment which is default with OpenGL also.But it's best to manually set it with the pixel storage modes in order to be on the safe side:

// use fast 4-byte alignment (default anyway) if possible glPixelStorei(GL_UNPACK_ALIGNMENT, (image.step & 3) ? 1 : 4);

//set length of one complete row in data (doesn't need to equal image.cols) glPixelStorei(GL_UNPACK_ROW_LENGTH, image.step/image.elemSize());

Then you have to account for the fact that OpenCV stores images from top to bottom, while the GL uses bottom to top. To take care of this use following

cv::flip(image, flipped, 0); image = flipped;

OpenCV stores color images in BGR format so uploading it as RGB would distort the colors. So use GL_BGR in glTexImage2D.

edited Dec 1, 2018 at 7:10

Sergey K.

25.5k14 gold badges112 silver badges176 bronze badges

answered Dec 1, 2018 at 0:49

Paritosh Kulkarni

8727 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Jonas Bräuer Over a year ago

actually none of that solved my particular case, but I think it's because as you said 4-byte alignment is default anyway. Also no flipping was needed and as you can see in my code above I was using a openCV RGB2BGR conversion. So I could use your proposed method which does the same regarding the color format.

Jonas Bräuer · Accepted Answer · 2018-12-03 07:18:49Z

So I solved my problem, which I should have posted more code to make an answer easier, by adding a VertexBufferObject which I totally ignored from the tutorial. Also and this is very important, I use SFML lib at the same time to display my frames. SFML also uses OpenGL internally. So I need to save and restore the OpenGL states which interfere with eachother.

I drop this link in case somebody has similar problems: using-opengl-together-with-the-graphics-module

This added state save/restore before I use SFML together with the missing VAO code and a reset of OpenGL bindings after my Draw() method made it work. This might not be the most performant solution but for now it is a way to achieve what I needed.

Collectives™ on Stack Overflow

OpenGL render to a OpenCV mat as texture and reuse it in OpenCV

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related