Vulkan: how to safely replace a commandBuffer

Question

When my scene updates (e.g a new model introduced), I re-record my draw commmand buffers into new command buffers (tempBuffer) using the function buildCommandBuffers.

I have an array of drawCmdBuffers which are used in the main draw routine.

To complicate matters slightly, my buildCommandBuffers function could be getting called from a timer in a separate thread, so the render function is potentially still looping while this update occurs. This is because my scene update can take a while and I need to lock the scene memory while I create the command buffer (this is a separate problem.. but every mesh added triggers a scene update and I need to be able to buffer those updates).

I have a fence for each command buffer, so once I've recorded into my "tempBuffer", I do the following:

        VK_CHECK_RESULT(vkEndCommandBuffer(tempBuffers[i]));        
        // wait for this command buffer to finish (we might be re-recording)
        if (imagesInFlight[_image_index] != VK_NULL_HANDLE) 
        {
            vkWaitForFences(device, 1, &imagesInFlight[_image_index], VK_TRUE, UINT64_MAX);
        }
        // destroy in use command buffer
        
        vkFreeCommandBuffers(device, cmdPool, 1, &drawCmdBuffers[i]);
        drawCmdBuffers[i] = tempBuffers[i];

Is this safe? Or is there something I can do to stop access to drawCmdBuffers[i] while I'm updating it?

It looks like you have a fence per frame, not per command buffer, is that right? — Jesse Hall
– Jesse Hall, Commented Jan 14, 2021 at 12:12
I've posted an answer below, but I'd also recommand double-checking that you need to reuse command buffers in this way. Most Vulkan apps, including all the "professional" ones I'm aware of, just re-record command buffers each frame. Unless recording requires a lot of processing in your code, the actual recording calls are cheap enough that there's not a lot of benefit to reusing command buffers. And reuse can add a lot of complexity. — Jesse Hall
– Jesse Hall, Commented Jan 14, 2021 at 12:28
recording is really slow in my application, there are tens of thousands of objects. Re-recording each frame is not practical. I do need to speed the recording up but that's another problem for another day! — mike
– mike, Commented Jan 14, 2021 at 13:36

Jesse Hall · Accepted Answer · 2021-01-14 12:26:59Z

It looks like you need CPU-CPU synchronization (typically a mutex) around access to the imagesInFlight[] array and the drawCommandBuffers[] or the individual elements within it. Otherwise you could be waiting for the fence for the most recent frame, while the other thread is submitting command buffers for the next frame -- in which case even when your wait completes, the GPU is still going to be using the command buffer.

Assuming you really need to reuse command buffers rather than just re-recording them each frame, one approach would be to have the buildCommandBuffers thread just build a list of changes to drawCmdBuffers. Something like:

Render thread:

for each frame:
  acquire mutex for command buffer edit list
  for (i, cmdbuf) in command buffer edit list:
    push drawCmdBuffers[i] and most recent frame fence to a deferred-destroy queue
    drawCmdBuffers[i] = cmdbuf;
  clear edit list
  release mutex
  submit render commands for frame, using drawCmdBuffers

Update thread:

on_timer:
  acquire mutex for command buffer edit list
  while fence at front of deferred-destroy queue has signaled:
    free command buffer at front of queue
    pop queue
  release mutex
  for each command buffer that needs to be replaced:
    record command buffer
    acquire mutex for command buffer edit list
    append (cmdbuf_index, cmdbuf) to edit list
    release mutex

That way only the render thread accesses drawCmdBuffers directly, so it doesn't need to be protected. You just have a queue of edits going from the timer thread to render thread, and a queue of command buffers to free going from the render thread to the timer thread (or to some other thread which actually waits on the fences, instead of just polling them on each timer). At any given moment (except briefly while the mutex is held), a command buffer handle is in only one of these lists.

Collectives™ on Stack Overflow

Vulkan: how to safely replace a commandBuffer

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related