2

I am running a real-time process from the live media stream of the webcam, which requires the FPS to stay around 30FPS. getImageData was one of my main bottlenecks, it was requiring around 22ms to extract the ImageData object from the canvas, leaving me only 11ms to do my other operations.

Here is a simple snippet of what I had before

  video = document.createElement("video");
  inputCanvas = document.createElement("canvas");
  inputCanvas.width = width;
  inputCanvas.height = height;
  inputCanvasCtx = inputCanvas.getContext("2d");
  video.width = width;
  video.height = height;
  video.srcObject = stream;

// And using the requestVideoFrameCallback API, whenever a frame is presented, this callback is triggered
  function onFrame(now, metadata){

     inputCanvasCtx.drawImage(video, 0, 0);;

     // Get frame from our canvas.
     frame = inputCanvasCtx.getImageData(0,0,inputCanvas.width, inputCanvas.height);

     // Other computations
  }

After doing this simple change to my setup, without changing anything else I managed to extract the frame in 11ms, half the time it took before, and I cannot understand why. I only converted the regular canvas to an offscreencanvas in two lines.

  video = document.createElement("video");
  inputCanvas = document.createElement("canvas");
  inputCanvas.width = width;
  inputCanvas.height = height;
  offscreen = inputCanvas.transferControlToOffscreen(); // Changed line
  inputCanvasCtx = offscreen.getContext("2d"); // Changed line
  video.width = width;
  video.height = height;
  video.srcObject = stream;

I would appreciate any insight on this matter.

8
  • Weird, I do have the exact opposite results on my end. Using the OffscreenCanvas takes about x2 times as when using the <canvas>. However I get the same (poor) result as the offscreen when I append the <canvas> in the doc. And I get the same (good) perfs as the <canvas> when I use the willReadFrequently option to create the context. Given all these I'd personally think this is because they don't manage GPU acceleration the same way for OffscreenCanvas and <canvas> when there is a read-back to CPU, but I'm very surprised we have contradictory results. jsfiddle.net/hw1r35sy Commented Jun 30, 2022 at 2:03
  • @kaiido this is really puzzling. Did you make sure that hardware acceleration is on in your browser? And can I know your machine specifications? Commented Jun 30, 2022 at 14:42
  • 103.0.5060.53 (Official Build) (x86_64) and 105.0.5150.2 (Official Build) canary (x86_64) both on macOS, with or without "experimental web-features". And obviously with HWA on, otherwise there would be no variation at all. Do you confirm that in my test you get better results when ticking OffscreenCanvas? Commented Jul 1, 2022 at 2:13
  • @Kaiido I tried using "willReadFrequently " when creating the 2d context for the offscreencanvas . I noticed it reduces the performance back to 20ms from 10ms for getImageData (same value for the regular canvas). Commented Jul 1, 2022 at 13:38
  • With my fiddle, right? That's completely weird. willReadFrequently is supposed to prevent GPU acceleration, so that readbacks aren't needed since everything is always on the CPU. It makes no sense that you've got worst results (in my fiddle) when using this option, unless there is such a difference of performance between your GPU and your CPU that would make the CPU rendering actually slower than the GPU rendering + the readback. What when you change the size of the canvas? Commented Jul 1, 2022 at 13:48

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.