This week I’ve been messing around with the HTML5 video element and have discovered you can do some pretty cool stuff. For instance, Mozilla shows how to do greenscreen image replacement while a video is played. This is very cool but there’s an annoying intermediate step required: since you cannot directly fetch raw frame image data from a video element, you must write each frame to an intermediate canvas element and then use its getImageData method to get raw image data you then transform and write to the final canvas element.

I brought this up in an email to the HTML5 Doctors last night. Remy suggested that I write up a blog post with some benchmarks, so here are some quick findings. You can find my code on GitHub. In the example I take a video and invert all its colors, frame-by-frame.

Here’s my html:

And my Javascript:

Using Chrome 5.0.375.55 on OS X and its developer tools, I took a heap snapshot after loading benchmark.html. There were 2171 code (what, I don’t know) and 12463 objects, taking up 464.70 kb and 866.97 kb respectively. I then started the profiler and executed the following Javascript: var video = document.getElementById("video); video.play();. Once the video appeared to have stopped (confirmed by video.ended) I stopped the profiler and took another heap snapshot. This time there were 2202 code and 13037 objects taking up 471.58 kb and 922.11 kb. Not surprisingly the profile shows that processFrame() takes up the vast majority of execution time (77.63% self, 83.65% total).

This all leads me to believe that, at least for smaller videos like the 480 x 280 px video I used from Apple’s documentation, there are not significant performance issues when transforming videos frame-by-frame in Javascript.

That being said, I still find the process ineffiecent and would like to be able to get a sort of FrameData object directly from video elements, just like you can get an ImageData object from canvas elements. This would mean I would only need to use one canvas element (canvas2 in my example), instead of two.