Browser Video: Codecs, Formats & Hardware Acceleration

Web video is ubiquitous. We take it for granted that browsers play video in high resolution, over any connection, on any device. Behind the scenes, a complex machinery of video formats, codecs, and GPU acceleration techniques is at work to make it all happen. This post explains what is what.

Video Container Formats

Video containers store audio and video tracks, which may be encoded in a variety of codecs. Two video container formats are being used on the web today: MP4 and WebM.

MP4

MP4 contains audio encoded in AAC or MP3 and video encoded in H.264 (AVC) or H.265 (HEVC). The MP4 format is not royalty-free.

MP4 containers with H.264 video are supported in basically any browser:

MP4 containers with H.265 video (HEVC), on the other hand, are only supported in Edge and Internet Explorer, and only on devices that can decode H.265 in hardware. The reason is simple: Edge and IE do not decode H.265 by themselves, so Microsoft does not have to pay licensing fees. Edge/IE pass the video stream for decoding to the OS which in turn sends it to the GPU. Licensing costs are paid by the GPU vendor.

WebM

WebM contains audio encoded in Ogg Vorbis and video encoded in VP8 or VP9. The WebM format is royalty-free.

WebM containers are supported in Chrome and newer versions of Edge and Firefox only. Internet Explorer does not support WebM.

Hardware-Accelerated Video Decoding

GPUs have dedicated units that are much more efficient at decoding video than general-purpose CPUs which lack similar capabilities. However, for hardware decoding to work, all the components involved need to work hand-in-hand:

• The GPU’s decoder needs to support the video properties (codec, 4K resolution, HDR)
• The graphics driver needs to make the GPU’s capabilities available to the OS
• The operating system needs an abstraction layer so that applications do not have to deal with individual vendor driver APIs
• The application (the browser) needs to make use of the OS’ decoding API

DirectX Video Acceleration API 2.0

The Windows API of choice for accessing the machine’s hardware video decoding units is called DirectX Video Acceleration 2.0. DXVA is a little-known API; not many tools exist for exploring it. One of the few is DXVA Checker. It shows the video capabilities reported by the graphics drivers.

The screenshot below depicts the capabilities of the integrated Intel HD Graphics 620, built into the Kaby Lake i7-7500 CPU. Video decoding is supported for H.264, H.265 (HEVC) and VP9 (all at least up to 4K).

H.264 Hardware Acceleration

Hardware-accelerated decoding of H.264 (AVC) video is available on all CPUs/GPUs sold in the past 4+ years. 4K resolution, however, is only supported by newer hardware sold in the past 2+ years, approximately.

H.265 Hardware Acceleration

Hardware-accelerated decoding of H.265 (HEVC) video is available only on newer CPUs/GPUs. More specifically, at least the following are required:

• Intel Skylake
• Nvidia Maxwell (later models) or Pascal
• AMD Fiji or Polaris

VP9 Hardware Acceleration

Hardware-accelerated decoding of VP9 video is available only on the newest generations of CPUs/GPUs. At least:

• Intel Kaby Lake
• Nvidia Maxwell (later models) or Pascal
• AMD Polaris

Checking the Browser’s Acceleration Status

Chrome reveals which video codecs are hardware-accelerated on the current platform at the special URL chrome://gpu.

The screenshot above was taken on a device with the Intel Kaby Lake CPU i7-7500U. Hardware decode support for VP9 is available (shown as VPx).

Firefox provides some information at about:support, but does not show which codecs are accelerated. There is no known way to determine the acceleration status for Edge or IE.

Codecs Used by Video Sites

Almost all video websites use the H.264 codec, and for a very simple reason: H.264 ist most broadly supported in browsers and on mobile devices.

There is just one exception: YouTube, when accessed from Chrome or Firefox, serves VP9 encoded video.

That brings us to an important caveat: YouTube videos in Chrome or Firefox are inefficiently decoded in software unless the playback device is equipped with one of the newer GPU models.