The Four Pillars of Video Encoding

Four pillars of video encoding

Figure 1. The four critical aspects to consider before video encoding.

Executive Summary: While video quality is always top of mind, consider deliverability, compatibility, and playability before finalizing your video encoding presets.

Too often we look at video encoding with a single point of focus; achieving the best possible quality for the lowest possible data rate. However, as I was reminded when I spoke on a panel at Streaming Media East with several encoding pros from Beamr, Brightcove, Verizon, and Yahoo, achieving the best possible file at the lowest possible data rate is useless if you can’t deliver the file, or play it once delivered. It was a great reminder that whenever you encode a file, you have four major concerns, quality, compatibility, deliverability and playability. In this brief article, I’ll define each term and suggest some tools you can use to help measure performance in each.

Quality

Achieving the best possible quality for each bitrate is certainly the most obvious goal for compressionists, and there are two sides to this equation, quality and file size. Regarding quality, I’m a big user of objective video quality metrics, which attempt to mathematically compute how real world viewers would subjectively rate the videos.

The tool I use the most is the Moscow University Video Quality Measurement Tool (VQMT), which can compute multiple quality metrics like Peak Signal-to-Noise Ratio (PSNR), Structured Similarity Index (SSIM), and Video Quality Metric (VQM). In addition to producing a mathematical score, the program can compare two files and display the results visualization graph shown below, which tracks the comparative score over the duration of the file and lets you compare frames from the original file to the encoded files by clicking Show frame on the bottom right. You can read my review of the tool here.

Figure 2. The Results Visualization screen of the Moscow University VQMT.

Another tool I use frequently for quality comparisons the SSIMPlus Quality of Experience Monitor (SQM). SQM has two unique and useful features. First, it uses the SSIMPlus algorithm to rank videos, which produces a score that purports to directly correlate with viewer’s subjective perception; 80-100 scores predict an excellent rating, 60-80 predict a good rating, and so on. In addition, the tool can predict subjective ratings on a variety of platforms, from smartphone to 4K TV. Here’s my review of SQM.

The other aspect of the quality is file size. Don’t assume that your encoder is delivering the selected data rate unless you’ve actually checked. When I’m working with test files, I calculate the target file size, which is seconds x (video + audio bitrates)/8 (for the bits to byte conversion) and verify accuracy within 5% or so. I also use the tool Bitrate Viewer, shown below, which can analyze H.264 files and report the data rate.

Note that you can’t test a single file and feel comfortable with the result. For example, while a talking head video might compress accurately, an action video might stress the encoder and deliver a much higher bitrate stream. At the other end of the spectrum, synthetic videos like screencams or PowerPoint tutorials are often way too low, which can lead to degraded quality video. So test multiple times, checking all types of videos that you typically encode.

Deliverability

The concept of deliverability relates to the ability to smoothly deliver the file, which is particularly important over constrained connections like cellular. This involves the bitrate control technique used to encode your file (constant vs. variable bitrate encoding), which I detail in a long discussion here, or capped CRF, as shown in Figure 3. Though the average bitrate of the file shown in Figure 3 is 509 kbps, the peak bitrate of 1990 kbps is nearly four times higher, which definitely could cause deliverability issues over constrained connections.

Figure 3. Bitrate Viewer showing an average bitrate of 509 kbps and a peak of 1990 kbps.

No matter which bitrate control technique you deploy, you should visualize the bitstream with a tool like Bitrate Viewer. Obviously, this is particularly useful if you’re experiencing stoppages during playback and want to learn why. You can learn more about where to get Bitrate Viewer and how to use it in a video tutorial located here. Another tool you can use for this is Telestream Switch, which gives a lot more detail about the encoded file, though you’ll need to spend $300 to get the Pro version for this feature.

Compatibility

Compatibility relates to the player’s ability to load and play the file. Typically, the biggest issue here is the H.264 (or soon HEVC) profile used for the video. When encoding for computers and OTT, you can use the High profile, which delivers the best quality. When encoding for mobile, you have to take into account that older, slower devices can only play the Baseline profile. Most iOS producers use the encoding ladder recommended in Apple Tech Note TN2224, shown below, as a starting point for this. 

Table 1. The compatibility matrix presented in Apple’s Technote TN2224.

Note that the Android operating system can only decode Baseline-encoded video. While this is usually supplemented by hardware-assisted playback from the device manufacturer, this isn’t always the case. My Nexus 10 Android tablet, for example, will only play Baseline-encoded video. For this reason, some producers encode all streams targeted towards mobile Android devices using the Baseline profile.

Playability

The fourth pillar, playability, has two related components; the CPU required to playback the file, and the battery life that playback consumes. Playability is obviously critical with mobile devices, but is also an issue with desktops for producers distributing in 4K or with UHD codecs like HEVC and VP9.

When recommending an encoding configuration for mobile, always check the CPU required to playback that stream. On the Android platform, I use an app named CPU Monitor Advanced Pro (Figure 4). If I see a number like 100%, as shown on the top graph in Figure 4, I know that frame rate will likely be less than 100%, and that battery life will suffer. On iOS, I use the System Status app to check CPU utilization. On Windows computers I use Performance Monitor, while on the Mac platform I use Activity Monitor.

Figure 4. Android CPU utilization as shown in CPU Monitor Advanced Pro.

Interestingly, it was the playability aspects that the two gents from Verizon and Yahoo claimed they spent the most time debugging, particularly with mobile. This will become an issue on desktops and notebooks when producers start deploying HEVC, VP9, or the AV1 codec from the Alliance for Open Media.

Video Encoding Recommendations

The bottom line is that before deploying a new encoding configuration, you should:

1.         Verify output quality both objectively and subjectively

2.         Verify that the encoder is meeting the target bitrate with a variety of files.

3.         Check the bitrate profile with a tool like Bitrate Viewer or Switch.

4.         Verify playback compatibility by checking the requirements of each target platform.

5.         Check playback CPU utilization on the lower end of your target platforms to ensure playback at full frame rate and that your files don’t consume too much power.

Additional Resources

Encoding for Multiple Screen Delivery, Udemy ($35)

Producing Streaming Video for Multiple Screen Delivery, Jan Ozer (on Amazon)

About the Streaming Learning Center: The Streaming Learning Center is a premiere resource for companies seeking advice or training on cloud or on-premise encoder selection, preset creation, video quality optimization, adaptive bitrate distribution, and associated topics. Please contact Jan Ozer at [email protected] for more information about these services.

About Jan Ozer

Avatar photo
I help companies train new technical hires in streaming media-related positions; I also help companies optimize their codec selections and encoding stacks and evaluate new encoders and codecs. I am a contributing editor to Streaming Media Magazine, writing about codecs and encoding tools. I have written multiple authoritative books on video encoding, including Video Encoding by the Numbers: Eliminate the Guesswork from your Streaming Video (https://amzn.to/3kV6R1j) and Learn to Produce Video with FFmpeg: In Thirty Minutes or Less (https://amzn.to/3ZJih7e). I have multiple courses relating to streaming media production, all available at https://bit.ly/slc_courses. I currently work as www.netint.com as a Senior Director in Marketing.

Check Also

Single-Pass vs Two-Pass VBR: Which is Better?

Let’s start this article with a quiz regarding how the quality and encoding speed of …

My FFmpeg-Related New Year’s Resolution: Document Before I Test

My typical workflow for testing-related articles and reports is to create the command string, run …

Transcoding UGC: ASICs Are the Lowest Cost Option

The predominant use for ASIC-based transcoders like NETINT’s Quadra Video Processing Unit (VPU) has been …

Leave a Reply

Your email address will not be published. Required fields are marked *