Streaming Learning Center Where Streaming Professionals Learn to Excel
Related Articles
While few companies of any kind are actually making money from HEVC (H.265) today, the successor to H.264 will become increasingly important during the next 2–3 years, perhaps even earlier in some markets for some producers. So understanding the current status of the technology and how to encode and potentially deploy HEVC in the near term is very relevant for most streaming media producers. Accordingly, in this article, I’ll review the state of HEVC and take a high-level look at the first generation of HEVC encoders.
Let’s start with where we are and what we know.
Contents
Current Technology Status
As an overview, HEVC offers two key advantages over the reigning codec king, H.264: quality equivalent to H.264 at half the data rate and the ability to deliver 4K video, or Ultra HD video (UHD). The first advantage can be used with existing HD and lower-resolution content, whether to save bandwidth costs or to deliver higher-resolution video over bandwidth-restricted connections. As such, it potentially impacts nearly every streaming video producer and all companies in the distribution food chain.
The second advantage, Ultra HD video, is totally dependent upon a number of factors that most streaming producers can’t control, from the availability of UHD content to how quickly consumers start buying 4K TV sets. The ability to distribute UHD video will beneficially impact only a small percentage of streaming producers in the foreseeable future. For this reason, I’ll focus this article on HEVC’s initial advantage: essentially H.264 quality at half the bitrate. So the first topic to address is whether HEVC actually delivers on this promise.
HEVC Quality
That turns out to be a pretty interesting question without a simple answer. Before addressing this question, you should understand a couple of key points. First, few, if any, vendors have incorporated all of the encoding algorithms and techniques available in the HEVC spec into their existing HEVC codec. Second, even once the specification is fully implemented, codec quality tends to improve over time, as the engineers and programmers optimize their codec-related algorithms and techniques.
So, while H.264 has probably reached most of its potential quality-wise, HEVC is just getting started. Therefore, the HEVC quality we see today doesn’t represent HEVC’s full potential, and certainly not the quality it should achieve when you actually implement it down the road.
That said, where are we today? This depends. I looked at three test cases — 640x360x29.97@700Kbps (H.264)/350Kbps (HEVC), 1280x720x29.97@400Kbps/800Kbps, and 1920x1080x29.97@2Mbps/4Mbps. For each of the technologies, I evaluated the quality of full-motion playback and multiple still-frame comparisons.
In the 640×360 comparison, HEVC did not deliver equal quality to H.264 at half the data rate. Specifically, H.264 retained more detail and showed very few motion-related artifacts, while HEVC exhibited significant mosquitoes, or hovering artifacts, around large regions of moving content, such as a banjo player or dancer. In the 720p comparisons, HEVC frequently, but not always, showed better detail retention than H.264 at half the data rate. It also exhibited more motion artifacts, again in the form of mosquitoes.
The most favorable test case was the last, at 1080p. Here, the 2Mbps HEVC file won most of the still image comparisons while the mosquitoes evident in the first two comparisons were no longer obvious, though this test involved a different test clip than the first two. In some panning shots, however, the 2Mbps HEVC clip showed some tearing, which was offset by judder in the 4Mbps H.264 clip (Figure 1).
The obvious lesson is that HEVC will deliver more bandwidth savings at larger resolutions, and you should expect this dynamic to continue. Looking forward, during the next 12 months, as vendors fully implement the HEVC feature set and continue to improve their code, I would guess that 720p will be the tipping point at which HEVC will deliver equivalent quality at half the bitrate of H.264. At smaller resolutions, you should expect less savings, with increasing savings as resolutions increase.
When you make your own HEVC/H.264 comparisons, remember to assess both still-frame and real-time playback quality. Judging from the admittedly small sample of test clips that I reviewed, low bitrate HEVC streams showed more motion-related artifacts than H.264. Perhaps this relates to my test clip, which involve lots of stage performances by musicians and dancers. In particular, the mosquitoes surrounding these moving performances were the most noticeable artifact exhibited in the HEVC encoded clips. If you often encode similar content, make sure your test comparisons contain this type of footage.
Where Will It Play?
Quality is great, of course, but if the video won’t play at full-frame rates, no one will care. Let’s consider computers and notebooks first, and then segue into mobile. Table 1 shows the CPU required to play back H.264 and HEVC video at 720p and 1080p resolutions on some of the lower-power computers in my office. Note that I played all files using the VLC Player.
As you can see, playing back the 720p HEVC videos was well within the reach of all tested computers. Since Intel’s Core 2 Duo started shipping right around the time I acquired the Precision 390 in 2006, this means that there’s a huge installed base of computers capable of 720p HEVC playback. On the other hand, it looks like a four-core computer will be necessary to comfortably play back 1080p video, which eliminates much of that huge installed base. That said, judging from the tests and surveys that I’ve performed, very few producers outside of YouTube stream 1080p video to computers, so the inability to play back 1080p may not be a significant negative for general-purpose streaming.
Issues are a bit more contentious in the mobile space. In a frequently cited report titled “HEVC Decoding in Consumer Devices,” senior analyst Michelle Abraham from the Multimedia Research Group estimated that the number of consumer devices shipped in 2011 and 2012 that were capable of HEVC playback with a software upgrade totaled around 1.4 billion, with more than a billion more expected to be sold in 2013.
While Frost & Sullivan analyst Avni Rambhia doesn’t necessarily disagree with those numbers, she identified a big “yeah, but” in her Streaming Media West 2013 presentation, “Cutting Through the Hype of HEVC (H.265).” Specifically, not counting PCs, Rambhia estimates that more than 1 billion HEVC-capable devices will ship in 2014 (Figure 2). However, since fewer than 5 million of those will include HEVC playback hardware, Rambhia feels that the power consumed during HEVC playback makes HEVC untenable.
In her presentation handout, Rambhia states, “Mobile use of HEVC hinges upon silicon. If you are not truly mobile, then HEVC is not a game changer for HD content delivery. The 1 billion capable device statistic is true, but essentially irrelevant to HEVC uptake in the short term.” Later in the presentation, Rambhia predicts, “HEVC and AVC will run as parallel, complementary technologies until late 2014. HEVC will gain momentum starting in 2015, particularly with telepresence, mobile OTT, and UltraHD. AVC will endure as a mainstream format at least until 2018, and remain a significant format well beyond 2020.”
Today, many producers prepare separate streams for mobile and desktop delivery, so it’s not unreasonable to anticipate that some producers will encode with HEVC to harvest bandwidth savings in streams delivered to computers while continuing to use H.264 to distribute to mobile. However, If you’re hoping that HEVC will enable the delivery of HD video to mobile devices, you may have to wait until hardware-accelerated playback, or much higher capacity batteries, are the norm.
What’s the Status of Playback?
Another obstacle to using HEVC for general-purpose streaming is the lack of a pervasive player. For example, H.264 was not widely used for streaming until Adobe incorporated H.264 playback into the Flash Player. Though Rovi boldly shipped DivX 10 with HEVC encode/decode in September 2013, with VideoLan adding HEVC to the VLC Player 2 months later, even combined, these players don’t offer the penetration necessary for publishers to deploy HEVC for general-purpose streaming.
Battery life issues aside, HEVC won’t be relevant in the mobile space until playback is incorporated into iOS and Android. Sure, in both the computer and mobile space, you could build a custom player that supplies HEVC decode, but that alternative is unavailable to all but the largest producers. Basically, general-purpose HEVC deployments are on hold until Adobe, Apple, and Google add HEVC playback to their respective players and operating systems.
What About Royalties?
In January, 2014, MPEG LA, the licensing group that administers the largest H.264 patent group, and has formed a similar group for HEVC, announced its proposed royalty policy for HEVC. Briefly, the proposal is publisher friendly, with no proposed royalty on HEVC-encoded video, even that distributed via pay-per-view or subscription. In contrast, after certain minimums, there is a content-related royalty on H.264.
On the other hand, the terms or more expensive for encoder/decoder vendors. Both H.264 and HEVC excepted the first 100,000 units, and both charge $0.20 per unit after the first 100,000. However, with H.264, the royalty dropped to $0.10 after 5 million units and is capped at $6.5 million through 2115. The HEVC royalty never drops, and is capped at $25 million.
The MPEG LA announcement made clear that not all companies with HEVC-related intellectual property joined the group, so there is a possibility that these terms may change, or that other IP owners may attempt to enforce their own royalties. Still, these terms are likely close to final for the MPEG LA group, and you’d have to assume that any proposed third-party royalties would be in the same ballpark or less. MPEG LA hopes to announce final terms in early 2014.
Even without a known royalty picture, virtually all mainstream encoding vendors had announced plans to integrate HEVC encoding into their product lines (more on this later). On the playback side, until HEVC encoding becomes mainstream, there’s no reason for Adobe, Apple, and Google to pay up to $25 million per annum to integrate HEVC playback into Flash, iOS, and Android. Several large publishers, most notably Netflix, have announced plans to distribute content in HEVC format, though without Flash, iOS or Android support, most smaller publishers have no incentive to start encoding their content into HEVC.
In this regard, it’s instructive to note that while it feels like HEVC deployment is dragging, HEVC adaption in the streaming space is proceeding much more quickly than H.264. Specifically, the H.264 spec was approved in March 2003, with royalty policies announced in November 2003. In a move that seems prescient in retrospect, Apple integrated H.264 into QuickTime in April 2005, and, of course, integrated H.264 playback into all video-capable iOS devices. However, H.264 didn’t become truly useful for general-purpose streaming until Adobe integrated H.264 playback into Flash in March 2008, five years after the spec was announced.
The HEVC spec was approved in January 2013, and it’s clearly taking longer than with H.264 for royalty terms to become known. However, it’s unlikely that streaming producers will have to wait five years to start using HEVC as they did with H.264.
The Encoding Side
What do we know about HEVC encoding so far? At a high level, we know that HEVC will have the same type of Profile/Level gradations exposed in H.264, as shown in Table 2 from Wikipedia. The HEVC specification finalized in January 2013 defined two video profiles, Main and Main 10, and one still-image profile, Main Still Picture. In August 2013, the five additional range extensions shown in the table were proposed.
As with H.264, as profiles integrate more advanced algorithms and features, the CPU, memory, and power required to play back the video increases. In this manner, Profiles allow hardware developers to choose the level of HEVC support in their devices based upon their own design goals and cost structures. Then, video producers can customize videos for those targets by encoding with that profile.
How will HEVC profiles be incorporated into the typical encoding interface? Taking a step back, the typical encoding interface must include multiple configuration options irrespective of the codec actually used to compress the video. These include the following:
- Resolution, pixel aspect ratio, and frame rate
- Bitrate targets and control technique (VBR, CBR)
- Group of Picture (GOP) controls, including I- and B-frame interval, the number of reference frames, and scene detection options.
- Some level of buffer control
So all of these familiar options will also be present in all HEVC encoding tools. Beyond these, most HEVC encoders will also enable HEVC profile selection, some general quality parameters, and perhaps some direct access to HEVC-specific encoding parameters.
To get a feel for what the typical HEVC encoding interface will look like, I asked a number of vendors for access to their encoders. While I did encode files using several different codecs and encoding tools, it’s premature to discuss comparative quality, though quality did vary noticeably from encoder to encoder. None of these products are actually shipping in final form, so quality comparisons at this point would be akin to analyzing the performance of football teams on the first day of training camp, which is neither useful nor relevant.
I was more interested in what encoding HEVC was going to look like from a user interface perspective, so that’s where I focused my attention, and that’s what I’ll describe going forward. As a preview, just like we saw with H.264 encoding, the configuration options exposed for HEVC encoding will vary dramatically from vendor to vendor. I’ll start with HEVC codec vendors, since they tend to expose more parameters in their encoding tools.
Codec Vendors
MainConcept is a subsidiary of Rovi (formerly Macrovision) and is primarily a codec vendor whose H.264 codec is used in a range of products sold by Adobe, Sorenson Media, and Telestream. In addition to developing their HEVC codec for licensing to third parties, the MainConcept codec will also be deployed in products sold by Rovi and other Rovi subsidiaries. Figure 3 shows the advanced dialog from Rovi’s TotalCode Studio encoder. Note that for presentation purposes, I cut the vertical screen in half and placed the controls side by side.
The screen contains a mix of old and new. You see the HEVC profile and level controls on the upper left, with familiar GOP Structure controls immediately below. The bottom left contains configuration options for coding tree units, a key HEVC innovation that is similar to macroblock units. Motion estimation controls are on the right, which manage the trade-off between searching precision and encoding time. Beneath that is the Deblocking filter, which is similar to that used in H.264 processing but with improved support for parallel processing.
Sample Adaptive Offset (SAO) is another HEVC innovation that should deliver increased quality and reduced banding and ringing artifacts. As the name suggests, Wavefront Parallel Processing is a more efficient technique for enabling parallelism than the threading technique available in H.264. Though documentation for these controls was not available when I looked at the program, MainConcept will document how to use the new features when the product ships in early 2014. The company also indicated that the controls shown in Figure 3 are advanced controls that many viewers don’t access, preferring to rely upon the default settings that MainConcept includes in its presets. Note that the initial version of TotalCode Studio will lack support for the Main 10 Profile and several other minor components of HEVC, which are all on the road map for release in 2014.
Ittiam Systems is an embedded software and systems design company that is offering an HEVC codec. Ittiam provided several sample files that were encoded via command line because no GUI was yet available. While most HEVC-related encoding parameters were available in the command line string, Ittiam did not yet support SAO and multiple tiles, though these will be introduced in 2014.
At the other end of the spectrum, Elemental Technologies takes a minimalist approach when it comes to exposing extensive codec-related encoding options to its users. This is shown in Figure 4, which contains all the HEVC-specific encoding options available in the Elemental Server real-time encoder. Beyond choosing the profile and level, you can select the number of slices, the adaptive quantization technique, enable Flicker Reduction, and via the Density versus Quality slider, manage the trade-off between encoding speed and quality.
In general, Elemental seems to prefer to optimize settings behind the scenes and present the user with a limited number of relatively well-defined parameters that enable critical decisions such as the quality/encoding speed trade-off. Certainly this works for H.264, where Elemental ranks very close to the top while presenting a set of configurations very close to those shown in Figure 5. This may be frustrating if you’re one of the few encoding professionals who can truly tune the extensive encoding parameters offered by some vendors to your advantage. For most users, however, simpler is always better so long as the quality is very high. Perhaps more to the point, if you’re an existing Elemental customer, the learning curve for encoding HEVC will be very short.
Harmonic takes a two-pronged approach to the HEVC configuration issue. In ProMedia Xpress, a service of its Workflow System (WFS) product (Figure 5), you can choose the profile, level, and little else. At the other end of the spectrum, Harmonic will expose many more HEVC-related configuration options in tools such as ProMedia Carbon that you can use to create HEVC presets deployed in Harmonic WFS. Specifically, Harmonic expects the HEVC-related configuration options available in Carbon to equal those exposed for H.264 in the same program, which are very extensive.
What’s the net/net from an encoding UI perspective? Unlike the changeover from VP6 to H.264, which involved many new concepts, encoding HEVC will be very similar to H.264 with the exception of a handful of options that some encoders will optimize behind the scenes and won’t make available to their users.
