Streaming Learning Center Where Streaming Professionals Learn to Excel
Related Articles
VVC reached First Draft International Specification in July 2020. We’re about 18 months past that date and it’s useful to check in on VVC progress to date, including licensing, performance, chipset development, and trials.
Contents
VVC Patent Owners
Video quality is nice, but money makes the world go-’round, so let’s start here. Understand that video compression standards like VVC will have unique patents that relate to new features, but will also incorporate patents originated with previous standards, like HEVC and H.264. Table 1 is a list of patent owners that includes both HEVC and VVC patents from this article from March 2021.
Note that the authors updated the information later with VVC-related patent owners here and that there are prominent VVC-only patent owners not listed in Table 1. However, the combination of the HEVC and VVC patent contributions plus the percentage ownership in the first reference was more useful for this discussion, which is meant to be high-level, not definitive.
I did want to point out that Apple, though not listed in Table 1, owns about 350 VVC-related patents (see slide 42 from the downloaded presentation). This, plus other patents that were likely acquired by Apple (see here) means that Apple has a significant investment in VVC.
I know, Apple elbowed its way into the Alliance for Open Media, becoming a “founding” member 27 months after the group was formed, but it clearly isn’t drinking the Kool-Aid. As detailed on CanIUse, the lack of AV1 support on Safari/Mac, Safari iOS, and Apple TV is the most glaring gap in AV1’s impressive compatibility profile to date. At the same time, Apple is the only major platform that provides browser-based access to HEVC, AV1’s most relevant standards-based competitor.
There are also other AOMedia member companies prominent in table 1, including Tencent, Intel, Netflix, and Samsung. So, at least as it relates to all these companies, you shouldn’t construe AOMedia membership as anti-VVC.
While Table 1 is still visible, let’s make a couple of additional points. First, note the diversity and sizes of the businesses represented. Each of these obviously has a significant investment in VVC, and a significant incentive to make sure that it succeeds. As an example, Table 2 shows the global smartphone chipset market share from Q2 2020 to Q3 2021. As you can see, most of the names represented there are quite prominent in Table 1, particularly Qualcomm, Huawei, and MediaTek.
Given this data, you’d have to predict that all these companies will strongly consider adding VVC to future chipsets, which MediaTek has already done. More on all this in the implementation section below.
Second, if you add up all percentages, which include most, but not all, shown by iPlytics, they total over 100%. I’m not sure why, but I’m sure there’s a good explanation. Also note that of the patent owners committed to the two pools listed in the table, which I’ll discuss next, HEVC Advance has 24.38%, and MPEG LA has none. If you check the other iPlytics presentation, you’ll see several primarily VVC-only patent owners that aren’t listed in Table 1, but have joined one or the other pool, though the percentage of ownership not covered in Table 1 is relatively small.
Of course, over 100% of the patents shown in Table 1 are uncommitted to either pool, so it’s early days. Still, when you take a license in a pool, the royalty should relate to the IP rights delivered by the license. At this point in time, the Access Advance pool delivers a lot more IP rights which justifies a higher royalty.
So, let’s jump to the two pools.
VVC Patent Pools
Patent pools are exactly what they sound like; groups of patents owners join together to offer a combined license to their patents. Patent pools simplify accessing technology rights for technology implementors; rather than signing an individual license with each and every patent owner that contributed technology to a standard, the implementor can sign one license covering all patent owners in the pool. Pools only transfer the rights to use the patents within the pool, so if there are multiple pools, implementors will have to license from all the pools, plus reach an agreement with the patent owners that aren’t in a pool.
Though there was an effort to convince VVC patent owners to form one pool, ultimately two were created. The Access Advance pool launched first, with a royalty structure that depends upon multiple factors, including whether you’re in compliance with the license, whether you elect to show the Access Advance trademark, and whether you are licensing both VVC and HEVC from Access Advance. The least expensive rates are shown in Figure 3; note the categories, category royalties, category annual caps, and the overall cap.
Access Advance is also offering a Multi-Codec Bridging Agreement (“MCBA”) for companies that license both VVC and HEVC. According to the press release, companies who license VVC and HEVC, and sign the MCBA, “will enjoy a discounted royalty rate for their products that include both VVC and HEVC, paying the same royalty rate as they would pay for a product that includes only VVC, providing an effective 45% discount.” You can access a list of patent owners in the Access Advance VVC pool here.
MPEG LA announced its terms in January 2022 and list of patent owners here. Terms are shown in Figure 4 from here. As you can see, there are different royalties for hardware and paid software and free software like browsers. For those not conversant in Latin, de minimus means that below these quantities, no royalty is due. You can check the source article for more details on how the caps apply.
For perspective, many technologies like Wi-Fi and cellular standards have two or more patent pools and several significant patent owners like Qualcomm that prefer to license directly and not join a pool. So, what’s shaping up for VVC licensing is definitely more positive than negative. If anything, the pools have coalesced more quickly than with HEVC and with terms that should be more immediately tenable to potential implementors. While IP owners sitting on the sidelines does cause uncertainty, the fact that both pools have announced terms signals that VVC is now open for business.
OK, now you’re up to speed on what VVC costs; let’s cover the performance side.
VVC Performance
I evaluated Fraunhofer’s VVC codec for Streaming Media magazine here. The bottom line quality comparison is shown in Table 4. Readers should note the difference between a codec and a standard. As it relates to VVC, VVC is a standard and Fraunhofer’s VVenC is an implementation of that standard, just as the other codecs are implementations of those standards, whether HEVC, H.264, EVC, or AV1.
For these codecs, using the VMAF metric, I found that Fraunhofer’s implementation of VVC was about 5.55% more efficient than Libaom-AV1 and only 2.36% more efficient than the Main profile of EVC using the XEVE open-source encoder. As you’ll see in a moment, Moscow State University achieved much different results.
I tested encoding time on an HP workstation with a 3.4 GHz Intel i7-3770 CPU with 16GB of RAM and 4 cores and eight threads with HTT enabled. VVenC encoded in 66x real-time as compared to 33x real-time for x265 using the very slow preset, and 34x the Libaom-AV1 codec with a CPU used setting of 3. Encoding with x264 using the very slow preset was 7x real-time. So, VVenC was in the ballpark.
In playback tests on the same computer, Fraunhofer’s VVC decoder achieved 39 fps, compared to 357 for AV1 and 388 for HEVC. From my perspective, this puts VVC in the hardware codec category, which means that I would predict that most publishers wouldn’t use software playback on available platforms because of poor performance, battery life concerns, or both. Of course, VVC won’t play on smart TVs and similar devices without hardware support. Note that some implementation studies discussed below present vastly superior playback results that may cause many to rethink this conclusion. .
Moscow State VVC Quality Comparisons
In this MSU report, the Moscow State Team looked at multiple AV1, VVC, HEVC, and H.264 codecs; the table presented below shows the comparisons using the YUV 4:1:1 SSIM metric, with other comparisons available in the free report (and many more in the paid report). To explain the scoring, the MSU comparison sets the reference x265 codec at 100%, and then gauges the other codecs in comparison to that.
You’ll note that in MSU tests, VVenC performed only 2% better than x265, where I found VVenC 43% more efficient. Though our tests were completely different and I used VMAF instead of SSIM, this was still pretty surprising. I asked Fraunhofer about this discrepancy during my testing and they could offer no explanation.
MSU also found that the Huawei HW266 codec was 50% more efficient than the reference x265 implementation with the Alibaba S266_V1 VVC codec about 47% more efficient, and Tencent at 41%. The Huawei VVC codec was about 20% superior to the most efficient AV1 implementation, which was Tencent VAV1.
Though we got there in different ways, both studies show that VVC delivers substantial improvements over HEVC (43-59%) and significantly less benefit as compared to AV1 (5-20%).
VVC Developmental Activity
In January 2022, Gary Sullivan from Microsoft (and founding co-chairman of the Joint Video Experts Team that developed VVC) wrote a document entitled Deployment Status of the VVC standard detailing developmental activity surrounding VVC. It’s a treasure trove of information that you can download here. Most of what I’m presenting below is pulled from that document.
Hardware implementations are the longest lead time item and there is surprisingly little that’s been announced. Here’s what we know ’s coming:
- In November 2021, MediaTek announced the Pentonic 2000, which the company claimed “is the first commercial 8K TV chip with Versatile Video Coding (VVC) H.266 media support.” The release further claims that the chip will support Dolby Vision out of the box and that it expects smart TVs with the chip to launch in the global market in 2022.
- In July 2021, Allegro announced its “its AL-D320 video decoder semiconductor IP core supporting the latest Versatile Video Coding (VVC/H.266) format.” According to the release, the decoder is “is immediately available for integration by SoC vendors” so actual chips with the decoder won’t be available until after this deployment.
- The 8K Association spied a TCL Smart TV with VVC playback at NAB, though details like the decoder chipset were not available.
- Note that Qualcomm’s Snapdragon 8 Gen 2 chipset will likely support AV1 but there’s no mention of VVC.
MediaTek expects TVs with the Pentonic 2000 to hit the market in 2022, a prediction already met by TCL. Still, even if more models become available from more manufacturers, it will still take a several years for a critical mass of TVs with VVC support to attract independent publishers (e.g. those without VVC patents). If anything, it feels like VVC patent owners like Qualcomm, Broadcom, and Intel are really holding VVC back by not accelerating VVC support in their respective chipsets.
Here’s a table with the other VVC implementations listed in the Sullivan document.
Software Decoding of VVC
If VVC does require hardware for full-frame playback on computers, and for full-frame, battery-efficient playback on mobile devices, it pushes back adoption by 2-3 years. For this reason, trials relating to software playback are particularly interesting.
One study is authored by Kwai, Inc, the creator of the Kuaishou video-sharing mobile app. Here the researchers focused on the Android platform since 80-90% of views on their platform are on Android phones. Testing with 1280×720 videos uploaded by app users, the researchers achieved up to 97.49 fps running on a single thread in Android, and 249 fps on a single thread on an iPhone 12.
Figure 7. Impressive playback performance on Android phones.
Researchers from ByteDance tested their decoder with mixed GPU and CPU decoding and achieved faster than real-time playback of 4K video on an M1-based Mac and an iPhone 13 (Figure 8).
What’s interesting about these results is how the use of VVC by popular platforms such as TikTok can accelerate hardware support for the codec, particularly if the publisher ties some enhancement, like higher-resolution videos, to VVC codec support. For example, YouTube finally convinced Apple to support VP9 by encoding all streams larger than 1080p in VP9 or AV1.
While I remain skeptical that any independent publisher would deploy VVC for software-only playback on battery-powered devices, these results are impressive. You’d expect any publisher included in Table 1 to be more aggressive with VVC deployment, much the way YouTube, Meta, and Netflix were with AV1.
VVC Browser and TV Support
[Author’s note: In October 2022, Google Chrome initiated playback support for HEVC. What this means for VVC remains to be seen, but certainly, the picture is more positive for VVC support in Chrome than it was before this move.]
The big problem for software playback of VVC is likely to be software support. HEVC never broke through for browser-based playback because it’s not supported in Chrome, Firefox, or Edge on the Windows or Android platforms. There will be workarounds publishers can pursue, like using a specific video player, but then royalties become an issue. If you have a mobile app you can use whatever codec you like, but this excludes all but the largest streaming companies, and again, royalties come into play.
After nearly 8 years in the market, HEVC has failed to carve a presence for browser-based viewing but succeeded wonderfully for premium content displayed in the living room, and that’s likely where VVC will also excel, though it will take 2-3 years. In that vein, note that VVC (and LCEVC) were included in Brazil’s recent TV 3.0 project and VVC (along with AV1 and AVS3) will be included in an upcoming DVB specification for European TV sets.
For the sake of completeness, I should note that VVC also brings unique enhancements to VR and AR applications, which will start to come together over the next few years.
The net/net for me is this:
- VVC contributors include many very significant companies with significant investment in VVC, and a significant incentive for it to succeed. Like the members of AOMedia, they also yield significant market power to help VVC get deployed and succeed.
- VVC should be around 50% more efficient than HEVC and 10 – 20% more efficient than AV1.
- VVC will likely see early adaption among stakeholder companies like Kwai, ByteDance, and Tencent, much like we saw with YouTube, Meta, and Netflix for AV1.
- Within traditional streaming markets, VVC will likely mimic HEVC with near-ubiquitous penetration in the living room but little applicability for browser-based playback. That said, it will likely take 2-3 years for the installed base of TV sets with VVC playback to attract independent publishers.
- VVC appears to have an early technological lead for VR and AR.
As a final note, we talk about adding another codec to the distribution mix as if it was as simple as flipping a switch. It’s not, and plummeting bandwidth costs have significantly diminished the economic benefit of supporting more efficient codecs. See, for example, the article entitled Computing Breakeven on Codec Developments.
For this reason, most new codec deployments seem to occur when they enable support of new markets or dramatically changing requirements, like 4K and HDR with HEVC. Will premium content publishers switch from HEVC to VVC to harvest the 50% bandwidth savings? I’m skeptical, at least over the next 2-3 years. On the other hand, if AR/VR become a thing, and VVC-support is the price of admission, we could see adoption and deployment en mass.
For the sake of completion, I should note that 20% of respondents of the Bitmovin Video Developer report planned to support VVC in 2022, slightly behind AV1 at 22%, and HEVC at 25% (Figure 9). Another report from Rethink Technology, was also quite bullish on VVC, starting in mid 2023. As far as the Bitmovin numbers, the VVC predictions might make sense for the companies in the streaming media ecosystem, who need to ramp up to encode, distribute, monitor, or playback VVC content, but feel quite aggressive for actual content publishers.
