Streaming Learning Center Where Streaming Professionals Learn to Excel
Related Articles
Microsoft recently open-sourced the Machine Learning Video Codec (MLVC) under the MIT License, publishing the codebase and model weights on GitHub, along with a technical announcement on the Microsoft Tech Community blog. The research paper, “MLVC: A Multi-platform Learned Video Codec for Real-World Deployment,” details the underlying architecture and makes this bold claim.
MLVC is the first neural video codec to combine competitive compression performance, real-time speed, and cross-platform robustness across diverse consumer devices, making it suitable for widespread deployment.
While marketed as an NPU codec, MLVC is actually a hybrid CPU/NPU design, with a C++ entropy coding stage on the CPU and neural inference on the NPU. This was necessary to enable cross-NPU compatibility. As you’ll see, this is irrelevant to its initial target market of real-time communications. Down the road, it may impede MLVC’s deployment in a living-room OTT environment where devices don’t have sufficiently capable CPUs.
From my perspective, this release is significant because MLVC is the first neural video codec from a major platform vendor to move beyond lab research and into actual production testing and use. Microsoft is currently running MLVC in live peer-to-peer Microsoft Teams calls with active telemetry and A/B testing.
Evaluating MLVC requires separating real-world engineering achievements from marketing benchmarks, understanding the hardware constraints of consumer NPUs, and evaluating the shifting intellectual property landscape.
I address five key areas in order:
-
Performance Analysis: Unpacking Microsoft’s bitrate claims, subjective vs. objective metrics, and real-world RTC baselines.
-
NPU Compatibility and Cross-Platform Integration: How scale-sharing solves floating-point drift, runtime tooling, and bitstream generation.
-
The Power and Thermal Reality: Model parameter scales, NPU utilization budgets, and power trade-offs versus fixed silicon.
-
Intellectual Property Impact: Legacy patent pools versus emerging AI patents under MIT licensing.
-
Practical Deployment Timeline: Resolution boundaries, dataset caveats, and realistic adoption phases.
Contents
1. Performance: Claims vs. Real-World Baselines
In its public announcement, Microsoft reported substantial bitrate savings for MLVC at both 360p and 540p resolutions for H.264 and H.265. You can safely ignore the H.264 numbers. These comparisons weren’t tested or reported in the actual research paper. They only appear in the marketing blog post and rely on an inflated anchor. Specifically, Microsoft claimed an 87.8% reduction by comparing MLVC at 122 kbps to H.264 at 1 Mbps, a highly inflated number, without actually detailing any comparisons.
On the other hand, the HEVC comparisons are valid and impressive in their own right. Microsoft benchmarked two versions of MLVC, MLVC, and MLVC-S. MLVC is the full model, while MLVC-S is “a smaller model tuned for 150 FPS at 540p.” As shown in Figure 2, the researchers compared both against Intel QuickSync HEVC on the Video Conferencing Dataset under strict single-pass, real-time encoding conditions. That matches an actual hardware pipeline used in enterprise video calling today.
The research paper shows an MOS BD-Rate improvement of over 70% compared to hardware HEVC. As you probably know, MOS stands for Mean Opinion Score, which means humans actually watching the videos and rating them. MOS comparisons are the gold standard for evaluating real-world video quality, since human perception is what actually matters, not PSNR or other scores.
The bottom row of Figure 3 contains MLVC-multi, which appears to be an interesting bit of specsmanship that stitches 360p MLVC (-75.5% MOS, 103/99 FPS avg) and 540p MLVC-S (-65.4% MOS, 121/114 FPS avg) into one ladder for -71.8% MOS. It’s like using H.264 Baseline for 360p and Main for 540p, except H.264 uses one decoder. Here you need two resident models (18.3M + 5.4M params), with load/swap costs and potentially noticeable quality deltas at switch points.
This appears to be a strategy to support the GitHub statement that “MLVC achieves >70% MOS-based BD-rate improvement over hardware HEVC while averaging around 100 FPS for both encoding and decoding on commodity NPUs from Apple, Intel, and Qualcomm (540p).” As Figures 3 and 6 show, no single model does all three at once: MLVC at 540p is 49/48 FPS avg, MLVC-S at 540p is 65.4% MOS, and 100 FPS + >70% MOS is MLVC at 360p.
Given that the vast majority of RTC is single stream anyway, it seems kind of silly.
Quality Analysis
MLVC’s advantage over hardware HEVC is clear at very low bitrates. But the chart also shows why MLVC-S is more than an ultra-low-bitrate curiosity. At a 720p display, MLVC-S at ∼900 kbps hits ∼3.83 MOS while HEVC-QSV 720p needs ∼2.5-2.8 Mbps to reach the same score. And unlike full MLVC, MLVC-S is real-time at 720p on all tested NPUs.
In practical terms, that makes MLVC-S a workable RTC solution up to 720p today at roughly one-third the bitrate of hardware HEVC for the same MOS. If you want true native 1080p, the ceiling shifts. MLVC-S 1080p is 39.5/34.6 FPS on Apple, but only 24.0/25.8 FPS on Intel and 26.5/19.6 FPS on Qualcomm, so it’s real-time only on Apple today.
2. NPU Support: Cross-Platform Interoperability and Runtime Realities
By using NPUs in place of dedicated hardware for decoding, Microsoft claims it has significantly accelerated MLVC’s adoption curve. This was also a claim made by Deep Render, though my testing was exclusively on Macs, which raised questions about cross-NPU compatibility.
The blocker for neural codecs is that, unlike H.264 or HEVC, they are not bit-exact by spec. As Microsoft describes in the white paper, “existing state-of-the-art neural video codecs (e.g., DCVC-RT [dcvc-rt]) suffer from catastrophic decoding failures due to numerical divergence between platforms.” A tiny difference in floating-point math between encoder and decoder breaks entropy decoding completely.
Microsoft fixes that with what it calls a “scale sharing mechanism.” As the paper puts it, “We develop a scale sharing mechanism and transmit scale indices within the hyperprior, enabling reliable inference across diverse hardware platforms without requiring bit-exact arithmetic.”
In practice, that means MLVC doesn’t work like a traditional hardware decoder. Specifically, a traditional decoder performs the full decode in one fixed function block, requiring little from the device’s main CPU. This is shown on top of Figure 4.
MLVC splits the decode. The CPU runs the C++ entropy decoder, the part that has to be bit-exact, and passes the processed bitstream to the NPU to complete the decoding. This is shown at the bottom of Figure 4.
This is fine for RTC running on computers and capable smartphones, but may not be for future potential applications of MLVC like OTT viewing on living room devices. These devices use small, low-power ARM cores. MLVC’s production entropy coder is a real-time C++ program that needs a desktop- or mobile-class CPU to keep up. A living room device’s CPU isn’t powerful enough to run the program at 1080p in real time, so you would still lose real-time performance even with the NPU.
Even on mobile phones, because MLVC requires both the CPU and NPU to be active simultaneously, power consumption will be higher than with a dedicated ASIC. While Microsoft doesn’t publish numbers, the paper notes that “Power consumption is likely to become the next optimization frontier now that real-time performance has been achieved.”
Running Encode and Decode
Operationally, the GitHub repository supplies both encode and decode tools. To encode, Microsoft supplies a Python utility with the trained model that runs on Windows, Mac, and Android. For Apple, the same CoreML model runs on both macOS and iOS/iPadOS. The package includes the trained neural model and all the code needed to compress video to a bitstream.
The utility will run on a regular CPU, but for real-time operation you need either an Apple device with a Neural Engine, a Windows laptop with an Intel NPU, a Snapdragon X Elite device with a Qualcomm NPU, or a Windows or Linux desktop with an NVIDIA GPU. On all platforms, you point the encoder at your YUV video, and it outputs a bitstream.
The utility will run on a regular CPU, but for real-time operation you need either an Apple Mac with a Neural Engine, a Windows laptop with an Intel NPU, a Snapdragon X Elite device with a Qualcomm NPU, or a Windows or Linux desktop with an NVIDIA GPU. On all platforms, you point the encoder at your YUV video, and it outputs a bitstream.
To decode, you need two programs on the device: the C++ program that unpacks the bitstream on the CPU, and an NPU-specific neural decoder to complete the decoding. Again, Microsoft supplies all of the above in the GitHub repository, including separate NPU-specific versions for Apple, Intel, and Qualcomm.
What happens when one person doesn’t have a compatible NPU?
In a 1:1 Teams call, you can check both ends. In a real meeting with 5 or 10 people, you can’t. If one laptop in the call can’t run MLVC, you have to decide what to do.
Either you drop everyone back to H.264, or you simulcast – encode MLVC and H.264 at the same time and let the SFU route the right one to each receiver, or you transcode MLVC to H.264 in the cloud. That is how AV1 rolled out in conferencing for three years until hardware decode became common.
So outside of a controlled environment, you don’t replace H.264 with MLVC. You run both. Which undercuts the power story we get to next.
3. Power, Thermal, and Model Size Realities
That’s the NPU compatibility issue. Now let’s look at power consumption and how that might impact suitability for mobile deployments. According to Microsoft, MLVC is engineered to run at 540p30 and 360p30 while capping NPU utilization at or below 50% on Apple Neural Engine, Qualcomm, and Intel NPUs. Microsoft doesn’t specify the target percentage of the CPU component.
Converting this and the hardware decoding of HEVC, AV1, or H.264 into power consumption numbers for comparison was hard to nail down. My best guess is that 50% of an NPU plus some percentage of the CPU probably uses about 5x the power of a dedicated AV1 or HEVC decoder in a SoC. If I’m off by an order of magnitude, please let me know. This differential is fine for intermittent conferencing or plugged-in use, but not for long periods of unplugged conferencing or OTT viewing.
So, what happens? Phone manufacturers will likely continue to include traditional decode hardware in the SoCs for OTT viewing and perhaps conferencing. So, the future isn’t an NPU or dedicated silicon decode; it’s both. Just because you can play video in an existing NPU doesn’t mean it’s the preferred method, especially when battery life is important.
What about in the living room? As mentioned above, few living room devices currently have NPUs or sufficiently powerful CPUs to support MLVC’s hybrid operation. Given how important cost is to the living room market, it’s hard to see a hybrid approach catching on.
4. Intellectual Property Impact: Legacy Pools vs. Emerging AI Patents
One of the attractions of an AI codec was a clean slate from a patent perspective. It’s true that moving from traditional block codecs like H.264, HEVC, and AV1 to neural architectures like MLVC replaces classical primitives with end-to-end learned networks. While this departure places MLVC outside traditional patent pools managed by MPEG LA, Access Advance, or Via LA, it doesn’t mean the codec is infringement-free. There are a number of existing patents that may cover techniques used by MLVC. I mention these to burst the perception bubble that AI codecs might be royalty-free.
To begin, note that Microsoft released MLVC on GitHub under the MIT License, which grants only copyright. It contains no express patent license and no patent retaliation clause.
Beyond that, the aggressive IP activity around next-generation video compression shows how established licensors are building thickets around neural video coding. Analysis from IP research firm GreyB, which examined 3,916 patents and 1,357 JVET contributions for its Beyond VVC (H.267) landscape report, shows that contributions to the “Beyond VVC” track are dominated by commercial licensors. Rather than waiting for a finalized neural spec, these entities are filing heavily across both the Enhanced Compression Model (ECM) and the dedicated Neural Network Video Coding (NNVC) track to secure future SEP positions.
Similarly, reporting from Streaming Media Magazine highlights InterDigital’s R&D shift toward overlaying neural tools onto traditional frameworks. Per that piece, InterDigital began ECM work in 2021 and by late 2024 was showing a 28% coding gain over VVC, with a parallel hybrid approach described as VVC overlaid with AI tools termed Neural Network Video Coding.
How This Impacts Open-Source Neural Codecs Like MLVC
This land grab underscores that neural compression is already heavily covered by patent filings, with early-granted patents like Qualcomm’s US Patent 11,388,416 already in force. Rather than being a blank slate, the domain is being systematically filed on. Most neural codecs would use end-to-end rate-distortion autoencoders, learned entropy bottlenecks, and feature-space alignment, all of which are targets of broad filings.
And unlike H.264, HEVC, or VVC, where Access Advance, Avanci, or Via LA offer one-stop licensing, no pool exists for neural video coding today, though many are likely in the formulation stages. This means that implementors must deal with individual patent owners in an inefficient, administrative-heavy licensing process.
What’s the practical impact? Though Microsoft’s open-source license sounds enticing, potential implementors need to conduct due diligence to ensure their use doesn’t infringe on any existing or pending patents. The bottom line is that adopting a standalone, non-standardized neural codec doesn’t avoid exposure to compression IP; it shifts risk from established block-codec pools to an emerging web of bilateral AI compression patents that will have to be negotiated one by one.
5. Deployment Timeline and Practical Caveats
Understanding when MLVC becomes viable depends on the operating environment and current technical boundaries.
Resolution Ceiling Today and the Compute Scaling Barrier
The benchmarks and anchors published in Microsoft’s repository target 360p, 540p, and 720p resolutions. The paper reports playback up to 1080p, as shown in Figure 6. As you can see, MLVC-S achieves faster-than-30-fps 1080p playback on the Apple NPU, but not on Intel or Qualcomm, and the full MLVC model doesn’t come close on any hardware. Still, particularly for low-bitrate conferencing, MLVC appears ready to deploy today, if you can work around the compatibility issues and resolve the IP issues.
You’d expect Microsoft to significantly optimize playback on existing NPUs over the next 12-24 months, and faster NPUs are the rising tide that lifts all boats. But there are other reasons why it will take some time to transition MLVC to OTT usage.
Why High-Definition Movies Are Out of Reach for Now
Beyond playback rate, transitioning MLVC from low-resolution RTC to high-definition OTT requires addressing several core codec-engineering hurdles. These include:
1. Retrain at high res. The released checkpoints are trained on 360p-720p VCD and OpenVidHD. These must be retrained at high res.
2. NPU graphs are fixed size. The repo makes you re-export for each size: 640×368, 960×544, etc. For 1080p/4K you have to re-export and re-quantize for CoreML, OpenVINO, and QNN separately.
3. RTC rate control ≠ film rate control. MLVC’s control is tuned for low-latency camera feeds. Film needs two-pass VBR.
4. 8-bit only. Current pipeline is YUV420 8-bit. Movies need 10-bit HDR10/Dolby Vision.
Combine all this with the uncertainty as to whether a hybrid NPU-based codec is best for OTT living room viewing, and you have an uncertain future beyond RTC.
Summary
Microsoft’s MLVC is unquestionably a wonderful technical achievement, perhaps the world’s first ML-based codec capable of production use. But beyond RTC, MLVC’s proximity to the sun made us seriously consider deployment issues that previously seemed unquestionably favorable.
Chief among these is whether an NPU-based codec is the best technology for OTT viewing, whether in the living room or on mobile. While NPUs are great for immediate compatibility on supported NPUs, they are not the best solution for battery-powered devices. It also seems unlikely that Smart TVs, dongles, and STBs will incorporate both NPUs and sufficiently capable CPUs to enable MLVC.
And for those envisioning that greenfield AI codecs would deliver a potentially royalty-free future, even a casual look at the patent landscape brings back the timeless lyric from “We Won’t Get Fooled Again.” That is, of course, “Meet the new boss. Same as the old boss.”
