Streaming Learning Center Where Streaming Professionals Learn to Excel
Related Articles
Microsoft open-sourced Machine Learning Video Codec (MLVC) under the MIT License, publishing the codebase and model weights on GitHub alongside a technical announcement on the Microsoft Tech Community blog. The research paper, “MLVC: A Multi-platform Learned Video Codec for Real-World Deployment,” details the underlying architecture and makes this bold claim.
MLVC is the first neural video codec to combine competitive compression performance, real-time speed, and cross-platform robustness across diverse consumer devices, making it suitable for widespread deployment.
From my perspective, this release is significant because MLVC is the first neural video codec from a major platform vendor to move beyond lab research and into actual production testing. Microsoft is currently running MLVC in live peer-to-peer Microsoft Teams calls with active telemetry and A/B testing.
Evaluating what this all means requires separating real-world engineering achievements from marketing benchmarks, understanding the hardware constraints of consumer NPUs, and evaluating the shifting intellectual property landscape.
I address five key areas in order:
-
Performance Analysis: Unpacking Microsoft’s bitrate claims, subjective vs. objective metrics, and real-world RTC baselines.
-
NPU Compatibility and Cross-Platform Integration: How scale-sharing solves floating-point drift, runtime tooling, and bitstream generation.
-
The Power and Thermal Reality: Model parameter scales, NPU utilization budgets, and power trade-offs versus fixed silicon.
-
Intellectual Property Impact: Legacy patent pools versus emerging AI patents under MIT licensing.
-
Practical Deployment Timeline: Resolution boundaries, dataset caveats, and realistic adoption phases.
Contents
1. Performance: Claims vs. Real-World Baselines
In its public announcement, Microsoft reported substantial bitrate savings for MLVC at both 360p and 540p resolutions.
You can safely ignore the H.264 numbers. These comparisons weren’t tested or reported in the actual research paper. They only appear in the marketing blog post and rely on an inflated anchor. Microsoft claimed an 87.8% reduction by comparing MLVC at 122 kbps to H.264 at 1 Mbps, a highly inflated number, without actually detailing any comparisons.
On the other hand, the HEVC comparisons are valid and impressive in their own right. Microsoft benchmarked two versions of MLVC, MLVC, and MLVC-S. MLVC is the full model, while MLVC-S is “a smaller model tuned for 150 FPS at 540p.” As shown in Figure 2, the researchers compared both against Intel QuickSync HEVC on the Video Conferencing Dataset under strict single-pass, real-time encoding conditions. That matches an actual hardware pipeline used in enterprise video calling today.
The research paper shows an MOS BD-Rate improvement of over 70% compared to hardware HEVC. MOS stands for Mean Opinion Score, which means humans actually watching the videos and rating them. MOS comparisons are the gold standard for evaluating real-world video quality, since human perception is what actually matters.
MLVC-multi appears to be an interesting bit of specsmanship that stitches 360p MLVC (-75.5% MOS, 103/99 FPS avg) and 540p MLVC-S (-65.4% MOS, 121/114 FPS avg) into one ladder for -71.8% MOS. It’s like using H.264 Baseline for 360p and Main for 540p, except H.264 uses one decoder. Here you need two resident models (18.3M + 5.4M params), with load/swap costs and potentially noticeable quality deltas at switch points.
This appears to be a strategy to support the GitHub statement that “MLVC achieves >70% MOS-based BD-rate improvement over hardware HEVC while averaging around 100 FPS for both encoding and decoding on commodity NPUs from Apple, Intel, and Qualcomm (540p).” As Figures 3 and 7 show, no single model does all three at once: MLVC at 540p is 49/48 FPS avg, MLVC-S at 540p is 65.4% MOS, and 100 FPS + >70% MOS is MLVC at 360p.
Quality Analysis
MLVC’s advantage over hardware HEVC is clear at very low bitrates. But the chart also shows why MLVC-S is more than an ultra-low-bitrate curiosity. At 720p display, MLVC-S at ∼900 kbps hits ∼3.83 MOS while HEVC-QSV 720p needs ∼2.5-2.8 Mbps to reach the same score. And unlike full MLVC, MLVC-S is real-time at 720p on all tested NPUs.
In practical terms, that makes MLVC-S a workable RTC solution up to 720p today at roughly one-third the bitrate of hardware HEVC for the same MOS. If you want true native 1080p, the ceiling shifts. MLVC-S 1080p is 39.5/34.6 FPS on Apple, but only 24.0/25.8 FPS on Intel and 26.5/19.6 FPS on Qualcomm, so it’s real-time only on Apple today.
2. NPU Support: Cross-Platform Interoperability and Runtime Realities
By using NPUs in place of dedicated hardware decoding, the traditional path of codecs like H.264, HEVC, and AV1, Microsoft claims it’s significantly accelerated the adoption curve. This was also a claim made by Deep Render, though it tested only on Macs, raising questions about cross-NPU compatibility.
By way of background, traditional video codecs like H.264 and HEVC are bit-exact by specification. A hardware decoder on any vendor must produce identical pixels. Neural video codecs don’t share that property, which is the single biggest blocker to real-world deployment. The break occurs in entropy coding, which will fail catastrophically if the prior probability differs even slightly between encoder and decoder. This is shown in Figure 4.
The Microsoft Workaround
Microsoft addresses this with what the paper calls a “scale sharing mechanism.” The core idea: “We develop a scale sharing mechanism and transmit scale indices within the hyperprior, enabling reliable inference across diverse hardware platforms without requiring bit-exact arithmetic.”
The paper details it as: “We derive scale indices from the quantized hyperlatent through deterministic expansion” and “Since the hyperlatent is coded with a fixed factorized entropy model, it is identical on both the encoder and decoder sides, guaranteeing that the resulting scale parameters are also identical and thus avoiding catastrophic decoding failures.” In the diagram description: “A slice of the hyper decoder output is passed through Abs, Expand, and Lookup operations to deterministically obtain the scale parameters.” This is shown in Figure 5.
For deployment, the repository is explicit about the vendor split. It provides “Conversion tools: Export to CoreML (Apple), ONNX for OpenVINO (Intel) and QNN (Qualcomm)” and examples like convert.py export –model-type coreml –target-device apple / onnx –target-device intel / onnx –target-device qualcomm.
To get a decodable bitstream, the README notes you need the native coder: “C++ entropy coder: Production-ready, optimized for real-time deployment” and “To produce actual bitstreams, you need to install the entropy coder (requires C++ compiler)” via uv pip install packages/msrtc_rans. There is no full FFmpeg/WebRTC SDK in the repo today – Microsoft has solved determinism, but you are still maintaining three vendor conversion pipelines.
3. Power, Thermal, and Model Size Realities
That’s the NPU compatibility issue. Now let’s look at power consumption and how that might impact suitability for mobile deployments. According to Microsoft, MLVC is engineered to run at 540p30 and 360p30 while capping NPU utilization at or below 50% on Apple Neural Engine, Qualcomm, and Intel NPUs.
Numbers are hard to nail down. I couldn’t find any decode-only power numbers for hardware AV1 or HEVC decoding, but as best I can figure, 50% of an NPU probably uses about 5x as much power as a dedicated AV1 or HEVC decoder in the SoC. If I’m off by an order of magnitude, please let me know. This might be fine for intermittent conferencing or plugged-in use, but not preferable for long periods of lean-back viewing, where that 5x translates to ∼40-50% shorter battery life.
So, what happens? Does MLVC get formalized in hardware? Not any time soon; the codec is nascent. That means phone manufacturers will keep building decode hardware into the SoCs. It’s not NPU or dedicated silicon; it’s NPU and dedicated silicon decode.
Just because you can play video in an existing NPU doesn’t mean it’s the preferred method, especially when battery life is important.
What about in the living room? Power is hardly ever an issue, but neither is low-bandwidth quality. It might be that a general-purpose NPU would be preferable to a fixed codec decoder chip for a Smart TV manufacturer, but given the paucity of NPUs currently in Smart TVs, any living-room relevance is a good 4-6 years away at best.
4. Intellectual Property Impact: Legacy Pools vs. Emerging AI Patents
One of the attractions of an AI codec was a clean slate from a patent perspective. It’s true that moving from traditional block codecs like H.264, HEVC, and AV1 to neural architectures like MLVC replaces classical primitives with end-to-end learned networks. While this departure places MLVC outside traditional patent pools managed by MPEG LA, Access Advance, or Via LA, it doesn’t mean the codec is infringement-free.
That said, there are a number of patents out there that may cover techniques used by MLVC. I mention these to burst the perception bubble that AI codecs might be royalty-free.
To begin, note that Microsoft released MLVC on GitHub under the MIT License, which grants only copyright. It contains no express patent license and no patent retaliation clause.
Before diving into the patent list, note that the foundational hyperprior architecture underpinning many learned codecs was published open access by Ballé et al. in 2018. This establishes significant prior art that predates many recent filings. Many patent filings covering neural video coding are still pending applications rather than granted patents.
Beyond that, the aggressive IP activity around next-generation video compression shows how established licensors are building thickets around neural video coding. Analysis from IP research firm GreyB, which analyzed 3,916 patents and 1,357 JVET contributions for its Beyond VVC (H.267) landscape report, details how contributions to the “Beyond VVC” track are dominated by commercial licensors. Rather than waiting for a finalized neural spec, these entities are filing heavily across both the Enhanced Compression Model (ECM) and the dedicated Neural Network Video Coding (NNVC) track to secure future SEP positions.
Similarly, reporting from Streaming Media Magazine highlights InterDigital’s R&D shift toward overlaying neural tools onto traditional frameworks. Per that piece, InterDigital began ECM work in 2021 and by late 2024 was showing 28% coding gain over VVC, with a parallel hybrid approach described as VVC overlaid with AI tools termed Neural Network Video Coding.
How This Impacts Open-Source Neural Codecs Like MLVC
This land grab underscores that neural compression is already heavily covered by patent filings, with early-granted patents like Qualcomm’s US Patent 11,388,416 already in force. Rather than being a blank slate, the domain is being systematically filed on. Most neural codecs would use end-to-end rate-distortion autoencoders, learned entropy bottlenecks, and feature-space alignment, all of which are targets of broad filings.
And unlike H.264, HEVC, or VVC, where Access Advance, Avanci, or Via LA offer one-stop licensing, no pool exists for neural video coding today. This means dealing with individual patent owners in an inefficient, administrative-heavy licensing process.
The bottom line is that adopting a standalone, non-standardized neural codec does not avoid exposure to compression IP; it shifts risk from established block-codec pools to an emerging web of bilateral AI compression patents that will have to be negotiated one by one.
5. Deployment Timeline and Practical Caveats
Understanding when MLVC becomes viable depends on the operating environment and current technical boundaries.
Resolution Ceiling Today and the Compute Scaling Barrier
The benchmarks and anchors published in Microsoft’s repository target 360p, 540p, and 720p resolutions. The paper shows playback at up to 1080p as shown in Figure 5. As you can see, MLVC-S achieves faster-than-30-fps playback on the Apple NPU, but not on Intel or Qualcomm, and the full MLVC model doesn’t come close on any hardware.
You’d expect Microsoft to significantly optimize playback on existing NPUs over the next 12-24 months, and faster NPUs are the rising tide that lifts all boats. But there are other reasons why it will take some time to transition MLVC to OTT usage.
Why High-Definition Movies Are Out of Reach for Now
Beyond playback rate, transitioning MLVC from low-resolution RTC to high-definition OTT requires addressing several core codec-engineering hurdles. These include:
1. Trained at high res. The released checkpoints are trained on 360p-720p VCD and OpenVidHD. These must be retrained at high res.
2. NPU graphs are fixed size. The repo makes you re-export for each size: 640×368, 960×544, etc. For 1080p/4K you have to re-export and re-quantize for CoreML, OpenVINO, and QNN separately.
3. RTC rate control ≠ film rate control. MLVC’s control is tuned for low-latency camera feeds. Film needs two-pass VBR.
4. 8-bit only. Current pipeline is YUV420 8-bit. Movies need 10-bit HDR10/Dolby Vision.
Combine all this with the uncertainty as to whether an NPU-based codec is best for OTT viewing, and you have an uncertain future beyond RTC.
Summary
Microsoft’s MLVC is unquestionably a wonderful technical achievement, perhaps the world’s first ML-based codec in production use. But beyond RTC, MLVC’s proximity to the sun made us seriously consider deployment issues that previously seemed unquestionably favorable.
Chief among these is whether an NPU-based codec is the best technology for OTT viewing, whether in the living room or on mobile. While NPUs are great for immediate compatibility on supported NPUs, it’s probably not a permanent solution for battery-powered devices. It’s also unclear if and when SmartTVs, dongles, and STBs will choose this direction.
And for those envisioning that greenfield AI codecs would deliver a potentially royalty-free future, even a casual look at the patent landscape brings back the timeless lyric from “We Won’t Get Fooled Again.” That is, of course, “Meet the new boss. Same as the old boss.”
