Streaming Learning Center Where Streaming Professionals Learn to Excel
Related Articles
Microsoft open-sourced Machine Learning Video Codec (MLVC) under the MIT License, publishing the codebase and model weights on GitHub alongside a technical announcement on the Microsoft Tech Community blog. The research paper, “MLVC: A Multi-platform Learned Video Codec for Real-World Deployment,” details the underlying architecture and makes this bold claim.
MLVC is the first neural video codec to combine competitive compression performance, real-time speed, and cross-platform robustness across diverse consumer devices, making it suitable for widespread deployment.
From my perspective, this release is significant because MLVC is the first neural video codec from a major platform vendor to move beyond lab research and into actual production testing. Microsoft is currently running MLVC in live peer-to-peer Microsoft Teams calls with active telemetry and A/B testing.
Evaluating what this all means requires separating real-world engineering achievements from marketing benchmarks, understanding the hardware constraints of consumer NPUs, and evaluating the shifting intellectual property landscape.
I address five key areas in order:
-
Performance Analysis: Unpacking Microsoft’s bitrate claims, subjective vs. objective metrics, and real-world RTC baselines.
-
NPU Compatibility and Cross-Platform Integration: How scale-sharing solves floating-point drift, runtime tooling, and bitstream generation.
-
The Power and Thermal Reality: Model parameter scales, NPU utilization budgets, and power trade-offs versus fixed silicon.
-
Intellectual Property Impact: Legacy patent pools versus emerging AI patents under MIT licensing.
-
Practical Deployment Timeline: Resolution boundaries, dataset caveats, and realistic adoption phases.
Contents
1. Performance: Claims vs. Real-World Baselines
In its public announcement, Microsoft reported substantial bitrate savings for MLVC at both 360p and 540p resolutions.
You can safely ignore the H.264 numbers. These comparisons weren’t tested or reported in the actual research paper. They only appear in the marketing blog post and rely on an inflated anchor. Microsoft claimed an 87.8% reduction by comparing MLVC at 122 kbps to H.264 at 1 Mbps, a highly inflated number, without actually detailing any comparisons.
On the other hand, the HEVC comparisons are valid and impressive in their own right. Microsoft benchmarked MLVC against Intel QuickSync HEVC on the Video Conferencing Dataset under strict single-pass, real-time encoding conditions. That matches the actual hardware pipeline used in enterprise video calling today.
The research paper shows an MOS BD-Rate improvement of over 70% compared to hardware HEVC. MOS stands for Mean Opinion Score, which means humans actually watching the videos and rating them. MOS comparisons are the gold standard for evaluating real-world video quality, since human perception is what actually matters. MLVC beats hardware HEVC so handily on MOS BD-Rate because neural codecs handle low-bitrate compression more effectively.
Why did MLVC beat HEVC so soundly at lower bitrates? As Microsoft explains in its research paper, “By replacing discrete block transforms and coarse quantization with continuous feature-space representations, the model avoids traditional macroblocking artifacts at ultra-low bitrates, resulting in smooth visual degradation rather than structural breakdown.”
In English, this means that by training on perceptual loss functions, MLVC doesn’t collapse into square blocks when bandwidth drops. Rather, it smooths out high-frequency artifacts and keeps facial features and background edges sharp.
Beating hardware HEVC by 65%-75% appears to represent a major perceptual leap over current live silicon. Just remember that while this baseline is completely fair for real-time video calls, it doesn’t tell us how MLVC compares to multi-pass VOD software encoders like x265 or SVT-AV1.
2. NPU Support: Cross-Platform Interoperability and Runtime Realities
By using NPUs in place of dedicated hardware decoding, the traditional path of codecs like H.264, HEVC, and AV1, Microsoft claims it’s significantly accelerated the adoption curve. This was also a claim made by Deep Render, though it tested only on Macs, raising the question of cross-NPU compatibility.
By way of background, traditional video codecs like H.264 and HEVC are bit-exact by specification. A hardware decoder on any vendor must produce identical pixels. Neural video codecs don’t share that property, which is the single biggest blocker to real-world deployment.
.The break happens in entropy coding, which will fail catastrophically if the probability prior differs even slightly between encoder and decoder. In learned codecs, that prior’s typically the scale sigma from the hyperprior.
As an example, encode on an Apple M series NPU and decode on an Intel Lunar Lake or Qualcomm Snapdragon NPU, and tiny floating-point rounding differences accumulate. Different rounding modes, kernel fusion, and reduction orders make scale values drift. Once drift crosses a quantization boundary, the decoder picks a different lookup table index, decodes the wrong latent, and errors cascade through temporal prediction. In a few frames, the video is fully corrupted. Early SSF codec tests showed it’s not suitable for real-world apps due to this mismatch https://arxiv.org/abs/2410.20145.
The Microsoft workaround
Microsoft addresses this failure through a scale-sharing hyperprior mechanism. Instead of calculating entropy parameters dynamically on both sides using floating-point math, the encoder explicitly transmits scale indices directly inside the hyperprior bitstream. The decoder uses simple lookup tables based on these transmitted indices, ensuring deterministic entropy decoding regardless of underlying floating-point rounding modes across chipsets.
While this design solves bitstream drift, running MLVC across platforms relies on vendor-specific execution runtimes. Deployment requires exporting models to specific framework targets: CoreML for Apple, ONNX with OpenVINO for Intel, and QNN for Qualcomm.
To produce and decode binary bitstreams, developers must compile the production-ready C++ entropy coder (msrtc_rans) from the repository, as Python wrappers won’t suffice for real-time deployment. Full native C++ SDKs and integration wrappers for FFmpeg or WebRTC remain on Microsoft’s roadmap. For multi-platform deployment, the repository provides tools to export checkpoints to CoreML (Apple), ONNX/OpenVINO (Intel), and QNN (Qualcomm). Microsoft has solved the cross-platform determinism problem, but you’re still left maintaining vendor-specific conversion pipelines.
3. Power, Thermal, and Model Size Realities
That’s the NPU compatibility issue. Now let’s look at power consumption and how that might impact suitability for mobile deployments. According to Microsoft’s announcement on the Microsoft Tech Community blog, MLVC is engineered to run at 540p30 (and 360p30) while capping NPU utilization at or below 50% on Apple Neural Engine and Qualcomm and Intel NPUs, leaving remaining compute headroom for system tasks like background blur, spatial framing, and audio noise suppression.
Traditional video codecs decode video using dedicated, fixed-function media silicon, such as Intel QuickSync, NVIDIA NVENC, Apple’s Media Engine, or Qualcomm’s hardware video processing units. These hardwired ASIC gates execute H.264, HEVC, and AV1 decoding at minimal power—with dedicated hardware decoders in mobile SoCs consuming as little as 113 mW to 380 mW for decoding, and total device playback averaging around 1.6 Watts for 4K video. Because the decoding logic is hardwired directly into silicon, the main CPU stays asleep and thermal throttling is virtually nonexistent during playback.
Running a neural codec like MLVC flips that operational model entirely. Instead of passing the bitstream to a dedicated video decoder, the system routes tensor operations to mobile NPUs such as Apple’s Neural Engine, Qualcomm’s Hexagon NPU, or Intel’s NPU (https://arxiv.org/abs/2606.28027). While NPUs are far more efficient for neural inference than CPUs or GPUs, running continuous matrix multiplications every frame pushes power draw into the 1 to 3 Watt range depending on resolution and frame rate (https://arxiv.org/abs/2403.06109). On a smartphone battery, that extra power draw directly impacts thermal limits and total watch time.
The strategic choice for mobile deployment comes down to a clear technical tradeoff. Fixed-function ASIC decoders deliver unmatched battery life, but they require multi-year silicon design and hardware rollout cycles, which is why hardware AV1 decoding took years to land on mainstream mobile SoCs like the Apple A17 Pro and Snapdragon 8 Gen 2 (https://www.qualcomm.com/products/mobile/snapdragon/smartphones/snapdragon-8-series-mobile-platforms/snapdragon-8-gen-2-mobile-platform). NPUs trade some of that raw milliwatt efficiency for immediate software versatility, allowing neural codecs like MLVC to deploy instantly across existing smartphones without waiting for a new generation of dedicated ASIC hardware to be printed onto silicon (https://arxiv.org/abs/2606.28027).
In contrast, running an end-to-end neural network across an NPU engages general matrix-multiplication tensor cores continuously. On battery-powered laptops and mobile devices, allocating up to 50% of active NPU capacity to video processing draws significantly more power than handing off frames to fixed hardware media blocks. For stationary desktop systems plugged into wall power, this power trade-off is negligible. On laptops running on battery, continuous neural video processing introduces thermal and battery draw penalties that dedicated media silicon avoids.
4. Intellectual Property Impact: Legacy Pools vs. Emerging AI Patents
Microsoft released MLVC under the MIT License on GitHub. From an architectural standpoint, MLVC replaces classical block-based video compression primitives (motion estimation vectors, discrete cosine transforms, and hand-tuned macroblock partitioning) with end-to-end learned neural models. This departure places the core architecture outside the scope of traditional MPEG and AOM block-codec patent pools, sidestepping standard pool royalty claims from licensing entities managing HEVC, VVC, or AV1 portfolios.
However, the choice of license carries an important legal distinction:
The MIT License provides a copyright grant, but unlike licenses such as Apache 2.0, it contains no express patent grant. Open-sourcing code under MIT with downloadable pre-trained weights does not grant an explicit patent license to downstream implementers.
Furthermore, major telecommunications and technology companies (including InterDigital, Qualcomm, Samsung, Huawei, and Microsoft) hold extensive patent portfolios covering foundational neural image and video compression techniques. These include patents on hyperprior architectures, learned entropy bottlenecks, neural quantization methods, and feature-space motion compensation. Moving from classical block codecs to neural codecs swaps classical pool royalties for uncharted neural compression patent territory.
5. Deployment Timeline and Practical Caveats
Understanding when MLVC becomes viable depends on the operating environment and current technical boundaries:
-
Resolution Ceiling Today: The benchmarks and anchors published on GitHub target 360p, 540p, and 720p resolutions. No 1080p or 4K anchors exist in the repo, reinforcing that MLVC is designed as an RTC-first codec for lower-resolution video grids rather than a high-resolution streaming codec.
-
Training Data Realities: The pre-trained checkpoints available for download were trained on the OpenVid dataset, whereas numbers reported in portions of the research literature utilized Vimeo-based datasets. Downstream developers should expect minor rate-distortion variations depending on the fine-tuning dataset used.
Phase 1 (0–2 Years): Closed Ecosystems and Fixed Power
MLVC provides immediate value in controlled corporate applications like Microsoft Teams P2P calls running on desktop hardware. In environments where Microsoft controls both endpoints and the application stack, trading NPU compute for 60% to 70% real-world bandwidth reduction keeps video calls stable on congested cellular or satellite connections.
Phase 2 (2–4 Years): Native C++ Tooling and RTC Integration
Broader enterprise WebRTC adoption requires full native C++ wrapper libraries and integration into media processing frameworks like WebRTC and FFmpeg. Once Microsoft delivers full C++ SDK bindings beyond the standalone RANS entropy coder, third-party RTC platforms can evaluate MLVC natively.
Phase 3 (5+ Years): Silicon Standardization and OTT Distribution
For mainstream OTT video streaming (Netflix, YouTube, live broadcast), MLVC is not an immediate replacement for AV1, HEVC, or VVC. Mass OTT distribution requires universal, low-power hardware decoding across billions of smart TVs, mobile phones, and streaming sticks. Until neural execution layers are burned into low-power, fixed-function media silicon across all consumer chipsets, traditional standards will remain the primary delivery mechanism for large-scale video distribution.
