Streaming Learning Center Where Streaming Professionals Learn to Excel
Related Articles
On March 11, 2025, I (Jan Ozer) from the Streaming Learning Center interviewed Yoann Hinard, COO of Witbe. We discussed Witbe’s innovative approach to quality of experience (QoE) testing, their findings from Super Bowl performance monitoring, and what Witbe plans to showcase at NAB 2025. Key topics included the impact of device variability on performance, the role of low latency in streaming, and insights into 4K quality when content is upscaled from 1080p.
You can read the full summary below or watch the video on YouTube.
Contents
About Witbe: Ensuring Real-World Performance with Real Devices
Witbe specializes in automated testing and monitoring solutions designed to measure performance directly from the end-user perspective. Unlike traditional network monitoring tools that rely on simulated conditions, Witbe’s approach emphasizes real-world testing by analyzing services on physical devices just as a customer would experience them.
Central to Witbe’s solution is its proprietary Witbox (Figure 1). The Witbox is a hardware device that connects to streaming devices such as OTT dongles, smart TVs, and set-top boxes via HDMI. This connection allows Witbe to assess video quality and performance without altering the device’s operating system or modifying its applications. Because the Witbox interacts with real devices in actual locations, it captures performance metrics that accurately reflect what viewers experience.
One of the key strengths of the Witbox is its remote management capability (Figure 2). Operators can control Witboxes from a central location, enabling them to test services on devices located across multiple regions. This centralized control allows engineers to examine live video feeds from various test locations, providing a clear view of performance across different networks, devices, and environments. This combination of real-world testing and remote control ensures that broadcasters and service providers gain actionable insights without being physically present at each test site.
Witbe’s customers include major broadcasters and service providers worldwide. Companies such as Cox, Rogers, Orange, and Vodafone rely on Witbe to monitor and improve the quality of their video services.
Measuring Quality of Experience: A Focus on the End User
Witbe’s approach to measuring quality of experience (QoE) centers on how viewers experience the streaming performance. Their technology monitors various metrics that influence the user experience, ensuring that issues like slow load times, video defects, or service interruptions are detected and addressed.
The Witbox captures detailed QoE data including metrics like application load times, initial buffering delays, and video playback stability. Witbe uses a custom-built non-reference metric to measure video quality. Unlike traditional metrics such as VMF or PSNR, Witbe’s model emphasizes visual defects that viewers are likely to notice, such as excessive blurring, macroblocking, or frozen images. This focus on real-world performance helps broadcasters understand how their content actually appears to users.
The platform’s monitoring capabilities also provide detailed insights on availability and performance trends. Figure 3 shows a sample dashboard highlighting test results for a Fire TV Stick 4K streaming Fubo TV during the Super Bowl. The results indicate 100 percent availability with no service interruptions. The data also shows an average load time of 3.479 seconds, a strong VQ MOS (mean opinion score) rating of 4.38, and stable audio performance measured at -35 dB. This level of detail allows engineers to quickly identify and troubleshoot potential issues.
By combining these metrics, Witbe empowers content providers to proactively manage performance and ensure that viewers receive a high-quality streaming experience. This data-driven approach is especially valuable during major events like the Super Bowl, where demand spikes can challenge network stability.
Super Bowl Testing: Real-World Performance Monitoring
Witbe leveraged its technology to monitor the performance of major streaming services during the 2025 Super Bowl. Using Witboxes deployed across North America and Europe, Witbe evaluated how different services delivered the high-profile event.
The testing setup included Witboxes placed in New York, San Francisco, Montreal, and Paris, connected to various devices, including Fire TV 4K, Apple TV 4K (Gen 2), Roku 4K, Samsung Smart TVs, Spectrum cable boxes, Verizon cable boxes, and satellite set-top boxes. This variety ensured that Witbe could measure performance across a wide range of viewing conditions and platforms. Figure 4 shows a visual representation of these deployment locations and tested devices.
To collect detailed performance data, Witbe logged in and out of each service every 15 minutes during the Super Bowl, enabling Witbe to track application load times, initial buffering performance, and service availability throughout the event. By repeatedly starting fresh sessions, they could measure real-time changes in network performance and identify potential congestion or service instability.
Witbe enabled YouTube TV’s experimental “Broadcast Delay” feature on supported devices to observe its impact on latency and overall performance. This approach ensured that Witbe’s data captured both steady-state performance and fluctuations that could affect viewers during peak moments like halftime.
Witbe employed automated latency tracking using TAG Video Systems’ real-time Content Matching technology. This system matched video frames across different platforms to calculate precise latency. Witbe’s Virtual NOC further enabled engineers to monitor performance remotely, ensuring comprehensive coverage across platforms and geographic regions.
The combination of real-device monitoring, automated latency measurement, and Witbe’s observability platform, Smartgate, ensured a detailed analysis of the Super Bowl’s streaming performance. This comprehensive approach confirmed that real-time testing on actual consumer devices is crucial for understanding how viewers experience live events — particularly as OTT services continue to push the boundaries of low-latency streaming.
Key Findings and Results from Super Bowl Testing
Witbe’s Super Bowl testing showed that OTT streaming services achieved record-low latency, with some platforms even outperforming traditional cable broadcasts — a significant milestone in the industry (Figure 5).
To better categorize performance, Witbe grouped services into three latency tiers, which are color coded as green, yellow and red in Figure 5.
- Faster than cable broadcast, representing an ideal experience with minimal delay.
- Less than 10 seconds behind the cable broadcast was considered acceptable unless viewers were near loud sports bars or neighbors celebrating early.
- More than 10 seconds behind cable, with some services reaching 40+ seconds — a risk for viewers trying to avoid spoilers.
Tubi emerged as one of the top performers, outperforming the reference cable broadcast (Spectrum) by 2.6 seconds when streamed on Fire TV. Tubi on iOS over 5G was an additional second ahead, but since Witbe’s focus was on large-screen performance, that result was excluded from the final analysis.
Seeing the latency discrepancies, Jan asked if Witbe knew which ABR technologies each broadcaster was using. In response, Yoann explained, “We did not capture the traffic itself during this particular test. But based on what we know from the industry, it’s likely that platforms delivering faster streams used optimized MPEG-DASH configurations, while others using HLS may have experienced more delay.”
As a follow up, Jan asked if services with higher latency delivered improved visual quality. Surprisingly, Yoann responded, “No, not at all. The slower services didn’t show better quality — the higher latency didn’t deliver any noticeable visual improvement.”
Disparity in Device and Service Performance
Platform and device combinations had a notable impact on results. While Tubi excelled on Fire TV, it showed slightly higher latency on Apple TV, likely due to differences in streaming protocols and packager configuration choices. Tubi on Samsung Smart TVs experienced even greater delays, reinforcing the importance of testing across multiple device types.
When Jan asked why performance varied so dramatically between devices, Yoann explained, “It’s largely about application behavior and hardware differences. Smart TVs are particularly challenging because their software environments are fragmented. Even between models from the same brand, the way applications run can vary a lot. Devices like Fire TV and Apple TV are generally more consistent.”
Witbe’s testing also highlighted performance challenges during peak traffic moments. During the halftime show, both DAZN and Tubi on Samsung TVs experienced loading issues, emphasizing the strain high-traffic events can place on streaming platforms (and how the halftime show is often more popular than the game itself). Similarly, Fubo TV’s latency increased from an average of 3 seconds to 5.5 seconds during this period, though the overall quality remained strong.
When Jan asked about timing drift — the variation in timing between devices running the same service — Yoann responded, “We measured drift very carefully, and it was minimal — usually within one to two seconds between devices running the same service. That’s well within what’s expected for live streaming.”
Upscaled 1080p 4K Quality
Jan also asked about quality, which was a concern because Fox’s broadcast was widely reported as 1080p content that had been upscaled to 4K. When Jan asked how this influenced Witbe’s results, Yoann shared, “We did measure the quality and availability. We did see very good quality, even though the signal was 1080p upscaled to 4K. It was indeed a very good success for them.”
This result highlighted that strong visual performance is possible even with upscaled content, reinforcing that bitrate management, encoding quality, and delivery methods are just as critical as the source resolution itself.
The combination of real-device monitoring, automated latency measurement, and Witbe’s observability platform, Smartgate, ensured a detailed analysis of the Super Bowl’s streaming performance. This comprehensive approach confirmed that real-time testing on actual consumer devices is crucial for understanding how viewers experience live events — particularly as OTT services continue to push the boundaries of low-latency streaming.
Key Takeaways and Industry Implications
Several important themes emerged from Witbe’s testing and broader industry experience.
Device support proved to be a significant performance factor. Variations in device capabilities significantly affected results, particularly with fragmented platforms like smart TVs. Devices like Fire TV and Apple TV delivered more consistent performance, while smart TV applications — which often vary widely in software support — showed greater instability.
The testing also showed that low latency doesn’t always come at the cost of quality. Witbe observed that services like YouTube TV managed to combine low latency with stable performance during the Super Bowl, proving that with the right configurations, streaming platforms can deliver fast and reliable service simultaneously.
Witbe’s results further reinforced the importance of testing on real devices. Simulated environments often miss real-world variables such as device behavior, app performance, and network conditions. By monitoring actual consumer hardware, Witbe provided insights that lab-only testing would have overlooked.
When Jan asked how Fox should feel about the results — given that Fox didn’t commission Witbe to conduct the tests — Yoann responded, “Celebrate, right? No, really. I mean, it’s impressive. Whatever they did on their end, they did a great job. It was a success.”
For broadcasters and streaming platforms, these insights emphasize the value of proactive testing on real devices to ensure consistent performance — especially during high-stakes events like the Super Bowl.
Super Bowl Wrap Up
Witbe’s unique approach to QoE testing — focusing on real devices, real conditions, and detailed performance metrics — provides crucial insights for broadcasters and service providers. Their Super Bowl testing highlights how services can balance low latency with quality while navigating the complexities of device variability.
Witbe at NAB 2025
At NAB 2025, Witbe will showcase advancements in its monitoring technology, focusing on its Virtual NOC (Network Operations Center) solution. This tool enhances remote troubleshooting and performance analysis by giving engineers access to real devices from anywhere in the world.
During the interview, Yoann explained that the Virtual NOC aims to “close the gap between Operations, Field Ops, and development teams” by allowing the right people to access the right devices without the complexity of shipping hardware or relying on VPN setups. He highlighted how this solution makes it faster and easier to reproduce issues, test solutions, and improve services.
Witbe will also highlight improvements in its Witbox technology, designed to make it “more simple, easy to use, and very fast.” This streamlined solution ensures minimal latency when engineers take control of devices remotely, providing immediate access to troubleshoot issues in real time.
Yoann emphasized that while Witbe’s Virtual NOC is built for comprehensive monitoring, it’s designed for a manageable number of users — “hundreds of users, not millions” — reinforcing its focus on targeted troubleshooting for critical teams.
