Streaming Learning Center Where Streaming Professionals Learn to Excel
Related Articles
This article is derived from a lesson in Streaming Media 101: Technical Onboarding for Streaming Media Professionals. If you’re looking for an efficient way to get up to speed on key streaming terms, technologies, workflows, and best practices, check out the course here.
Companies distribute videos for a myriad of reasons; sometimes as the product, sometimes to sell or market another product, sometimes for training or communications. Whatever the motivation or business model, your videos are most effective when the viewing experience is favorable. Factors like slow start times, buffering, poor-quality videos and player crashes diminish the user experience and decrease viewing time, both for that video and your platform as a whole.
Most conversations about the user experience center around two terms, QoS, or Quality of Service, and QoE, which is Quality of Experience. While related, these are two separate concepts that are measured and utilized in totally different ways.
This article presents a high-level view of QoS and QoE. I’ll start by defining both and discuss how they inter-relate. Then we’ll briefly look at how companies measure and use QoS and QoE.
Contents
What are QoE and QoS?
So what are QoE and QoS? According to the International Telecommunications Union standards body (ITU), QoS is the “totality of characteristics of a telecommunications service that bear on its ability to satisfy stated and implied needs of the user of the service.” Under this definition, QoS measures the performance of the delivery infrastructure, including third-party or internal content delivery networks (CDN), usually tracking data like overall throughput, latency, error rates, and cache hit ratio.
In contrast, the ITU defines QoE as “the overall acceptability of an application or service, as perceived subjectively by the end-user. It includes the complete end-to-end system effects (client, terminal, network, services infrastructure, etc.) and may be influenced by user expectations and context.” So, QoE measures the user experience directly, including factors like playback success, startup time, rebuffering, and visual quality, both encoding and decoding.
Figure 1 shows how QoS and QoE interrelate. As you see, on the left is content preparation, through encoding and packaging. Then the files are handed off to the delivery infrastructure for distribution, which QoS measures. Once received at the viewing location, the video is decoded and watched on a player. As the figure shows, QoE involves the complete end-to-end experience, while QoS is the distribution portion that sits in the middle.
Clearly, network performance is critical to good QoE. But so are other factors like source quality, encoding quality, packaging integrity, as well the video viewing environment. it’s one thing to create content that looks great on a smartphone; quite another to produce and deliver content that rocks an 85” LED in your living room.
You can have great QoS but poor QoE, for example, if the quality of the source material was sketchy or if the player doesn’t switch to the highest quality ABR stream because of faulty logic. You can also have high-quality QoE and poor quality QoS, for example, where the network is fast enough to deliver the video on time but caching performance is poor, which increases bandwidth and distribution costs. This means that to get a complete picture you have to measure both QoS and QoE.
How do you Measure QoS?
So how do you measure QoS? Some systems install probes on the network between the systems that you want to monitor. These probes are hardware devices or software programs installed in service centers around the Internet that monitor traffic from a defined source or sources to a destination or destinations. In Figure 2, showing Telestream’s Surveyor product, the probes are represented by the surveyor theodolite (tripod-thingies) between the various locations.
So, if you wanted to monitor performance between the content prep headend and the core network, you would install a probe at the core network. If you wanted to monitor the streaming service between the core network and the edge, you would install a probe at the edge. These probes allow you to identify the location of the delivery issues and therefore the source.
Other systems track QoS by installing software in the player itself which is the most common deployment schema for QoE (as you’ll learn in a moment).
What QoS metrics should you track? This will vary by product and service provider but metrics like throughput, bitrate, latency, jitter, and packet loss are very commonly tracked and compared by various vendors.
How is QoE Measured?
QoE is usually measured by plug-ins in the player that report performance data back to the central database for analysis and presentation (Figure 3). The method of data extraction and data tracked varies by service provider.
Figure 3. NPAW’s YOUBORA plugin tracks the playback experience.Two organizations have weighed in on the key data points to track. According to the Streaming Video Alliance, the key metrics are video start time, re-buffering ratio, average media bitrate, and video start failures (Table 1).
The Consumer Technology Association recommends monitoring playback failures, startup time, playback stalling, bitrate, player failures, and other metrics, including some related to advertising insertion and many other player and video playback data points like player width and video resolution. Anyone trying to derive a set of data points to track should review both documents.
What Functions Do QoS/QoE Systems Perform?
What functions do QoS and QoE systems perform? One function is to alert when scores drop below a certain level. You see this in Figure 4 which shows TestTree’s OTT Monitoring Probe. This view allows a technician to track all available channels and instantly identify when performance drops below a certain threshold, as you can see in the red alerts in the bottom two rows. Usually, these alerts can trigger an email or text so you don’t have to be monitoring to learn of the problem.
Once alerted, the service provider can diagnose and resolve the problem before a lot of customers are having bad experiences, perhaps by routing the data around the choke point by using a different CDN. Many QoS/QoE systems can automatically switch CDNs based upon performance data as well as enable manual switching.
At a different level, QoS and QoE systems provide the data necessary to compare different products or service providers. For example, if you’re using several different CDNs you can track any number of key performance indicators like throughput achieved by each and the average bitrate delivered. You can use this data to compare the price/performance of the different vendors, identify potential problems, and/or control which customers are assigned to each service provider.
Many larger QoE and QoS service providers can also provide industry-specific data so you can compare your company’s performance against others distributing the same kind of content in the same geographical regions. This data is obviously invaluable when assessing your competitive strengths and weaknesses and opportunities to improve your service.
As I said at the start, however you monetize your video streaming, it’s most effective when the viewing experience is positive. The only way to ensure this is to track and monitor QoS and QoE data relating to your video deliveries.
