Bitrate Control and QoE-CBR is Better

Home/Articles/Bitrate Control and QoE-CBR is Better
By | 2017-03-27T21:04:40+00:00 August 7th, 2016|Articles|Comments Off on Bitrate Control and QoE-CBR is Better

Summary (The MPD)

When distributing video under constrained conditions, the bitrate control technique used to encode the files can have a profound impact on the quality of experience (QoE). Specifically, under some conditions, CBR-encoded files deliver a superior QoE to files encoded using 200% constrained VBR, while also reducing the overall bandwidth delivered.

For producers concerned with the potential for transient quality issues with CBR-encoded files, 110% constrained VBR avoids these transient quality issues and the QoE-related problems experienced with 200% constrained VBR. Producers currently distributing files encoded using 200% constrained VBR should consider running trials with CBR-encoded files to determine if this improves QoE.

In addition, the initial file played during HLS playback can have a profound impact on initial QoE. Optimal playback is achieved when the highest rate the playback device can sustain is played first. If it’s not possible to determine this, choose the highest quality “safe” rate, but avoid a situation where the majority of your audience will have to quickly switch down after attempting to play the highest quality stream.

This video shows an introductory video describing the test file and test procedure for tests run at 3200 kbps. Click over to YouTube to view the video in full screen.

 

 

Segment 7: Test Results – JWPlayer

With the VBR file, JWPlayer experienced one four-second audio/video stoppage during the first switch from talking head to ballet (same place as Safari), and one other slight audio stoppage, but no loss of synch. Fragments downloaded during playback are shown in Figure 9. JWPlayer spent significant time in the relatively low quality layer 4, but only switched streams four times (five index files less one). CBR playback was very much like HLS with the vast majority of the time spent in layer 2, and two apparent stream switches (three index files minus one).

vbr_cbr8.png
Figure 8. Fragments retrieved by JWPlayer (VBR on left, CBR on the right). Note that there at five index files in the left and only three on the right.

Table 2 shows the numbers. Again, VBR was much less efficient from a fragment retrieval perspective (ten more than were necessary), but since the majority of the time was spent in layer 4, less data was actually retrieved during CBR playback than during VBR playback. Looking at the table, it’s easy to see that the overall QoE delivered by the CBR streams was much, much higher.

QoETable2.png

Table 2. How the numbers looked for JWPlayer for VBR/CBR playback.

You can watch the JWPlayer tests performed at 3200 kbps in this video. Click over to YouTube to view the video in full screen.

Segment 8: Throttling at 4500 kbps

In these tests, the first file listed in the .m3u8 file was the 3100 kbps file, so this was the first file retrieved by both players. The throttle in Charles was set to 4500 kbps.

In VBR playback tests, both the Safari player and JWPlayer experienced complete stoppages of about six seconds just after the first ballet sequence appeared 30 seconds in, with a shorter audio break a few moments later. Both players lost audio synch for the rest of the file playback. Since the problems were nearly identical in both clips, we played the HLS streams from the index files (http://doceopub.com/VBR/Layer_2/index2.m3u8, http://doceopub.com/VBR/Layer_3/index3.m3u8) in Safari to determine if the problem existed in the packaged files, but the clips played normally. This proves that the problems were caused by the lack of ability to retrieve the VBR segments in time to continue smooth playback.

As you can see in Table 3, during VBR playback, both Safari and the JWPlayer retrieved segments in three layers, dropping down as low as the 1500 kbps stream. The retrieved index files in both VBR tests showed four index files, indicating that both players switched streams three times.

QoETable3.png

Table 3. Playback with the throttle set at 4500 kbps. Click the table to view it at full resolution in another browser window.

In contrast, CBR playback with both players was as efficient as possible; both players started with the 3100 kbps file, and remained at the level the entire time. Taking into account the stoppages, loss of synch and switches experienced in the VBR tests, it’s safe to say that the quality of experience produced by the CBR files was superior.

You can watch the Safari tests performed at 4500 kbps here. Click over to YouTube to view the video in full screen.

You can watch the JWPlayer tests performed at 4500 kbps here. Click over to YouTube to view the video in full screen.

Segment 9: Other Tests

For the record, I tested a number of times before arriving at the final test configurations, adjusting the encoding ladder, the initial stream called by the HLS player, and the throttling level, among other factors. In general, the QoE results were consistent with those discussed here, with CBR showing fewer stoppages and less switching.

A summary of these tests is shown in Table 4, which do not include the results presented above. As you can see, on average, with the Safari HLS player, CBR was 17% more efficient than VBR for both the number of fragments retrieved and total bandwidth retrieved. This number jumps to a 25.65% bandwidth saving for JWPlayer when deploying CBR over VBR. We expected that CBR would improve QoE under constricted playback conditions, with the bandwidth saving an unexpected benefit.

QoETable4.png

Table 4. Summary of other test results.

Segment 10: But What About CBR Quality?

As we have written about previously, most notably in a Technical Brief entitled, Switch from CBR to VBR to Improve Overall Quality and Avoid Transient Quality Issues , CBR-encoded video often exhibits transient quality issues as compared to VBR. With this particular test file, the 1080p VBR version had a slightly higher PSNR value than the CBR file (41.1 dB compared to 40.7 dB) a qualitative difference that would not be visible to the normal viewer.

As shown in Figure 9, the Moscow University Video Quality Measurement Tool revealed no transient quality issues with the CBR file (in red), which exhibited slightly higher quality in the talking head sequences (where the data rate was higher than VBR) and slightly lower in the ballet sequences (where the data rate was lower). Transient quality issues would appear as dramatic drops of the red results; for example the drop to 36 dB on the lower right in the last clump of results (we checked the frame and the difference wasn’t visible).

quality.PNG

Figure 9. PSNR comparisons shown no transient quality issues in the 1080p version of the test file.

In the aforementioned technical brief, we state that “Producing using 110% constrained VBR seems to avoid these quality issues without introducing significant data rate variability.” To test how 110% constrained VBR would perform under the conditions tested here, we produced a 110% constrained version that’s available at http://www.doceopub.com/110_CVBR/Playlist.m3u8 (we did not create a JWPlayer version). As shown in Figure 10, while the differences between the talking head and ballet segments are not as stark as those shown in Figure 4 for the 200% constrained VBR clip, they are significant. The average size of the talking head segments was 2.7 MB, compared to about 4.2M for the ballet clips, a difference of over 50%.

vbr_cbr10.png

Figure 10. The frame profile for the 110% constrained VBR file.

The VBR playback experience constrained to 3200 kbps is shown on the left in Figure 11, with most of the time spent in Layer 2, but seven segments spent in layer 3, which the CBR file did not experience. With five index files, it appears that Safari changed streams four times, as compared to two stream changes for the CBR file.

vbr_cbr11.png

Figure 11. 110% constrained VBR on the left, CBR on the right.

On the other hand, there were no stoppages during the playback of the 110% constrained VBR-encoded file, or loss of synch, so the major elements that detracted from QoE were not present in the 110% constrained VBR file. If you’re concerned about transient quality issues in CBR files (as you should be) 110% constrained VBR may be the best alternative.

Segment 11: Conclusions

What conclusions can we draw from this data? Obviously, the test file presents close to the worst-case analysis, and the constraint imposed by Charles is likely much more absolute than will be experienced in the real world. So it’s hard to shout that the sky is falling, and that those distributing at 200% constrained VBR (or similar) are significantly impacting their QoE. Still, we learned that:

1.Under some constrained playback conditions, files encoded using 200% constrained VBR are more likely to experience stoppages, audio breaks, and loss of synch, and to switch layers more often than CBR files. Under these playback conditions, files encoded using 200% constrained VBR deliver a degraded QoE as compared to CBR.

2.Under the same conditions, the stream switching necessitated by the VBR bitstream may result in more segments being downloaded, and increased bandwidth consumption.

How to apply this? In essence, the issue comes down to this; does encoding using 200% constrained VBR produce enough extra quality over CBR or 110% constrained VBR to outweigh the risks of QoE issues like those discussed above? As shown in Table 5, the answer is probably not.

This table shows six test files encoded to 1080p resolution using the encoding techniques listed atop each column. Data rates vary by file as we’ve used per-title encoding to compute the appropriate data rate for each file.

QoETable5.png

Table 5. PSNR quality for files encoded using different bitrate control techniques.

The table shows PSNR values, where higher values are better, with the highest value in green and the lowest in red. The Total Quality Delta shows the difference between the highest and the lowest value, while the Delta 110% to 200% column shows the quality difference between files encoded using 200% constrained VBR, and 110% constrained VBR. In only one instance, Big Buck Bunny, does this value exceed 4%, and even 4% likely would not be noticeable by even the most golden-eyed viewer. In other words, it’s highly unlikely that using 200% constrained VBR over 110% constrained VBR would noticeably improve QoE.

As you know, QoE has two components, file quality and delivery quality. As we’ve seen, under some conditions, 200% constrained VBR can seriously degrade delivery quality and overall QoE. Accordingly, any files that may be delivered over constrained connections, like mobile, should be encoded using 110% constrained VBR, or CBR. If you’re creating one set of files for all of your targets, you should strongly consider using 110% constrained VBR.

3.If you’re already producing files using 200% constrained VBR, you should consider testing files encoded via CBR or 110% constrained VBR in either a split or sequential test. All analytics packages are different, but if you can isolate the layers played during VBR and CBR playback, or stoppages or other issues experienced, you should be able to tell if encoding in CBR or 110% constrained VBR does result in less stream switching than 200% constrained VBR files, fewer stoppages, and/or less overall bandwidth for each video play.

Segment 12: Ancillary Conclusion: Customize Your M3U8 files (or MPD Files) To Match Sustainable Bandwidth

As mentioned above, the stream listed first in the M3U8 file is the stream played first by the HLS player. Our tests revealed that choosing the right stream could have a dramatic impact on the efficiency and initial QoE of video playback. This is shown in Figure 12. Specifically, Figure 12 shows the segments pulled during HLS playback of the 110% constrained VBR file in the Safari browser with bandwidth constrained at 3200 kbps. On the left, the initial file in the M3U8 file was layer 0, or the 4500 kbps file. On the right, the initial stream is set to layer 2. In both streams, we recorded until the eighth segment started downloading.

vbr_cbr12.png

Figure 12. Playback of the 110% constrained VBR stream with layer 0 called first on the left, and layer 2 called first on the right.

On the left, with layer 0 called first, Safari knew immediately there was a problem, and started retrieving layers as low as layer 6 to address it. The viewer sees a quick panoply of multiple quality levels, finally settling in at layer 2 about 36 seconds in. During this scramble, Safari retrieved 16 segments to play the first eight, which obviously wastes bandwidth.

On the right, with layer 2 the initial stream, Safari immediately settles into smooth playback, and downloads eight segments to play eight segments.

Most encoders like the Capella Systems Cambria encoder used in our tests, create the .M3U8 file based upon the order of layers in the encoding preset. If you build your encoding ladder from the top down, like we did, the highest quality file is listed first. If you build from the bottom up, the lowest quality stream is listed first. Seldom, if ever, are either of these choices the correct one.

For desktop or OTT playback, the initial stream should be a reasonably high quality stream, but the not the highest; perhaps the lowest quality 720p stream (see Figure 2). That way, quality starts at a good level, and playback quality should improve for most viewers. For mobile, the initial stream should be lower quality, perhaps the lowest quality 360p stream. What you want to avoid is a high bitrate stream that will push the player into panic mode, resulting in the low QoE and inefficient playback shown on the left in Figure 12.

About the Streaming Learning Center

The Streaming Learning Center is a premiere resource for companies seeking advice or training on cloud or on-premise encoder selection, preset creation, video quality optimization, adaptive bitrate distribution, and associated topics. Please contact Jan Ozer at jan@streaminglearningcenter.com for more information about these services.

 

Comments