Promising Initial Results with AV1 Testing

AV1 encoding is now only 2x slower than x265. See the latest results here. 

I began testing AV1 early this week. Briefly, my tests involve 16 ten-second clips in four genres (movies, sports, animations, gaming) and an “other” category (music video, nature video). I’ve completed the first set of tests with FFmpeg 4.3, benchmarking x264, x265, and the latest version of the Alliance for Open Media AV1 codec, now at version 2. I wanted to publish these results and command scripts for comments ahead of the testing I’ll be doing next week of the Intel and Visionular AV1 codecs.

Here’s the sexy stuff, first the rate-distortion curve for VMAF quality that’s presented above as the featured image. AOM AV1 in blue is the clear leader over HEVC in green, with x264 well below.

And then the BD-Rate computations that put numbers to the curves. Pick a row and then scan through the columns. The AV1 row tells you that you can produce the same quality as x264 at a 48.66% lower bandwidth and the same quality as HEVC at a 16.22% lower bandwidth. With x264, you’d need to boost your bandwidth by 94.79% to match AV1 quality and 64.10% to match HEVC quality. With x265, you’d need to boost bandwidth by 19.36% to equal AV1 quality and can produce the same quality as x264 with a bitrate reduction of 39.06%

The last time I benchmarked AV1 quality was in my Streaming Media First Look here. In that article, I found that AV1 had a BD-Rate advantage of 18.18% over HEVC and 50.4% over H.264, so the quality numbers haven’t really changed since then.

Command Strings

Some details about the encodes:

• I encoded first with the AV1 codec targeting data rates that resulted in VMAF scores ranging from about 83 to about 94 which is the most relevant range for the top quality file in the encoding ladder. Then I encoded the other two codecs at those data rates.
• To produce the command strings, I tested with all available tuning mechanisms and used the tuning mechanism which produced the highest VMAF score.
• I used a two-second GOP size for all encodes and a closed GOP.
• I used the AV1 -cpu-used setting of 3 as offering the best balance of performance and quality. See here for more details. I used veryslow for x264 and x265 because placebo almost never delivers any additional quality over very slow and dramatically increases encoding time, so is almost never used by actual producers.
• Encoding time for the files produced by these strings on my HP Z840 was 2:09 for x264 (min:sec), 2:18 for x265 (checked it twice), and 50:43 for AV1. While the quality is good, it will cost you 25x to encode using the Alliance for Open Media’s AV1 codec, making it best suited for producers with lots of eyeballs where the bandwidth savings can offset the additional encoding costs.

Here are the command strings.

AV1

ffmpeg -y -i Football_10.mp4 -c:v libaom-av1 -strict -2 -b:v 2700K -g 60 -keyint_min 60 -sc_threshold 0 -row-mt 1 -tile-columns 1 -tile-rows 0 -threads 8 -cpu-used 8 -pass 1 -f matroska NUL &
ffmpeg -y -i Football_10.mp4 -c:v libaom-av1 -strict -2 -b:v 2700K -maxrate 5400K -bufsize 5400k -g 60 -keyint_min 60 -sc_threshold 0 -row-mt 1 -tile-columns 1 -an -tile-rows 0 -threads 8 -cpu-used 3 -pass 2 Football_4.mkv

x264

ffmpeg -y -i Football_10.mp4 -c:v libx264 -b:v 2700K -preset veryslow -g 60 -keyint_min 60 -sc_threshold 0 -tune ssim -pass 1 -f mp4 NUL &
ffmpeg -i Football_10.mp4 -c:v libx264 -b:v 2700K -maxrate 5400K -bufsize 5400k -preset veryslow -g 60 -keyint_min 60 -sc_threshold 0 -tune ssim -pass 2 Football_x264_ssim_4.mp4

x265

ffmpeg -y -i Football_10.mp4 -c:v libx265 -preset veryslow -tune ssim -x265-params bitrate=2700:keyint=60:min-keyint=60:scenecut=0:open-gop=0:pass=1 -f mp4 NUL &
ffmpeg -y -i Football_10.mp4 -c:v libx265 -preset veryslow -tune ssim -x265-params bitrate=2700:vbv-maxrate=5400:vbv-bufsize=5400:keyint=60:min-keyint=60:scenecut=0:open-gop=0:pass=2 Football_x265_ssim_4.mp4

Perspective

For perspective, x265 is not the highest quality HEVC codec and libaom, AOM’s AV1 codec, is likely not the highest quality AV1 codec, either. You can see this in my Streaming Media article here and in the chart below. So, the results above are probably representative if you’re using FFmpeg-based codecs, but not if you source an alternative codec.

As mentioned above, I’m happy to report that my Streaming Media comparison will include the Visionular Aurora AV1 encoder which I’m just about to dive into. I’ll also be looking at the AWS Elemental and Intel SVT AV1 codecs.

If you have any questions or comments, please send them to me at [email protected]. I’d love some feedback while there is still time to make some changes.