How PSNR Compares to VMAF and SSIMplus

As quality metrics go, PSNR is considered a blunt tool obsoleted by higher end metrics like Netflix’s VMAF or SSIMwave’s SSIMplus. That said, it’s accessible and understood, and has been used for years. Of course, none of that matters if it delivers misleading results.

So that’s what I decided to test in a round of two tests. This is the first, which includes H.264 configuration options like those I cover in my recent book, Video Encoding by the Numbers, which relies very heavily on PSNR results. The second will see how the metrics compare when evaluating the quality of codecs like x.264, x.265, and VP9.

For this series, I encoded four 720p files in configurations that tested the quality produced by different H.264 profiles, x264 presets, bitrate control techniques, and B-frame intervals. Then I computed PSNR, VMAF, and SSIMplus scores using tools defined in the attached PowerPoint.

I input the results into a table, multiplied PSNR values by 2.25 to get them close to VMAF and SSIMplus, and graphed the results. After a short test description, these graphs are presented in the downloadable PDF accessible below. I am not a math geek, and I know there are mathematical comparisons that I could have supplied. Rather, I decided to let the graphs stand on their own.

Here are the main conclusions from the final page of the PDF.

  • The key question is, how many times would I have reached a different recommendation by using VMAF or SSIMplus. The answer is not that often (and never where both SSIMplus and VMAF agreed).
  • So, for simple configuration decisions, PSNR results were reasonably consistent with VMAF and SSIMPlus. If PSNR is the only tool you have affordable access to, it appears useful for these types of comparisons.

My preliminary peeks at the codec comparisons show that I won’t be able to make the same claims for that analysis.

Here’s the PDF.

PSRN vs. VMAF vs. SSIMPlus

About Jan Ozer

Avatar photo
I help companies train new technical hires in streaming media-related positions; I also help companies optimize their codec selections and encoding stacks and evaluate new encoders and codecs. I am a contributing editor to Streaming Media Magazine, writing about codecs and encoding tools. I have written multiple authoritative books on video encoding, including Video Encoding by the Numbers: Eliminate the Guesswork from your Streaming Video (https://amzn.to/3kV6R1j) and Learn to Produce Video with FFmpeg: In Thirty Minutes or Less (https://amzn.to/3ZJih7e). I have multiple courses relating to streaming media production, all available at https://bit.ly/slc_courses. I currently work as www.netint.com as a Senior Director in Marketing.

Check Also

A real-world comparison of the cost per stream and OPEX associated with different transcoding techniques.

Understanding the Economics of Transcoding

Whether your business model is FAST or subscription-based premium content, your success depends upon your …

ASIC was the overall cost and performance leader.

Comparing CPUs, GPUs, and ASICs for High-Volume Transcoding

This case study compares CAPEX, power consumption, and co-location costs for GPU, CPU, and ASIC-based …

Codec Royalties on Content and the Jaws Moment

I was raised in a beachfront commuter town about 60 miles south of Manhattan and …

Leave a Reply

Your email address will not be published. Required fields are marked *