Lesson of the Week: Computing VMAF with FFmpeg on Windows

A previous blog post entitled Compute VMAF Using FFmpeg on Windows covered this same topic but with a version of FFmpeg compiled by my colleague Abharana (Abi) Bhat, Ph.D., who is the Video Architect, Video Quality at DAZN, the London-based OTT subscription sports streaming service. Now the standard Windows FFmpeg download from Gyan.dev includes VMAF; all you have to do is download the models and run the proper command strings.

So, in this post, I’ll detail how to download those models and how to compute VMAF and other metrics with FFmpeg on Windows. I’ll assume that you know how to download FFmpeg from the gyan.dev site and insert the FFmpeg path into your Windows environment.

Once FFmpeg is Installed and In Your Windows Path

Now you have to download the VMAF model that you want to use. Currently, there are three:

  • The default model – vmaf_v0.6.1.pkl, which is “trained to predict the quality of videos displayed on a 1080p HDTV in a living-room-like environment.”
  • The 4K model – vmaf_4k_v0.6.1.pkl, “which predicts the subjective quality of video displayed on a 4KTV and viewed from the distance of 1.5 times the height of the display device (1.5H).”
  • The anti-hacking model – vmaf_v0.6.1neg.pkl, where “neg” stands for “no enhancement gain.” This reverses out techniques like “sharpening, contrasting, histogram equalization, among others, could boost VMAF scores. While such operations could enhance the image quality as perceived by the end viewers if applied properly, it is evident that VMAF tends to overpredict the perceptual quality even when such operations are overused.”

Note that the VMAF phone model isn’t a separate model but a mode you can run with any of the models shown above. You’ll learn how below.

Here’s how you download the models and the associated .pkl files.

1. Go to the Netflix GitHub model page here.

2. Click the desired model to download. In this case, I’m choosing the .pkl file for the default model. This takes you to another GitHub page for that model. On the top right (partially hidden), right-click the Raw button and choose Save link as.

3.  This bit is important. Save the file in the same bin as ffmpeg.exe. If you don’t do this, the command strings shown below won’t work. For me, this is always c:\ffmpeg\bin as shown in the figure below. Note that you can save these models anywhere you’d like, but you’ll have to adjust the command strings below for that custom location. 

4.  Once you get the .pkl file, follow the same procedure for the .model file. Again, if you don’t get this file, and save it in the c:\ffmpeg\bin folder, the command strings shown below won’t work.

5.  Download the .pkl file and .model file for each model that you want to run.

OK, now let’s move onto the command strings.

Creating the Command Strings

Here’s a very simple command string:

ffmpeg.exe -i output.mp4 -i input.mp4 -lavfi libvmaf="model_path='C\:\\ffmpeg\\bin\\vmaf_v0.6.1.pkl':log_path=VMAF.txt" -report -f null -

Pretty simple stuff;

ffmpeg.exe – call the program
-i output.mp4 -i input.mp4 – list encoded file first, then source
lavfi– call librafilter
libvmaf – Identify the filter
"model_path='C\:\\ffmpeg\\bin\\vmaf_v0.6.1.pkl' – path to VMAF model (this is the default model). I’m not quite sure why you need the double backslashes, but I got the command string from the engineer who is creating the Zeranoe FFmpeg builds, and it works. You also need the single quote at the start and end of this path.
:log_path=VMAF.txt – path and name of the file to contain scores
-report – saves the log file (not essential)
-f null - – Tells FFmpeg to output a null file

Note the single and double quotation marks, which must be properly placed or the command string won’t work. So:

  • First double quote before model_path
  • The first single quote before the path to the model
  • The close single quote at the end of the path
  • The close double quote at the end of the libvmaf filter commands, in this case, the log path.

Other Options

You can find all the libvmaf options here. Here are the most relevant ones (IMHO)

model_path – Set the model path which is to be used for SVM. Default value: “vmaf_v0.6.1.pkl”

log_path – Set the file path to be used to store logs.

log_fmt – Set the format of the log file (xml or json).

phone_model – Invokes the phone model which will generate VMAF scores higher than in the regular model, which is more suitable for laptop, TV, etc. viewing conditions (phone_model=true to enable; false is the default).

psnr – Enables computing psnr along with vmaf.

ssim – Enables computing ssim along with vmaf.

ms_ssim – Enables computing ms_ssim along with vmaf.

n_threads – Set number of threads to be used when computing vmaf.

pool – Set the pool method to be used for computing vmaf. Options are minharmonic_mean or mean.

This string incorporates many of these options:

ffmpeg.exe -i output.mp4 -i input.mp4 -lavfi libvmaf="model_path='C\:\\ffmpeg\\bin\\vmaf_v0.6.1.pkl':log_fmt=xml:psnr=1:ssim=1:phone_model=true:log_path=harm_mean.txt:pool=harmonic_mean" -f null -

So, it computed VMAF using the phone model, as well as PSRN, SSIM, and MS_SSIM, and computed the scores via the harmonic mean, which does a better job weighing variations in the stream than the simple mean.

Here’s what the log file looked like (click the figure to see it at full resolution). You see the computed metrics and the frame scores for the first two frames. It took 13.5 seconds to compute all metrics on my HP Zbook notebook.

For those who care about these things, I checked and the VMAF scores computed via the arithmetic mean were identical to those produced by the Moscow State University Video Quality Measurement Tool, which is my go-to tool for metric computations.

About Jan Ozer

I help companies train new technical hires in streaming media-related positions; I also help companies optimize their codec selections and encoding stacks, and evaluate new encoders and codecs.

Check Also

Streaming Learning Center Releases Streaming Media 101: For Sales and Marketing Professionals

The course teaches the technology and market knowledge necessary to effectively market and sell streaming-related …

5 comments

  1. Hey Jan,

    Lot’s of folks using Macs for this type of stuff, including me. Especially true now that Apple has been releasing more powerful Macs over the last few years.

    Maybe some year when you are doing well you can get a pre-owned Mac?

    Hope you are well and good,
    Robert
    Robert A. Ober
    IT Consultant, Vidcaster, & Freelance Preditor(Producer/Editor)
    http://www.infohou.com
    Houston, TX

    • Rob:

      I’ve got a pre-owned Mac, but no one cares about how FFmpeg works on a used MacBook Pro.

      Probably expand into Linux/Ubuntu via cloud machines rather than Mac, unless Apple drops one in my lap (which last happened in 2009 so I’m not holding my breath).

      Thanks for writing in and take care.

      Jan

  2. I’m not so sure. Depends on how old it is. Without testing on Mac you are losing a lot of potential viewers.

    Robert

Leave a Reply

Your email address will not be published. Required fields are marked *