Streaming Learning Center Where Streaming Professionals Learn to Excel
Related Articles
[Author’s note: This article is from 2001, and some of the original images are gone. I’ve used the thumbnails from the Extremetech article here. Sorry for the low quality. The article is of limited value now, though square pixels continues to be a concept that confuses many producers, and aspect ratio mismatches continue to appear both on TV and on the web].
Some people want to win the World Series. Others want to become President of the United States. I want to be the guy who rid the world of non-square pixel video.
I can see it now, at my 50th high school reunion, the guidance counselor who never thought I would amount to much, ancient at 95 but still feisty, asking “Well, Ozer, what did you do with your life?” smirking in expectation.
I slowly pull myself up to my shrunken 6’1″ and respond haughtily “I, sir, rid the world of 352×240 MPEG-1 files created for playback on computers, and then, just for fun, exiled 720×480 MPEG-2 files. I explained the problems with rectangular pixels and the need for square pixels to the digital video glitterati, a previously inexplicable topic, gaining fame and fortune.”
When the Internet first came along, I banished 176×144 resolution forever, standing up against mighty Microsoft, stamping my feet and shouting:
I do not like it on ABC,
I do not like it on MTV.
I would not, could not on CNN,
I would not, could not on ESPN.
It can’t be 176×144, Bill G.
Because then it isn’t 4:3!
OK, so the vision still is a bit rough, and obviously impacted by reading time with the kiddies. But perhaps somewhat scarily, it’s not too far from the truth, this has been my quest now for several years. Here’s why.
Encoding your digital video files to the wrong video resolution can distort the appearance of your video, plain and simple (and make your boss look like she gained 20 pounds, never a good idea). And because many highly respected programs, like Media 100’s Cleaner, as well as standard tools like Microsoft’s Windows Media Encoder, use incorrect resolutions in their presets, many producers have inadvertently distorted their video.
As we’ll see, the issue first raised its ugly head back in the mid-1990s, when MPEG-1 encoders first hit the market. Unfortunately, the problem expanded into the streaming world, where it impacts many more producers. Then Moore’s law kicked in (duh!) and Pentium computers could play back MPEG-1 without special hardware. Suddenly, the format’s future seemed assured, and MPEG-1 encoders started appearing in droves. As a contributing editor to PC Magazine, I tested most of them.
Before MPEG-1, choosing an output resolution for your video was easy–you used 160×120, 240×180, or 320×240. Virtually all capture cards and video editors supported these resolutions, as did all codecs, and the video looked just like it did if you played your source video back full screen on your television set, (that is, if you could ignore the blocks, streaks and other compression artifacts of the day).
Testing two MPEG-1 encoders back in 1996, using standard analog BetaSP footage, I noticed something funny–playing the two videos back on a computer monitor, they looked like before and after shots for a diet program. One made me look fat. The other made me look skinny (guess which got Editor’s Choice?). Screen shots are shown in Figure 1.
Checking the video resolutions of the two files, I discovered that the file on the left had a resolution of 352×240 (making me looked horizontally stretched), while the file on the right was 320×240 and looked realistic. The reason why is a confusing jumble of computer and television standards that’s resisted simple explanation of years. Here’s my try.
It’s important to recognize that MPEG-1 was designed as a standard for video displayed on television screens. The original formats for MPEG-1 were VideoCD and CD-I (CD Interactive), neither of which succeeded in a big way in the United States, though VideoCD became very popular in the Far East. Both devices were very much like DVD Players, standalone devices that you connected to a television set.
Contents
NTSC Resolution Background
NTSC is the National Television Standards Committee (or, Never the Same Color Twice as the tired joke goes) standard defined in 1953. Characteristics relevant to this discussion include 525-line vertical resolution, 29.97 frames per second, with each frame divided into two fields. D1 was one of the first digital standards for NTSC video, and it was defined as 720×486 pixels, with the same frame rate and field-based schema as analog video. In other words, when digitizing an analog signal into D1 format, the signal is broken into 720 pixels across, 486 pixels north and south. However, this is per FRAME. Since each frame is divided into two fields, one containing uneven lines (1, 3, 5, 7), the other containing even lines (2, 4, 6, 8), each FIELD is really 720×243. As you probably know, your TV set actually displays 60 fields per second, first the uneven lines in the first 60th of a second, then the even lines in the next 60th, and so on. This promotes the appearance of smooth motion with roughly 30 complete frames per second.
Let’s divert for a moment. What’s interesting about the 720×486 format is that if you do the math, the pixels in the frame have an aspect ratio of 4:2.7. (To calculate this, divide 720 by 4 and get 180, then divide 180 into 486 to get 2.7). However, NTSC television has a display aspect ratio of 4:3. That’s why if you measure the active screen on your television set, you’ll get results like 8″x6″ for the 10″ diagonal TV set in the kitchen, and 16″x12″ for the 20″ set in the bedroom. Do the math on both of these sizes, and you get 4:3 (e.g. 16/4 =4, 12/4=3).
So how does a 720×486 format with a frame aspect ratio of 4:2.7 display on a TV set with a 4:3 display aspect ratio? Well, it squeezes each line horizontally by about 11%. That’s why NTSC pixels are said to be rectangular. Similarly, if you display a 720×486 D1 image (or even a 720×480 DV image) on an NTSC television set, it will shrink horizontally by about 10%, and look like it’s about 640×480 in resolution.
Want proof? Figure 2 is a DV frame of my friend Lisa. The frame resolution is 720×480. However, since this image is meant for display on an NTSC screen that would normally squeeze it by 11%, Lisa looks fat when all pixels are displayed on a computer screen.
Now let’s look at the same frame displayed on a television monitor, using the S-Video output from the DV camera, and shot with a digital camera. This is Figure 3. (Sorry about the quality, but I couldn’t get the editors to spring for a digital television set.)
Basically, what we did in Figure 3 was crop out the actual TV screen from the digital photograph (you can see the rounded corners of the TV tube in all corners). The cropped image was naturally at the 4:3 display aspect ratio (since it was a picture of an NTSC TV set, which displays in 4:3) and we scaled the cropped image down to 640×480 for comparative purposes. Here’s a comparison of Lisa’s face from the two images.
As you can see, Lisa’s face is noticeably wider in the DV image than the TV image. Why? A couple of reasons.
First, the DV frame contains every pixel of the frame. When displayed on the TV set, however, the outside edges of the original image are not seen, which is called overscan. If you’re ever added titles in Premiere, you’re used to seeing squares inside the titling utility to keep you from placing titles too close to the edge where they might be eliminated in the overscan (see Figure 5).
If you compare the DV frame with the TV image, you’ll see regions on all edges that are visible in the DV frame, but not visible in the TV image. That’s why Lisa’s face looks longer in the TV image, because fewer horizontal lines were displayed in the visible viewing area on the TV set (with others subsumed in the overscan).
But she’s OK with that, since it makes her look skinnier.
What frosts her is that the DV image looks fatter when displayed on a computer monitor, because the anticipated squeezing of the horizontal lines that occurs when outputting to NTSC doesn’t happen. And the reason she looks thinner on the TV, as we’ve been discussing, is because the DV frame is squeezed by about 11% when displayed on the TV set– that whole rectangular pixel thing. But on the square pixel computer screen, her face looks about 11% larger –pretty much the same effect we saw with the two MPEG-1 files on figure 1.
To recap, whenever you work with a video image, there are two relevant aspect ratios. First is the actual aspect ratio of the pixels in the image. Let’s call that the “frame aspect ratio.” In contrast, the “display aspect ratio” is how the video image is to be displayed on the intended playback medium. More on this in a brief moment.
OK, back to MPEG, and D1, it’s 720×486 starting point. Remember that in the early 1990s, when MPEG-1 was formulated, the fastest CD-ROM drives read at about 150 KB/second, so MPEG-1 had to meet these rates. Even today, encoding full screen video to 150 KB/second produces poor quality results, at least for the television audience towards which MPEG-1 was targeted.
For this reason, the MPEG-1 committee decided to use what’s called SIF, for (Standard Interchange format), which split the horizontal and vertical resolutions in half, and then rounded off to the nearest multiple of 16, which best matched MPEG-1’s encoding scheme. The committee also opted for frame-based operation as opposed to fields, which yielded a 352x240x30 fps second standard.
As we mentioned initially, MPEG-1 was designed for display on television sets. MPEG-1 files have spaces in the file header to define the frame aspect ratio, directing the CD-I and VideoCD devices (that MPEG-1 targeted) to automatically display in the proper 4:3 frame aspect ratio. The early MPEG-1 hardware decoders that shipped in the 1996 and 1997 did the same thing, in essence shrinking the 352×240 image down to 320×240 for display on a computer screen.
However, hardware players never really took off, leaving most computer-based playback to software players, most notably Microsoft’s Media Player, which didn’t shrink the video, producing an image that looked 10% too wide. Instead, Media Player displayed all 352×240 pixels, expanding the video by about 10%, and adding about 10-20 pounds (TV makes you look fat).
Why didn’t Media Player shrink the video? Originally, it was because the underpowered computers of the day had a hard enough time simply decoding the video, much less scaling it down to 320×240. Remember, this was before all computers had graphics cards that could independently scale the video without involving the CPU.
So Media Player merely decoded and displayed the video in its native resolution. The image was distorted by about 10%, but at least it played.
Interestingly, even today, though computers and their high-powered graphics subsystems are more than capable of decoding and scaling MPEG-1 files to the proper resolution, Media Player, now up to version 7.0, still ignores the aspect ratio instructions in the file header and displays the file at its native resolution, as does RealNetwork’s RealPlayer 8.0. In fact, the only player that implements the aspect ratio instructions was Apple’s QuickTime Player 5.0.
To test this, we encoded a file to 352×240 resolution in VideoCD format using Ligos LSX-MPEG encoder. The header of this file is shown in Figure 6, showing the .9132 pixel width-to-height ratio.
Then we played the files in all three players. As you can see from the screen captures in Figure 7, the QuickTime Player scales the 352×240 video file into 320×240 format for display, while the Real Network’s RealPlayer does not. More importantly, neither does Microsoft’s Media Player 7.0 (click here for image), which starts life as the default player for MPEG-1 on every Windows desktop.
This finding is more of an observation than a criticism because the ability to scale MPEG-1 video to the proper aspect ratio quickly became a non-issue. That’s because MPEG-1 encoders began including an option for “square pixel encoding” which produced an MPEG-1 file with a resolution of 320×240. In essence, rather than creating a 352×240 file and telling the decoder to shrink it to 320×240 before display to a computer screen, they shrink the file to 320×240 beforehand and encoded at the smaller resolution. The file on the right in Figure 1 was created using the square pixel option.
Note that this is exactly the same process that developers had been using for years when capturing NTSC video for compressing into AVI or QuickTime MOV format. You never ever saw an AVI or MOV file at 352×240, it was always at 320×240, a 4:3 frame aspect ratio that looks correct on square pixel computer monitors.
MPEG-1 encoder developers were simply adjusting to the reality that most users weren’t creating VideoCD titles with their encoded files. Rather, they were creating files for CD-ROM titles, to playback from within PowerPoint or to post to a website, in all cases for display on a computer monitor with square pixels.
There was a small risk of incompatibility with some MPEG-1 decoding chips designed to handle only Constrained Parameter MPEG-1 files, a much less flexible class of MPEG-1 files created to enable the creation of cheap, single-purpose MPEG-1 hardware decoders. However, the number of such hardware decoders in the general computer population was so small that this risk was insignificant even back in 1998.
Accordingly, a clear development dichotomy emerged. When developing for VideoCD, encode at 352×240, or better yet, use a VideoCD preset that automatically creates a file with the proper parameters. When encoding for computer display, opt for the 320×240, square pixel output.
Of course, most developers were sufficiently knowledgeable to understand the difference, since few people other than professionals created MPEG-1 files back in the late 1990s. In addition, since most MPEG-1 encoding tools were professionally oriented, the square pixel option became nearly ubiquitous.
Today, however, MPEG-1 file encoding is a checklist feature of most consumer-oriented video editors, and though many programs offer both square and 352×240 encoding, only one, MGI’s VideoWave, explains when to use the various options. In addition, in professionally oriented products like Media 100’s Cleaner (formerly Terran Interactive’s Media Cleaner Pro) encoding tool and Ligos’ LSX-MPEG Encoder, square pixel output was not available as presets before we started work on this article.
Changing the Presets
Users knowledgeable of the various aspect ratios could get good results, but unwary users could produce video that was technically compliant but stretched in appearance when played back on computers. To help resolve this, and other issues discussed below, we contacted many developers of encoding tools with suggestions on how to change their presets and other user interface elements.
We’re happy to report that most developers agreed with our development dichotomy, and agreed to modify their presets accordingly. A summary of results and copies of all correspondence sent to the various vendors will be appended to this story during the week of Oct 15th, 2001, so you can how your encoding tool stacks up.
Making the world safe for MPEG-1 video is obviously a mighty quest, but somewhat esoteric, given that this durable but aging format is quickly being supplanted by streaming media on the low end, and MPEG-2 on the high end. We’ll deal with MPEG-2 in a moment; let’s turn our attention to streaming media, a market where MPEG-like rectangular pixel resolutions insidiously infiltrated, at least in the early days.
When streaming media first arrived, the 28.8 kbps modems these technologies targeted dictated highly compressed files. As we’ve discussed, the QuickTime and Video for Windows markets had standardized on 160×120 or 240×180 for low bit rate video, but streaming codec vendors were more drawn towards 176×144 resolution, which was frequently used in video conferencing.
Interestingly, the 176×144 resolution has a frame aspect ratio of 4:3.27, which has the reverse effect of MPEG-1’s 4:2.72. That is, in order to make the file look correct on an NTSC monitor, the device would have to stretch the video, rather than shrink it. This means that the file shown on the computer will make you look skinny rather than fat; perhaps one reason the format has hung for so long.
When creating files for the 1997 PC Magazine review, we discussed using the square pixel 176×132 resolution, but opted to go with the vendor presets. Since then, however, every comparison that I’ve written has used exclusively square pixel formats, which is clearly the direction that the industry has taken, with the sole exception of Microsoft, who continues to use several non 4:3 resolutions in their Windows Media Encoder.
Not surprisingly, all vendors who have integrated Microsoft’s encoder into their products, which includes Media 100 (Cleaner), Ulead (VideoStudio), MGI Software (VideoWave), and Adobe (Premiere), adopted Microsoft’s presets exactly, including those with non 4:3 resolutions. However, none of these tools use a non 4:3 resolution for any other streaming media format. For example, Cleaner includes extensive presets for Apple’s QuickTime and Real Network’s RealVideo, all at 4:3 resolutions.
Apple and Real sidestep the issue by not having resolution presets in QuickTime Pro or RealProducer Plus, though Real does advise users in the RealProducer Plus help file that “the standard video frame size for the Internet is 176 x 144.”
As with the MPEG-1 resolution issues, we contacted vendors regarding the use of these non 4:3 resolutions. Before contacting Microsoft, we checked the resolutions used by news, sports and other sites distributing video originally shot in NTSC format. We focused primarily on the low bit rate streams, since Microsoft’s higher bit rate presets were all at 4:3 resolutions. Our search was not exhaustive, and doesn’t claim to be. Nonetheless, here’s what we found:
| Table 1. Sampling of pixel resolutions used by major Internet broadcast sites. | |||
 |
Site | Pixel Resolution | Pixel Aspect Ratio |
| ESPN | 176×132 | 4:3 | |
| CNN | 240×180/ 176×132 | 4:3 | |
| ABC News | 192×144 | 4:3 | |
| Yahoo Finance | 288×216 | 4:3 | |
| CBS | 160×120 | 4:3 | |
| Golf Channel | 160×120/320×240 | 4:3 | |
| Tech TV | 176×132 | 4:3 | |
| Oxygen | 176×132 | 4:3 | |
| Animal Channel | 176×132/240×180 | 4:3 | |
| Comedy Central | 240×180 | 4:3 | |
Interestingly, the only anomaly we found was Bloomberg TV, which encoded in 264×224 format, with a frame aspect ratio of 4:3.39, even more pronounced than the 4:3.27 that 176×144 resolution produces. In theory, assuming that Bloomberg was scaling to 264×224, (as opposed to cropping their video) this should make the anchor personalities look skinnier in their streaming format displayed on a computer screen than they appear on TV.
To test this, we shot a digital still image of Bloomberg TV. Once again, the television image had a 4:3 display aspect ratio that we compared to the RealVideo screen capture, with the results shown in Figure 9. As you can see, the anchor looks much skinnier on the right, just as we expected. That’s because the incorrect frame aspect ratio distorted the appearance of the video, which is, of course, in a nutshell, is why 4:3 frame aspect ratios must be used for streaming video
Armed with this data, we sent Microsoft the following note on June 13:
I’m writing an article for www.extremetech.com on proper resolutions for digital video. Specifically, as it relates to streaming media (as opposed to MPEG-1/2 video that might be played back on an NTSC monitor), we recommend that all producers distribute their NTSC-based video with a aspect ratio of 4:3. Moreover, our conclusion is that for NTSC-based video, any frame aspect ratio other than 4:3 will distort the video. Accordingly, we are recommending to our readers that they encode NTSC-based video using a 4:3 aspect ratio.
Upon reviewing the presets in version 7.1 the Microsoft Windows Media Encoding utility, we find the following resolutions/ aspect ratios. Screen copies of the presets are presented below:
Profile Resolution Aspect Video for Web servers (28.8 kbps) 160×120 4:3 Video for Web servers (56 kbps) 176×144 4:3.27 Video for single-channel ISDN (64 kbps) 240×176 4:2.93 Video for LAN, cable modem, or xDSL (100 to 768 Kbps) 320×240 4:3 Video for broadband NTSC (2 Mbps total) 640×480 4:3 It looks like we agree for 3 of 5 recommended resolutions, and disagree about the other two. Is there any reason that your presets resolution s other than 4:3? If so, can you please explain? If not, do you plan to change the encoder presets, and if so, when?
After several e-mail messages back and forth, we got the following reply from Microsoft on June 20 —
I wanted to be explicit that we plan to change the presets as you are suggesting – our customers definitely see a lot of value in those 4:3 presets, and we’re planning to increase the number that we provide. Until then, we recommend users set up custom encoding profiles or crop 4:3 content. That way, they still get the best possible quality at the lowest possible bitrate by using our v8 Windows Media codecs.
Microsoft declined to commit to a date that these changes would be made, but the intention is clear. They are currently updating the Windows Media Encoder for Windows Media Video 8, which now encodes only from a DOS command line interface; hopefully we’ll see the changes in this release.
Working with DVDs and 720p
Now that we’ve addressed MPEG-1 and streaming, let’s move on to MPEG-2. On its face, this format seems to present the same dual development paradigm as MPEG-1, since MPEG-2 files are viewed both on television screens (DVD/Satellite/Digital Cable) and computer monitors.
The obvious simple answer is to encode at 720×480 for DVD development, and 640×480 resolution for other desktop uses. However, as we’ll see, the simple answer may not be the best in this instance.
So what about MPEG-2 files created for general computer playback? Well here’s where things get a bit wiggy. The short answer is use 640×480, but test with as many MPEG-2 players in your target population as possible. Of course, short answers are rarely optimal, so the rest of the story follows…
The MPEG-2 market differs from both MPEG-1 and streaming in one key respect – the lack of player ubiquity. In other words, with both MPEG-1 and streaming, you can assume that your target customers either have, or can obtain, a free player to play the video. With MPEG-2 you can’t. Here’s why—
The MPEG-1 market is highly stabilized into three main players because there is no per unit royalty on MPEG-1 decoders. This enables Microsoft and Apple to distribute free players within their respective operating systems and downloadable players. Ditto for RealNetworks’ RealPlayer.
The bad news is that it’s very difficult to ship MPEG-2 files into the general computer population, on CD-ROM, over the Internet or over any medium, simply because you can’t be sure that your target audience has an MPEG-2 player. The good news is that in most enterprises, you’ll know whether the targets have a player or not, and if so, which one. This allows you to test encoder/player compatibility to ensure you get the results that you want. And our anecdotal testing reveals that you should definitely perform these tests.
As with our MPEG-1 testing, our goals were two-fold. First, consistent with our 4:3 vision, was to test whether 640×480 MPEG-2 files would “break” any software decoder. We also wanted to learn whether any software MPEG-2 player would properly squeeze the 720×480 stream into the proper 4:3 frame aspect ratio.
We tested with three different MPEG-2 software players from ATI, Ligos, and Ravisent Cinemaster. These tests were not designed to be exhaustive, we just ran what we had in the lab or could quickly download to see if any patterns emerged.
We tested MPEG-2 files from three different sources, Ligos’ LSX-MPEG Encoder, MGI Software’s VideoWave, and Ulead’s VideoStudio. Once again, we used what we had around the lab.
Regarding the 640×480 video, the verdict was unanimous; all software programs played the 4:3 ratio files without problem. This means that you can safely encode your files at 640×480 resolution and they will display correctly during playback.
Regarding the 720×480 files, we assumed that the players would shrink the horizontal resolution to approximately 640, consistent with how Apple’s QuickTime Player shrank MPEG-1’s horizontal resolution from 352 to 320 to achieve the 4:3 designation in the file header. However, our results varied by encoder, as you can see in Table 2.
For files encoded by Ligos and Ulead, all three players stretched the encoded file’s vertical resolution from 480 to 540 to achieve the requested 4:3 ratio in the file header. This created a very large presentation, but the frame aspect ratio was correct. However, files encoded by MGI’s VideoWave 4 played at 720×480 resolution on all three players, which looked stretched.
| Table 2. Playback resolutions exhibited by three MPEG-2 players. | |||
| Encoder | ATI Player | Ligos Player | Ravisent Cinemaster |
| Ligos LSX MPEG | 720×540 | 720×540 | 720×540 |
| MGI VideoWave | 720×480 | 720×480 | 720×480 |
| Ulead VideoStudio | 720×540 | 720×540 | 720×540 |
The differences relate to the aspect ratio field in the file header, which both Ligos and Ulead inserted the 4:3 aspect ratio, and MGI didn’t. We’re working with MGI to resolve this issue but it appears that unlike two of three MPEG-1 players, who ignored the aspect ratio flag, many MPEG-2 players are giving effect to these instructions, avoiding the potential for distortion we saw with MPEG-1.
This means that if you encode at 720×480 resolution, most players will blow it up to the next highest 4:3 frame aspect ratio, or 720×540, avoiding distortion. On the other hand, this also means that minor header errors in the encoded files can cause some surprising results; so test your encoded files on your target population before general release. If you use the DVD preset that most encoders provide, these files have the added benefit for being compatible with DVDs, should you ever decide to create them.
Of course, you can also encode at 640×480 and avoid all the playback resolution issues, but your files won’t be DVD compatible. Remember, however, that the first issue of MPEG-2 is the lack of software player ubiquity. Before you plan to use this format, be sure that your target audience has MPEG-2 decoders.
Conclusion
We’re working with many encoding tool vendors to create presets that output the proper frame resolution for the various target devices as outlined herein. Until then, when encoding, be sure to check the output resolution for all presets to ensure you’re your video displays correctly.
