Streaming Learning Center Where Streaming Professionals Learn to Excel
Related Articles
The Harmonic, Inc. Workflow System (WFS; created by Rhozet, which is now a part of Harmonic) is an architecture for retrieving and encoding audio and video files into multiple single file and adaptive formats, checking their quality, and delivering the encoded results to local or remote destinations. The primary encoding engine has been ProMedia Carbon (formerly Carbon Coder), a Swiss army knife program with extensive input and output options, excellent output quality, and many other fine features. Unfortunately, encoding speed was not one of them.
To remedy this, Harmonic recently released ProMedia Xpress, a highly focused product designed to fit neatly into the encoding workflows used by many broadcasters. Specifically, in its first iteration, the product can only input MPEG-2 transport streams using either the MPEG-2 or H.264 codec, and it can only output H.264 encoded MPEG-2 transport streams, though with additional licenses, the system can convert these outputs into chunked video files and metadata for HTTP Live Streaming (HLS), HTTP Dynamic Streaming (HDS), and Smooth Streaming.
While future Xpress versions will support additional file inputs such as AVI, MOV, and MFX input, including both ProRes and Avid DNx HD, output support will continue to be limited. For example, though DASH support is scheduled for the next software release, the development path does not include single or multiple MP4 file output; if you’re an RTMP streamer, either use ProMedia Carbon or find another solution.
You can purchase the software-only version of Xpress for $9,000; a rack-mounted turnkey system called the ProMedia 5200 application server, which is the unit I tested, costs $25,000. The license for each additional adaptive streaming package format is $2,500, with an additional license of $2,500 for encryption for each adaptive format.
The 5200 is a beast of a system, with two computers. One serves as the controller; the other serves as the encoding node. Both are equipped with two 6-core 3.33 GHz Intel Xeon X5680 processors (24 cores with HTT) with 12GB of RAM, and run Windows Server 2008 RT Standard, SP1. The system does not use GPU acceleration, and both units are equipped with a Matrox G200eW graphics card.
The technology that Harmonic developed to speed encoding is called MicroGrid parallel-computing, which, according to the marketing materials, “splits the large H.264 transcoding job into thousands of tiny ones, each of which is completed concurrently, removing the bottlenecks associated with traditional transcoding architectures.” Xpress can operate stand-alone, or it can serve as an encoding engine within the Harmonic Workflow System, operating via a GUI or API. When you drive the system directly, you use the ProMedia Xpress software and its components, which run exactly like the Harmonic WFS.
Contents
Building a Workflow
Figure 1 shows the Xpress Manager. As you can see, tabs atop the interface provide details regarding active jobs, queued jobs, pending QC jobs, and other functions. By way of background, all encodings are driven by a workflow, whether started manually, via watch folder or API.
Figure 1. The ProMedia Xpress Manager.
You build a workflow in the Workflow Editor, which uses encoding presets created in the Preset Editor and Packaging presets built in the Package Preset Editor. Figure 2 shows the Preset Editor, where you configure your adaptive streams and set your H.264 encoding options. Configuration options are very limited, including H.264 configuration options. For example, all encoding is single-pass, with constant bitrate (CBR) and average bitrate control options, but not true variable bitrate (VBR). This enforces the most conservative theories of adaptive streaming, but it may be too inflexible for some producers who seek the additional quality that VBR, or even 2-pass CBR, often delivers. You can get this if you incorporate Carbon Coder into the system but at a significant cost of encoding speed. There will be more on quality later.
Figure 2. The Preset Editor: Note that I’m editing the H.264 adaptive group that I deploy in the next screen.
H.264 configuration options are limited to Profile and B-frame interval, without control over options such as entropy encoding or search functions. In truth, for most users, fine control over H.264 encoding options is unnecessary; however, if you know what you’re doing, you may find this lack of control frustrating.
The Package Editor performs a similar function, allowing users to configure HLS, HDS, and Smooth Streaming options such as chunk size and closed caption options. The Xpress packagers can integrate captions from multiple sources, including CEA-608 and CEA-708, DVB Subtitle, and Teletext, and convert the input for use in all three packaged formats, though some custom development may be necessary for complete integration.
Once you have your presets and packages configured, you build them into a workflow in the Workflow Editor, shown in Figure 3. There are three major classes of components (Pre-Transform Tasks, Transform Tasks, and Job End Tasks), with success and failure components available at each level. You see this in the tree-like structure shown in the middle of Figure 3.
Figure 3. The Workflow Editor
Pre-Transform Tasks include quality control or file configuration checks that require the Harmonic quality control service (not included), or notification tasks. Transform tasks are the primary encoding functions that are driven by presets. In Figure 3, you see that I’m deploying the preset created in Figure 2. Upon a successful encode of the source file to this preset, Xpress deploys the HLS and HDS packages. The packagers input the MPEG-2 transport streams created during the initial encode and create the unique container formats and metadata files necessary for each adaptive format. Since no additional transcoding takes place, these operations are very fast.
All targets and packagers have output paths. At job end, you can trigger another set of tasks that deliver these files to CIF or FTP locations, invoke additional quality control stages, or create reports or notifications.
Once you create the workflow, you can trigger it via a watch folder that can poll universal naming convention (UNC) or common internet file system (CIFS) drives or retrieve via FTP, by direct file input via another tool called the File Queuer, or via the API. Irrespective of the triggering technique, you can set an encoding priority on a scale from one to 10, with 10 being the highest. While you can’t save a File Queuer action, you can easily requeue a file in the Xpress Manager by right-clicking the log entry and choosing Re-Queue. This is a convenient way to re-encode repetitive jobs or tasks that fail.
Speaking of that, though the high-level workflow is fairly easy to understand, expect a fair amount of errors when you first start using the system. To a degree, that’s to be expected, since absolute precision in addressing and other configuration options is obviously required. However, Harmonic could do more to make the system easier to use and to catch errors before you push the Go button and walk away.
For example, though the system works most efficiently when you use UNC addressing (such as IP addressing), there’s a Browse button that you can use to choose a target folder for your output. Of course, if you use the Browse button and don’t type in the UNC address, your encode may fail, making the Browse button the user interface equivalent of Lucy holding the football for Charlie Brown, then pulling it away just before he kicks. Sure, you can use the convenient Browse button, but your encode may not work.
Harmonic’s error messages are also particularly obtuse. For example, address errors are signified by an “error in the external framework.” I’d prefer something like, “Use a UNC address, dummy,” which would lead to a faster resolution of the problem.
In addition, Harmonic should consider a workflow checking mechanism similar to the “test connection” you see on some programs connecting to an FTP site. Significantly, Elemental checks under the hood when you create an encoding job in Elemental Server, so you can’t start a job that will fail; you get an instant error message.
At one level, this is all tempest-in-a-teapot stuff. Most users will set up workflows for repetitive use when they get the system and won’t experience these types of errors ever again. On the other hand, not all compression geeks are network-savvy, and Harmonic should do what it can to simplify the initial setup and workflow additions that may occur after system installation.
With this as prologue, let’s transition to the tests that I performed.
Performance
The first test involved encoding performance in two tests: The first encoded a single 52-minute 1080p file to 11 adaptive streaming presets, and the second encoded 24 1-minute DV files to the same presets. Table 1 contains the results, though some explanation is in order.
Table 1. Performance comparisons
The first two software-only tests involved Episode Engine and ProMedia Carbon, which I performed on a 3.3 GHz, 12-core (24 with HTT) HP Z800. I tested ProMedia Xpress on the ProMedia 5200 application server, which has two 3.33 GHz 12-core (24 with HTT) computers. I tested the two GPU accelerated units on the appliances themselves.
Irrespective of platform, Elemental Server remains the performance leader, though Xpress comes in a relatively close second. More importantly for Harmonic users, it’s significantly faster than ProMedia Carbon, which was the only other option for Harmonic WFS users.
Note that you can run ProMedia Carbon on the 5200 Application server and access it via the same Xpress interface illustrated previously. To compare encoding performance on the same platform, I encoded my 93-second 720p test file into a 6Mpbs 720p H.264 file with Xpress and Carbon. Xpress produced the file in 50 seconds, while Carbon produced it in 186 seconds, which is close to four times as long. Irrespective of the platform, Xpress is much, much faster than Carbon.
During the longest trial, I checked CPU utilization on the computer in the 5200 that was serving as controller, and I saw that it was hovering between 1% and 3%. Though all encoding scenarios are different, if you’re not going to be managing a server farm with the controller, the 5200 may be overkill. You may find it more cost effective to buy the Xpress software and install it on a very fast single computer. Note that other workflow functions, such as quality control, could boost overall system utilization as well, but for the type of episodic encoding that I was doing, the 5200 was clearly underutilized.
Video Quality
Analyzing the quality of the Xpress output raised some heretofore unseen issues. First, I had to convert my standard test files into an MPEG-2 transport stream that Xpress could input, which added another generation of compression artifacts, though at the high data rate used for this conversion, I’m sure that the affect was minimal.
In addition, in the past, I’ve compared MP4 output from the various encoders that I’ve been testing and its relevant contemporaries. However, Xpress doesn’t output MP4 files, and its MPEG-2 transport stream uses a different packing scheme that’s much less efficient than MP4. For this reason, comparing an MP4 file with a .ts file of similar size isn’t an apples-to-apples comparison.
To help work around this issue, Harmonic provided an H.264 elementary stream pulled from an MPEG-2 transport stream file produced by Xpress. I compared this file to output produced by Teletream, Inc.’s Vantage and Elemental Server from my standard test files (e.g., not converted to MPEG-2 transport stream) and found the Xpress file about the same in quality. Of course, this wasn’t a file I could produce from Xpress myself, so I hunted around for a test that I could run using unaltered Xpress output.
I decided to compare Xpress’ HLS output to the HLS output of Sorenson Squeeze 9. Specifically, I created HLS output in two configurations, 720p at 800Kbps and 640×360 at 240Kbps, making sure that the total size of the output files were within 5% of each other. Ultimately, I produced the Xpress files at the stated target data rates and the Squeeze files at 780Kbps and 200Kbps, respectively.
Why Squeeze? Because using the x264 codec, Squeeze’s output quality equals or is slightly superior to encoding tools such as Elemental Server and Vantage, though they are both an order of magnitude faster than Squeeze, particularly Elemental. Also, to be fair, I wanted to start with the test file that I had converted into the MPEG-2 transport stream so that Xpress could encode it, and Squeeze was the only encoder that I had access to.
In one sense, an HLS-to-HLS test is more relevant than comparing MP4 files, because HLS is the actual final output. On the other hand, when comparing HLS output of equal total size, you’re analyzing both the quality of the H.264 in the stream and the efficiency of the transport stream packing scheme, which varies from encoder to encoder.
For example, another approach would have been to encode both files to the same target data rate and compare the output, even if the output file size varied significantly. Unfortunately, that approach assumes that both encoders meet the target data rate precisely, which, in my experience, they seldom do. So if the total file size of one group of files exceeded the other by 25%, I couldn’t tell if this was packing scheme inefficiency or just missing the target data rate.
Overall, I’m going to have to rethink how I compare encoding tools going forward, and it raises some interesting questions for producers as well. For example, when you specify a combined audio/video data rate target of 800Kbps for HLS output, does your encoding tool create an audio/video stream of 800 Kbps — and then pack it in an MPEG-2 transport stream wrapper — which obviously adds to the data rate? Or does it create a bitstream that totals 800Kbps, including audio, video, and transport stream overhead?
Certainly, I think you want the latter, but do you know what your encoding tool is providing? Clearly, compression quality plays an important part in the overall quality of the output, but so does transport stream efficiency. Some vendors, such as Zencoder, Inc., promote the efficiency of their HLS output, claiming that they are as much as 13% more efficient than Rhozet and 10% more efficient than Squeeze. None of my analysis even looked at that issue, but it’s an issue that we’ll have to analyze going forward.
So, how did Xpress’ output quality compare to Squeeze? Again, it is very similar. Overall, after the long and winding road of testing, Xpress’ output quality proved very competitive. I should also note that I tested deinterlacing quality and found that competitive as well.
Closed Captions
Xpress supports closed captions, Teletext and digital video broadcasting (DVB) subtitling, as well as SCTE cut in insertion. I tested Xpress by passing through CEA-608 titles contained in an MPEG-2 transport stream through to an encoded MPEG-2 transport stream and HLS output, which all worked well.
The Net/Net
While not the fastest encoder in the land, Xpress was very competitive in both quality and performance; it is a vast improvement over ProMedia Carbon. If you’re a Harmonic workflow system user, it’s definitely worth checking out. The 5200 application server could be a workhorse in the right environment, though it’s clearly overkill when not used in a server farm operation.
At the 2013 NAB Show, Harmonic announced that Xpress will support HEVC and the Ultra HD codec for OTT video-on-demand application. These features were unavailable for our tests.
This article appears in the June/July 2013 issue of Streaming Media magazine as “Review: Harmonic ProMedia Xpress.”
