Video Formats Explained

Video Formats Explained

It used to be so easy to start out movie making - you'd buy an 8mm or Super8 camera; or if you were really industrious, a 16mm camera and shoot and edit your movies.

In the 1980s, shooting and editing movies got only slightly more complicated with the introduction of VHS, VHS-C and Betamax. You could read a two page article and know the differences between competing video formats. When video went digital though, everything went out the window. Suddenly there was a bewildering array of video formats — .wmv, .asf, .rm, .mov, .mpeg — for compressing files, and on top of that, many of these standards had their own substandards (MPEG-1, MPEG-2, etc.) How's anyone supposed to keep it straight?

Containers and Codecs

Possibly one of the most confusing things about digital video formats is the idea that there's a "container" and "codec."  You might think it's enough to make you yearn for the days when you could just put a tape in the camera and start recording, but you'd be wrong. The plethora of video formats mean that whatever type of video production you're doing, there's a good way to make it happen. Twenty years ago everybody was watching movies the same way — either on a screen via a projector, or on a television set. Today many, many more options exist and people are taking advantage of them all. From high-end HD video with surround sound shown in a top of the line home theater to video streaming from a cellphone, video is everywhere and having a grasp of the various formats will ensure that your video gets where it needs to in the best way.

An Analogy

Trying to figure out exactly what containers and codecs are can be a little bewildering because it's a very technical subject. You can think of containers as a type of publication — a hardback book, a glossy magazine, a newspaper, a pamphlet, a gum wrapper — all of which can contain words and other things like photographs, or cartoons, and yet each one works in a different way.

Think of the codec as the way of displaying the information you want in your publication. You can, for example, print War and Peace on Bazooka Joe bubble gum wrappers, but it will take thousands and thousands of them and nobody will want to read it that way; in the same way you can put your vacation footage on an uncompressed DV format, but you won't be able to stream it over the web. Similarly, you likely want your copy of War and Peace to be beautifully bound in hardback, but that's not the proper container for a takeout menu someone's going to shove in your mail slot. Words with drawings could properly be placed in a comic book, or a hardback book, or a newspaper, but the images in an art catalog or high-end fashion magazine require heavy-weight glossy paper to reproduce properly.

Every video application has a proper codec and container. To add one more layer of complexity: both codecs and containers go in and out of style — so a format which was popular a few years ago may be something few people are using today.

One thing that further complicates this is that some containers and some codecs are proprietary which means that there are occasional licensing issues about using one type of container with another type of codec.

In the past few years, the dust has somewhat settled on the video format wars with formats like DivX and mpeg-1 falling by the wayside and h.264 making a strong stand. However confusing it is now, it used to be worse.

Lossy vs Lossless Compression

Image of woman with close up inset showing artifacts.
Usually, when video data is compressed, some of it is lost. Video compression software works by looking for redundancies in a frame, such as one bit of blue sky next to another bit of blue sky, and representing the redundancies together. At very high rates of compression this becomes obvious, but at lower rates it's difficult for people to notice. There arises a desire in every movie maker to try to use lossless formats, which preserve all the original data, but the compression ratios aren't good enough (yet) to make them practical. Choices at the moment are to use the highest quality compression available for your intended purpose and to have multiple versions of your files for multiple uses — i.e. one file for web-streaming, another for disk-based distribution, another for standard definition DVD, another for Blu-ray, etc.

Edit and Distribute in the Highest Quality

When compressing for export, keep your video as high a quality as possible. Export video at a smaller size in order to reach a certain platform, e.g. iPhone.
The highest quality video format is going to be the format you captured your video in. While digital files do not degrade in quality during copying, every time they are compressed with a lossy compression they lose data, so converting your uncompressed HDV formatted files even into a high quality MP4 will result in a loss of quality. While it's necessary to compress files in order to be able to share them, you should avoid re-compressing any more than you have to. Keep your master files in the original format; edit and create versions at whatever sizes necessary — but always from the originals. Don't burn a Blu-Ray from a file that's already been compressed into a low resolution file for streaming over the internet, instead create a separate, high resolution, file from your editing software.

A lot of the work in deciding how to compress video files can be done by your video editing software. Most consumer editing software today will have presets for various methods of distribution the software's video compressor will use — such as DVD, tablet or phone video — your DVD/Blu-Ray burning software already knows which format to use in order to make a DVD that will play on home theaters.

Containers

Interface showing drop down menus for compression type.
We'll take a look at some containers and then at some codecs. A video's file extension usually refers to the container. A few containers have codecs that they almost always use and other containers are often used with many different codecs.

Audio Video Interleave (.avi)

Developed by Microsoft and released with Windows 3.1 way back when false teeth were still made out of wood, AVI files were once a workhorse of digital video. If I say “AVI is dead” the comments section will clog with people still using it, so I'll say that it's popularity has waned, but there is still lots of legacy AVI to be found all over the web. Short answer, don't output video to it, but keep a player handy.

Advanced Systems Format (.asf)

ASF is a proprietary Microsoft container that usually houses files compressed with Microsoft's WMV codec — to make things confusing, the files are usually designated .wmv and not .asf. The ASF container has the advantage over many other formats that it is able to include Digital Rights Management (DRM), a form of copy protection. It was designed for streaming video from media servers or over the Internet. Short answer, again, don't output video to it, but keep a player handy.

QuickTime (.mov or .qt)

QuickTime was developed by Apple and supports a wide variety of codecs. It's a proprietary format though and Apple decides what it supports. Quicktime, like Microsoft's version, .avi, looked like it was going to fade into the sunset but just as it was about to die, Apple released the Mavrick update and quietly replaced anything inside a .mov container with h.264. In fact, both Nikon and Canon DSLR's output h.264 video wrapped in a .mov container. Short answer: Sure, why not. Most people will be able to read .mov files for a while now.

Advanced Video Coding, High Definition (AVCHD)

AVCHD is a very popular container for data compressed with h.264 — it comes to us through a collaboration between Sony and Panasonic as a format for digital camcorders. It's a file based format, meaning that it's meant to be stored and played back on disks or other storage devices such as compact flash drives or SD cards. It supports both standard definition and a variety of high definition variants from 720 to 1080p in a variety of frame rates including 60 frames per second, 24p native as well as 3D modes.

It's an extraordinarily robust container format that includes not just things like subtitles, but menu navigation and slideshows with audio. Short answer: Yes.

Flash Video (.flv, .swf)

Flash was originally created by a company called Macromedia which was acquired by Adobe in 2005. Flash has been around for a while and comes in multiple versions, some better than others. Older Flash video often uses the Sorenson codec, newer Flash uses h.264. At one time, websites that appeared as a blank page with a message “You must download and install the latest version of the Adobe Flash Player to view this content” were ubiquitous. It looked really cool, but it didn't work with everything. Former Apple CEO Steve Jobs took Flash out behind the barn like a rabid dog when he called it "buggy” and blamed it for problems with the Apple Operating System, refusing to support it on mobile devices in favor of HTML5, we're just waiting for the shot. Short answer: not if you have other options.

.M4V and .MP4

.m4v and .mp4 are very similar and are both part of MPEG-4 which was based on the Quicktime file format. .m4v was created by Apple as an extension of MPEG-4 with the option of proprietary Apple DRM to keep their files from playing on non apple devices. It is used, among other places, when distributing content from iTunes. As a result, many non-Apple devices (such as my Sharp television) will refuse to play .m4v files. However, the formats are so similar that in instances where the DRM isn't being used, simply changing the file extension to .mp4 is enough to convince the device to play the file. Doing this will cause Apple Quicktime based applications such as Apple TV to be unable to find some contents of the file, namely chapter markers and AC3 encoded audio streams.

Codecs that are Also Containers

As if things aren't confusing enough, some containers have codecs with the same name.

MPEG-1

MPEG-1 is used almost exclusively for Video Compact Disks (VCD), which are extraordinarily popular in some parts of the world but never caught on in the U.S. — the video quality is substantially lower than DVDs. Short answer: No.

MPEG-2 (h.262)

MPEG-2 is a container format, but there is also a codec of the same name, which most people call h.262, so that it's not so confusing, though a world where we call something h.262 is already more confusing than it ought to be. MPEG-2 is used for DVDs and pretty much nothing else with the exception of broadcast High Definition Television (HDTV). Short answer: No.

Codecs

Windows Media Video (.wmv)

Once it was realized that the Internet was a delivery vehicle for things like video, people started trying to come up with ways to share video that wouldn't take up a lot of bandwidth and disk space. One of the big advances was the idea of streaming video — where your computer downloads only a part of a video and begins to play while the download continues — this means you don't have to wait two hours for a movie to download before you can start watching. Over the years the WMV format has grown to include support for high definition 720 and 1080 video. To make things complicated, files that end in .wmv are usually stored in an .asf container.

h.264

Not only do you need to call the MPEG-2 compression codec h.262, you have to keep from confusing it with h.264, which is used to compress Blu-ray disks as well as lots of web video. One of the very nice things about h.264 is that you can use it at very low and very high bitrates. The h.264 will send highly compressed low resolution video across the web and then happily encode your high definition movie at super high bitrates for delivery to a High Definition television. This is a very common codec for camcorders and digital video cameras. Its container is AVCHD.

What's the best video format?

While there isn't one "best video format," there are best video formats for particular jobs. Things to be aware of when choosing a video format, apart from the intended audience — Will they be watching video streaming over the Internet? Do they have a fast connection? Do they have a DVD player? Do they have a Blu-ray player? — are the longevity of the format and how widespread its adoption.

For a number of years now, a good bet for a forward-looking, high-quality, versatile video format is h.264, aka, MPEG-4/AVC (Advanced Video Coding), though it really needs a better name. h.264 is supported by a number of important players including Microsoft, Apple and Adobe, though in early 2011 Google dropped support for h.264 from its Chrome browser citing the desire to use only open-source (i.e. non-patented, royalty-free) standards, Microsoft swiftly made a Chrome extension which restored support. Google's answer was .vp9 (soon to be vp10) to be used as part of HTML5. Performance of the two codecs is very similar. Adoptions and usage will determine if there's an eventual winner.

Switching Between Formats

It's relatively easy to convert between most non-DRM (copy protected) video formats with any number of free or low-cost conversion utilities found on the web. If you do transfer between formats, remember that re-compression causes degradation, if quality is of paramount importance, don't delete the originals, archive them somewhere. Also keep in mind that when moving between containers some data streams such as subtitles and chapter data can be lost if the new container doesn't support them.

Conclusion

As viewing experiences and platforms evolve, the way in which moving pictures are delivered will continue to evolve and change. New containers and codecs will continue to be produced that will allow for larger amounts of data to be delivered more quickly and with additional data streams. There will never be a final format  but keeping in mind your audience and the type of video you’re delivering, there will always be some options that are superior to others. Good organizational skills and regular migrating to new formats will make sure that your video survives as technology changes.

Contributing editor Kyle Cassidy is a visual artist who writes extensively about technology.

Issue: 

Kyle
Cassidy
Thu, 03/01/2012 - 12:00am

Comments

setting up live stream

wendikaren's picture

This is a great article ,very informative, thank you. We are setting up live stream for our church. We actually switched from windows to mac. I would like to know which camcorder or professional camera under 1000 dollars that can do the job. I'm using wire cast as our encoding software and we're using a Mac with 10.8

I have more info. I can give you, but didnt want to inundate in a post. Thank you!

.avi

Hi, firstly...great article. Very simply put. 

 

I have an old .avi and only need to play it on a large TV in reception. Its only 800MB so I thought I would avoid compressing it altogether in order to keep the 'quality' (some of the video is over 10 years old). 

 

However, it does not play on any media device I have got but will play on my PC. The biggest compressed version i have got is a .vob file and it works fine. 

 

Any ideas why it would not work? As you say above it only needs to be compressed to share and the media device holds much more than 800MB so whats the issue?

 

Clearly my PC doesnt think there is an issue. 

Device support...?

It is also worthy of noting that not all stand-alone devices (DVD players, television sets etc.) support all codecs/containers, and not equally.

 

What is important to remember - and this may answer Karl Russell's question - is that, on a computer (be it a Windows/*nix PC, or a Mac) appropriate codecs can be installed as required, and processing speed is a non-issue! As in - even if the video playback is choppy beyond reason, this is still the user's problem. It's up to them to lower the quality (or find a lower-quality file), or upgrade their machine.

 

But! Not so with stand-alone players! These devices usually can't be upgraded, and certainly their hardware can NOT be refreshed, except by buying an entirely new unit. They don't have the kind of structure a PC does. Instead, all decoding is done in specialized chips that do nothing else. And as it is, the more calculation-intense an operation is, the more advanced a chip it requires - and the more pricey it becomes. In the interest of keeping design simple and prices competetive, corners are cut. (find me a stand-alone player, whatever it is, under $350, that can play High or even Main h.264 video with all its bells and whistles, I dare You) Enough to say that way back in 2006 You would be hard-pressed to find a device that supports even a hint of h.264 - it was too intensive to process back then, and everything was DVD really (h.262).

 

As such, support for many options within the codecs is usually dropped. Many older codecs are also dropped (to simplify things). It's a closed environment that nobody except the maker can modify. And even though the companies try hard to deliver robust devices, there will always be gaps - by necessity.

 

And even today, even with the widespread adoption of h.264 as a standard -- h.264 still has many options that have not been explored by device manufacturers. Why? Simple - they provide little gain in image quality, and require a lot of processing power in exchange. Of course, these little tricks that h.264 can do are the little details that set the mark between "great" and "awesome" video quality.

 

 

But what I'm trying to get at, the most, is the fact that especially when delivering to stand-alone playback devices, it is of utmost importance to first and foremost keep the final device in mind when compressing, and compress for it. Finding out the exact specs can be an issue (esp. with the cheaper models) -- manufacturers all tend to boast support for many codecs, but when they're up against the wall about exactly which options and variations they support, the fall silent for some reason. A file encoded in h.264 that plays on one device may or MAY NOT play on another. And that's where the problem is at.

A bit outdated

bgallagh's picture

Some good info here, but definitely needs to be updated.  Claiming that Adobe will support Flash for some time, no mention of h.265, and only an external reference to html5 standards are all glaring errors from the last year of technical developments. Since this article is promoted on the Videomake site, it would serve us readers well to post some updates, please.

Bad info...

bscenefilms's picture

H.264 
Not only do you need to call the MPEG-2 compression codec H.262, you have to keep from confusing it with H.264, which is used to compress Blu-ray disks as well as lots of web video. One of the very nice things about H.264 is that you can use it at very low and very high bitrates. The H.264 will send highly compressed low resolution video across the web and then happily encode your high definition movie at super high bitrates for delivery to a High Definition television. This is a very common codec for camcorders and digital video cameras. Its container is [b]AVCHD[/b].

 

It can be, but it is not limited to that...

huffyuv codec

pnoyes's picture

I recently had a vhs cleaned up.  I received the file back in huffyuv codec (22GB).   I installed the the codec in Windows/Syswow64 as directed.  I was able to play it back on XMedia recode and was very impressed with the color and clarity.  However, I was unable to use it on Premiere or Pinnacle 18 editing systems.  I finally recoded it in XMedia Recode to FFV1, altho it came out a little smaller, 11GB.  Then I was able to edit it in  Pinnacle.  Is there anyway to get Premiere of Pinnacle to "see" huffyuv? 

Seriously?!

bugmenot's picture

It was bad practice back then as well, to link to w 3 s c h o o l. Check w3fools.com for more info.

avi master?

arnold68's picture

I want to convert my father's 8mm home movies to a digital format with an editable master file. I'm a total editing novice, so I think a PC-editable format might be good enough. (But I'm thinking of transferring using a Pro 2K process.) An online company (videoconversionexperts.com) outputs to HD-AVI, and they don't offer AVDHD. They do offer MP-4 but call it streaming quality rather than editing quality. Your column says AVI is a dinosaur, and you wrote that four years ago. Am I missing something here?

Re:avi master?

VIDEOMAKER PLEASE BRING BACK NEWLINES FFS >.< !!!!
|
|
|
AVI is indeed a dinosaur. For one, it doesn't support timestamps on frames, which makes it pretty bad for NLE.
|
arnold68: the company is bullshitting You. There is no such thing as "HDAVI" or "HD-AVI".
|
AVI, MP4, MKV and OGM -- these are all containers, which means they can contain individual streams compressed with CODECS. MP4 typically holds H.264 video (this is the video stream encoded in H.264). AVI however does not support independent streams, instead opting to interleave video and audio together (presumably for the fact that it was less demanding and simpler to process back when it was introduced in 1996). As such, they are practically never used for anything nowadays, unless it needs to be simple and cheap. Even in-game cinematics have shifted from AVI to more controllable things like OGM or even RADVideo (which is a story unto itself). AVI spec is 20 years old this year - it's ancient by all accounts.
|
|
More notably, AVI was never meant for HD work, even though theoretically you can save such a stream to AVI (because it's essentially a container). It will probably violate AVI format specs (?), but it's doable.
|
|
Second, it's not AVDHD - it's AVCHD (AVDHD is appartently a DVD player by Audiovox, or somesuch). If You want to edit the file, You will want edit-quality, and that'll be something like MPEG HD422 (in a container like MXF or MP4) or PRORES (in a MOV container), or something similar.
|
|
In general, the point being -- there are two kinds of codecs/formats out there: edit-oriented and playback-oriented (streaming, also).
|
--- Playback-oriented ones will ensure good quality for as little processing power required and/or as little storage space required (ex. h.262 (MPEG-2 DVD), h.264 (MPEG-4 BluRay)). They achieve that by encoding data based on relations, so that the next frame may be "previous frame + changes" instead of an entire new image. A few may be "visually lossless", but some information (usually quite a lot of it) is typically lost regardless. The advantages are: +small size, +easy decoding, +low bitrates.
|
--- Edit-oriented ones will ensure each frame is kept separately (exclusively I-frames) instead of relying on predictions and changesets/relations. The resultant files are positively HUGE, but should ensure that the result coming out of an NLE (Non-Linear Editor, like Premiere Pro or Final Cut Pro) will be of equal quality (they tend to be lossless codecs). Of course - handling 25+ frames per second at full-frame sizes is taxing for the storage device, so real-time playback is rarely feasible with these (hence they're useless for playback/streaming).
|
|
A normal editing process typically sees editing on full-frame codecs, followed by compression to streaming formats (as required) for final delivery.
|
From where I'm standing, they're probably offering You a mixed bag of something they think will be editable, but may not be. I would stay away from those folk, or at the very least request they provide their original pulled material, even if it's all TIFF or DNG frames (which will be reasonably HUGE!).
|
|
If You want to edit on PC, Your editing go-to program will probably be Premiere Pro, and Premiere Pro natively works with AVCHD and DNxHD at high bitrates. But do not worry, You can often use an interim tool (I like to use ffmpeg or ffmbc, there are nice GUIs for using them through windows -- look up "AnotherGUI") to convert whatever the company gives You to Your prefered editing format.

RE:huffYUV codec

It seems that working with huffYUV is broken on many systems, for whatever reason. Recoding it to the open standard FFV1 is the right choice here (I would've probably gone with ProRes 4:2:2 HQ, but I'm typically on a Mac).
|
|
On a side note, Premiere Pro seems to be rather picky about its input video formats, so I wouldn't rely on it as the industry benchmark... ;-)