Movie Formats
Contents
The Basics
A typical movie file does not only contain video, but contains multiple different streams:
- 1 Video stream
- 1 or more Audio streams (e.g. English, German and director commentary)
- 1 or more Subtitles (e.g. English and Dutch)
A movie file may also contain metadata (title, episode, director, date of recording) and chapters. Not to mention advanced features such as stereo video for stereo television (often improperly called 3D television).
The file that combines (muxes) the different streams and the metadata together is called the movie container.
Your Choices
To create a movie files, you must decide on:
- The container format
- The video codec
- The audio codec
- The subtitle format
Unfortunately, there is no single solution that works everywhere. Your solution may depend on what is available on your platform (Mac, Windows and Linux support different containers and codes out-of-the-box), the features you require (WebM does not yet support subtitles or multiple audio tracks), if you are willing to pay royalty fees (formats such a H.264 are encumbered by patents), if you need hardware support (H.264 on the iPhone has hardware support and uses much less battery power than any other video format), etc.
Further Information
For more information on the background, you should read the following sites:
- Video on the Web, by Mark Pilgrim. Specifically aimed at using the HTML5
<video>tag. - A gentle introduction to video encoding, also by Mark Pilgrim. Contains a nice list of recent formats, although it is missing the WebM format that surfaced in 2010.
- Comparison of container formats on good old wikipedia. Not always up-to-date, but a great starting point anyway.
Overview of Common Formats
The following tables are taken from Mark Pilgrims' A gentle introduction to video encoding, with WebM added. (Original sources: DigitalContentProducer.com, doom9.org, HydrogenAudio.org, Wikipedia)
Video Codecs
| Codec Name | Inventor | Popular usage | Patent-free? | OSS tools | Non-OSS tools |
|---|---|---|---|---|---|
| MPEG-1 | MPEG WG | VCD | Expired ~2012 | ffmpeg, MP1E | QuickTime |
| MPEG-2 | MPEG WG | DVD | No | ffmpeg, mpeg2enc | Compressor, Sonic, TMPGEnc |
| MPEG-4 ASP | MPEG WG | Movie pirates | No | ffmpeg, Xvid | DivX, 3ivX |
| H.263 | MPEG WG | YouTube | No | ffmpeg | Adobe Premiere |
| H.264 | MPEG WG | YouTube HD, iTunes, Blu-Ray | No | x264 | Adobe Media Encoder, Apple Compressor, On2 Flix Pro, Sorenson Squeeze, Telestream |
| WMV | Microsoft | Microsoft world domination | No | none | Windows Media Encoder, Expression Encoder |
| VC-1 | Microsoft | Blu-Ray | No | none | Windows Media Encoder, Compressor, Squeeze |
| Theora | On2, Xiph.org | Free software hippies | Yes* | oggenc, Thusnelda | none |
| Dirac | BBC | HDTV broadcast | Yes* | dirac-research, Schroedinger | none |
| VP8 | On2, Google | YouTube, Free software hippies | Yes* | libvpx, ffmpeg, libav | none |
| H.265 (HVEC) | VCEG/MPEG WG | Many vendors (in progress) | No | OpenHEVC, GPAC | HM, VSS |
| VP9 | YouTube, Web browsers, Free software hippies | Yes* | libvpx, ffmpeg (in progress) | none |
- (*) As far as we know. In 2013, Google admitted VP8 infringes on h.264 patents, and has subsequently entered a license agreement with MPEG LA.
Audio Codecs
| Codec Name | Inventor | Popular usage | Patent-free? | OSS tools | Non-OSS tools |
|---|---|---|---|---|---|
| AMR | 3GPP | Ringtones, VoIP | Expired? | ffmpeg, RetroCode | 3GPP reference encoder |
| MP2 | MPEG WG | VCD, DVD, Digital TV | Expired | TwoLAME | Audio Transcoder, Easy CD-DA Extractor |
| MP3 | MPEG WG | Music | Expired | LAME | iTunes |
| AAC | MPEG WG | iPod, iTunes Store videos | No | FAAC | iTunes, Nero |
| WMA | Microsoft | Microsoft world domination | No | ffmepg | Windows Media Encoder |
| Vorbis | Xiph.org | YouTube (in progress), Free software hippies | Yes* | ffmpeg, aoTuV | none |
| AC3 | Dolby | DVD | No | ffmpeg, Aften | Dolby |
| DTS | DTS Inc. | Blu-Ray | No | none | DTS Inc. |
| Opus | IETF | VoIP, WebM, WebRTC | Yes* | libopus, VLC, ffmeg | Teamspeak |
- (*) As far as we know
Subtitles (Captions)
| Caption format | Inventor/Specs | Encoding | Popular usage |
|---|---|---|---|
| Line 21 | Electronic Industries Alliance | pseudo-ASCII | Analog TV (US + Canada) |
| Teletext | BBC | custom | Analog TV (Europe) |
| SubRip | A doom9.org forum post from 2004 | windows-1252 | AVI files, "fansubbing" |
| Advanced SubStation Alpha | Some Word document floating around the net | UTF-8 | AVI files, "fansubbing" |
| SAMI | MSDN page | (multiple) | Windows Media Player |
| SMIL | W3C Recommendation | (multiple) | QuickTime (SMIL 1.0 only) |
| DVD | DVD-Forum | Stores as images | DVD |
| MPEG-4 Timed Text | ISO 14966-17 | UTF-8 | MP4 files, iTunes Store |
Container Formats
| Container Name | Extension | Inventor/Owner | OSS tools |
|---|---|---|---|
| Audio/Video Interchange | .AVI | Microsoft | ffmpeg, AVIMux-GUI |
| Advanced Systems Format | .ASF, .WVM | Microsoft | ffmpeg (in progress) |
| Flash Video | .FLV | Adobe | ffmpeg |
| Matroska | .MKV | Some guys in Russia | ffmpeg, mkvtoolnix |
| Ogg | .OGG | Xiph.org | ffmpeg, oggmux |
| MPEG-4 | .MPV, .M4V | MPEG-LA | ffmpeg, MP4Box, mp4creator |
| QuickTime | .MOV | Apple | ffmpeg |
| WebM * | .WEBM | Google, Adobe, Mozilla, Opera | ffmpeg, mkvtoolnix |
- (*) Subset of the Matroska container format
Common combinations
| Platform/Audience | Container | Video | Audio | Subtitles |
|---|---|---|---|---|
| DVD | VOB | MPEG-2 | AC-3 | Line 21, DVD subtitles |
| DVD pirates | AVI | MPEG-4 ASP | MP3 | SRT, ASS |
| Blu-Ray | EVOB | H.264, VC-1 | AC-3, DTS | Blu-Ray Subtitles (SUP) |
| Blu-Ray pirates | MKV | H.264 | AC-3, DTS | SRT, ASS |
| Flash 8 | FLV | H.263 | MP3 | Custom, if any |
| Flash 9+ | MP4 | H.264 | AAC | Flash Timed Text |
| Windows | ASF | WMV, VC-1 | WMA | SAMI |
| Linux | Ogg | Theora | Vorbis | Kate |
| Mac/QuickTime | MOV | H.264 | AAC | QTTextTrack |
| iPod/iPhone | MP4 | H.264 | AAC | MPEG-4 Timed Text |
| AppleTV | MP4 | H.264 | AAC+AC3 | MPEG-4 Timed Text |
| HTML5 (Firefox, Opera, Chrome, Android) | WebM | VP8 | Vorbis | none (will follow future WHATWG / W3C guidelines) |
| HTML5 (Firefox, Opera, Chrome, Android) | WebM | VP9 | Opus | none (will follow future WHATWG / W3C guidelines) |
| HTML5 (Safari, IE, iPhone) | MP4 | H.264 | AAC | none (will follow future WHATWG / W3C guidelines) |
The Debate
One of the goals of the HTML5 working group was to define a video format that was supported in all webbrowsers. In the end, this proved unsuccessful, and Ian Hickson had to admit defeat. Basically Apple and Microsoft only support H.264 natively (and not WebM), while Mozilla and later Google only support WebM natively (and not H.264).
Recommended Formats
Stand-alone Movies
For movies and series use either:
- The DVD VIDEO_TS format (which retains the original menu structure), but is very large in size.
- Matroska (MKV) files with VP8 (or optionally H.264) for video codec, Vorbis for audio codec, and VobSubs for subtitles.
See for example Rip DVDs in MPEG-4 AVC (x264), multi audio, subtitles, Matroska.
Video on the Web
For videos on the web use both:
- MP4 container with H.264 video codec and AAC codec, for use on Safari, IE 9, iPhone and older Androids.
- WebM (A Mitroska container with VP8 video codec and Vorbis audio codec), for use on Firefox (4+), Chrome, Opera and Android.
If you care about support for older browers, you may add:
- Ogg container with Theora video codec and Vorbis audio codec, for Firefox 3.5
- Flash video (Flash Video (FLV) container, with FLV1 video codec and MP3 audio codec), for older webbrowsers, including IE 7 and IE 8.
- Download button so people can download it for later viewing.
See for example Video on the Web and Video for Everybody for the details.
Personal Video
Most photo and video cameras have output in .AVI or .MOV files. AVI files at least do not support metadata (such as time of shooting or location).
My recommendation is to convert both formats to something that does support metadata, although I have not found something my media browsers has good support for storing metadata in the movie file (I'm using iView MediaPro, which has subsequently bought by Microsoft and later Phase One and rebranded as Expression Media).
