I'm not sure exactly why (perhaps its because codec selection is mostly done without our knowledge?) but there is often a lot of confusion and mystery surrounding audio and video compression which can lead to many misconceptions. For example did you know that AAC is NOT an Apple codec? It is actually the successor to MP3 developed by Fraunhofer - the same people that created the original MP3 format. Also like MP3 (to use its full title MPEG1 Layer 3) it has also been included in the Motion Picture Expert Group's standards as MPEG2 Part 7 and MPEG4 Part 3.

In this blog I hope to describe the important elements of audio and video compression and summarise the standards which are out there in a way that people can understand.

Codecs and Containers

One thing that is often misunderstood is the difference between a codec and a container, and whether a standard refers to one or both. So before we move onto the what is out there let me explain...

A 'codec' is basically a small piece of code which can compress or decompress audio or video from its raw form (which is rather large) to a smaller more manageable size so we can then put on our MP3 player, transmit, burn to a DVD or whatever. Codec compression specifications can either be proprietary (which means the way the file is compressed is owned by a company which can restrict or license its usage) or they can be open (which means anyone can develop a codec which compresses the data according to the specification).

A 'container' is a means of multiplexing the audio and video together so that it can be played back simultaneously. Some containers can also include additional data such as subtitles, mutliple audio/video streams, etc. and some support streaming which means that the data can be sent as a never ending broadcast (such as Internet radio). Containers like codecs can be proprietary or open.

Popular Codecs and Containers

Over the years codecs have improved considerably, producing higher and higher compression ratios with better and better quality. For the purposes of this blog I have only really covered formats which you would use for storing movies and music because to cover every codec (such as those used in voice over IP etc.) would take a considerable document.

While I don't intent to discuss "which is the best quality" format (as that is an extremely hot topic) I will however separate the codecs into what I see as the two obvious generations of compression technologies.

First Generation

These are the codecs that started it all. Whilst not offering the best compression they can often yield good quality results. Also because of their age some are commonly available in hardware decompressors such as portable audio players.

Audio Codecs:
MP3, RealAudio

Video Codecs:
MPEG2, H263, RealVideo

Containers:
AVI, MOV/QT, VOB, FLV, RealMedia

(Proprietary formats shown in italics)

Second Generation

These are the codecs which were developed after everything took off. As such a lot of them have struggled to gain acceptance but as the new generation of high definition video formats come along they will recieve more attention.

Audio Codecs:
AAC, WMA, Vorbis

Video Codecs:
H264/AVC, WMV/VC1, Theora, DIVX, XVID

Containers:
MP4, ASF, Matroska, OGG, M4P

(Proprietary formats shown in italics)

Open Formats

Audio Codecs

MP3 is a nickname for the MPEG1 Audio Layer 3 audio compression codec used by many hardware devices. Due to it popularity and compatibility it has pretty much become the defacto standard for music compression. MP3 has now been superseded by the superior MPEG4 Part 3 known as AAC (see below).

AAC (Advanced Audio Codec) is another name for the MPEG4 Part 3 and MPEG2 Part 7 audio compression codec. It is the successor of MP3 (see above).

Vorbis is an audio compression codec maintained by Xiph which is comparable with WMA and AAC. It is used mainly in an OGG container (see below) but can also be used in others.

Video Codecs

MPEG2 is a nickname for MPEG2 Part 2 which is the video compression codec used by DVD, Digital Terrestrial, Sky, etc. Additionaly MPEG2 support is also mandatory on all HDDVD and BluRay players for backwards compatibility reasons. MPEG2 Part 2 has now been superseded by the superior MPEG4 Part 10 known as H264/AVC (see below).

H263 is a video compression codec used mainly by video conferencing machines and instant messenger systems. It is also used by RealVideo (see below) and in the FLV Adobe/Macromedia flash format on sites such as YouTube and Google Video. It is now superseded by H264 (see below).

H264/AVC (Advanced Video Codec) is another name for the MPEG4 Part 10 video compression codec. It is mandatory that all HDDVD and BluRay players support H264/AVC meaning it will soon be the defacto standard for high definition media.

Theora is a public domain video compression codec currently under development by Xiph. It is said to be comparable with WMV and H264/AVC. It is used mainly in an OGG container (see below) but can also be used with others.

XVID (DivX spelled backwards) is GPL version of the DivX video compression format (see below).

Containers

VOB is the codec container format used by DVD players. It can contain codecs such as MPEG2 (see above), DTS and Dolby Digital (see below).

MP4/3GP is another name for the MPEG4 Part 14 codec container which is based on the old Apple Quicktime .mov file. MP4 is the full implementation which can contain most codecs and 3GP is the cut down version for use on devices such as 3G phones (M4A is just another extension for MP4 used by Apple so that applications could determine if the MP4 container was music or a video). MP4 supports streaming.

MKA/MKV (Matroska) is a versatile container format often used with the H264 video compression codec (see above). It can contain most codecs and an unlimited number of audio and video streams.

OGG is a container format maintained by Xiph. It can contain most codecs and is often used by Vorbis and Theora (see above). It is also often used by Speex which is a open speech compression codec.

Closed Proprietry Formats

Audio Codecs

Dolby Digital is another name for the AC3 audio compression codec. It is mandatory that all HDDVD, BluRay and DVD players support it and it is used by many theatres and home cinema systems.

DTS (Digital Theatre Sound) is an audio compression codec developed by DTS and is often considered to be a superior alternative to Dolby Digital. It is mandatory that all HDDVD and BluRay players support it but is optional on DVD players. It is used by many of the higher quality theaters and home cinema systems.

Microsoft WMA (Windows Media Audio) is an audio compression codec and also a file extension used by Microsoft for the ASF codec container (see below). It is not based on any other codec.

Sony ATRAC is an audio compression format developed by Sony and used mainly in MiniDisc. Sony unseccessfully tried to promote this as a portable audio format in their range of Walkman devices but MP3 was already to heavily established.

RealAudio/RA is an audio compression format developed by RealNetworks. It has been based on a number of codecs over the years including AC3, ATRAC and AAC.

Video Codecs

Microsoft VC1/WMV3 (Windows Media Video 9) is a video compression codec initially developed by Microsoft for WMV and later BluRay and HDDVD. It is mandatory that all HDDVD and BluRay players are able to play VC1.

RealVideo/RV is a video compression format developed by Real Networks. It was originally based on H263 the video conferencing compression format (see above) but is now a proprietary format.

DivX was originally code hacked from Microsofts WMV format and was created so WMV files could be stored in AVI containers instead of ASF ones. It is now a legitimate compression format based on the MPEG4 (H624/AVC - see above) standard.

Containers

Microsoft AVI (Audio Video Interleave) is the old codec container format developed by Microsoft which had many limitations and was therefore replaced by ASF (see below). It cannot properly contain many of the new codecs without using hacks.

Microsoft ASF (Advanced Systems/Streaming Format) is the new codec container format developed by Microsoft. It can contain most codecs and supports streaming.

Adobe FLV is the codec container format used by Macromedia/Adobe Flash player and appears on YouTube and Google Video. It mainly contains video in the old H263 video conferencing format (see above).

Apple MOV/QT (Quicktime) is the old Apple codec container for multiple different codecs which has evolved into MP4 (see above)

Apple M4P is a container used by Apple for AAC and AVC, it can also contain DRM.

RealMedia/RM is a container for RealVideo and RealAudio formats.

Conclusions

One thing that is obvious (at least from my perspective) about all this, is if you ever want to encode any high quality video you should use H264/AVC. Not only is it an open standard but BluRay and HDDVD are already specified to be using it. This all means its will be around for a long while and additionally there will be hardware chipsets that can play it back. Of course your viewers may need to download a codec to play it but to avoid that your only real choice will be to compress your files in whatever format is supported on your viewers platform (which in most cases will be something Microsoft orientated).

For audio, the situation is less clear. If you are happy with 2 channel stereo and MP3 audio quality stick with MP3; its a safe choice with just about everything capable of playing it. If you want something a bit more advanced such as multichannel sound you could use the replacement AAC but hardware support is very minimal, Dolby Digital and DTS have better hardware support but then you have to pay for a license to use whatever encodes it.

With the containers there are quite a few options each having their own advantages and disadvantages. MP4 is probably a safe bet as at least its an MPEG standard but for Windows users ASF is probably going to be easiest.

Of course you could be different and use the Xiph OGG stuff. However IMHO their usage will always be specialist as H264/AVC and AAC are already out there, free, the same quality as the OGG and have a much larger hardware/software base. Its possible that when Theora is complete that might change but I think by that time H264/AVC will be firmly established as the standard for video compression.

Posted by Daniel: 14:56, Sun 13th May 2007