Multimedia Framework Overview

This chapter provides an overview of the Multimedia Framework, and covers:

Multimedia architecture

The Multimedia library architecture is modular, meaning that it consists of units that can be interchanged with each other to provide only the functionality you need. In addition, because it uses the Addon Interfaces library to define its standard interfaces, you can write new components (such as codecs), and easily incorporate new components written by QNX or third parties as they become available, in some cases without having to recompile your application.

A filter is the basic building block of the Multimedia framework. Multimedia Filters:

To process a multimedia file, your application uses only the filters it needs to get the multimedia data, process it, and send it to a target device. We classify filters based on their specific task. A filter can be a:

Reader
Reads data from a source. Examples are audio card readers, data stream readers, and video capture card readers.
Parser
Parses a stream of data into component parts. Examples are MPEG bitstream parsers, WAV format parsers, and AVI format parsers.
Decoder
Decodes compressed or encoded data. Examples are mpeg audio & video decoders, and divx decoders.
Encoder
Encodes raw data into a specific format. Examples are mpeg audio & video encoders.
Writer
Sends data to a destination. Examples are audio card writers, video output writers, and output file writers.

A filter is a generic object. Its functionality is determined by the way it processes data, and the interfaces it implements. A Multimedia library interface defines the set of methods that the library expects filters of a specific type to have. For example, all filters must implement the create and destroy methods defined in the AODeConstructor interface so that the library can create and destroy them, while only filters that read data from a source need to implement the MediaReader interface. The implementation of an interface is declared as a structure of pointers to implemented functions in a defined order.

Each filter can have multiple input and output channels, so if you have a file format that's interleaved with more than one data stream, you can have as many output data channels as the input data stream contains. This is how the MPEG System parser works, for example -- it parses MPEG System data into two streams, video and sound. An MPEG System encoder filter works the opposite way; it has separate input channels for sound and video, and one output channel.

We already provide a number of filters written for the Multimedia framework, but if you need a new one, you have to write only the decoder, parser, or other filter that hasn't already been written. Because the multimedia architecture uses a "plugin" framework, each filter component can easily be integrated (plugged in) with existing filters.

To use the Multimedia library, your application sets up a graph, an abstract object that encapsulates the multimedia filters needed to process a multimedia stream. A graph is stored as a MmGraph_t structure. It's built and destroyed at run time by the player application, and is specific to a particular media stream. Your application adds to the graph all the filters required to process multimedia data from source to destination.

If you know the specific filter required to process the multimedia stream, you can add it to the graph by name. However, a more flexible approach is to use functions that query the library and find the best filter for the data. In this way your application can handle any media type that the library's filters can handle, and use any new filters added to the library without being recompiled.

In the example below, six filters are used to read, parse, decode, and present an MPEG media stream. Each filter has an input channel and at least one output channel, except for special purpose "media reader" and "media writer" filters, which are at the beginning and end of the filter chain.


Multimedia graph example


Multimedia graph example.

In the filters we provide, multimedia streams are synchronized by the audio stream, if present. In the MPEG example above, the writer filters don't communicate directly. Instead, they use the audio writer filter MediaClock interface to the Multimedia library (all filters have hooks to the library) to determine the correct media time. In this way, the video writer continuously checks its media time against the audio writer, and synchronizes its playback accordingly.

QNX components

The Multimedia library includes filters provided by QNX Software Systems that support the multimedia formats listed in the table below. You can write your own filters, or obtain filters from third parties to handle additional formats.


Note: Some filters are available only with the Multimedia Technology Development Kit.

Format Filter(s) Mimetype
MPEG-1 (audio) mpega_parser.so, xing_mpega_decoder.so audio/mp1, audio/mp2, audio/mp3, audio/mpeg
MPEG-1 (video) mpegv_parser.so, ff_mpegv_decoder.so video/mpv
MPEG-1 System (audio and video) mpegs_parser.so, ff_mpegv_decoder.so video/mpeg
AU (audio format) au_parser.so audio/x-au, audio/x-basic
IFF (audio format) iff_parser.so audio/x-iff
AIFF (audio format) aif_parser.so audio/x-aif
AVI (file format, parser filter only, no codec support) avi_parser.so audio/x-avi
OGG (open source audio format) ogg_decoder.so, oggi_decoder.so audio/x-ogg
WAV (audio format, most versions supported) wav_parser.so audio/x-wav
MIDI (audio format) midi_parser.so* audio/midi

* The MIDI filter requires a configuration file, midi.cfg, that defines the mapping of MIDI programs to instrument files. See the MIDI Configuration File appendix for more information.

For more detail about each of the filters included with the Multimedia library, see the Multimedia Filter Reference chapter.

The Multimedia library offers these benefits:

Extending functionality

You may want to extend the Multimedia Framework's functionality by writing filters for various media types not handled by the QNX-provided components. The chapter on Extending the Multimedia Framework covers this subject in detail, and shows examples of writing a parser and decoder.


Note: The source for the QNX-provided filters, as well as the source for the Addon Library and Multimedia library, are available in the Multimedia TDK. You require this TDK to write your own filters.