[openal] MIDI support

[Chris Robinson]

Not sure who all is here yet, but I suppose it's a good a time as any to 
get a conversation going. This will be a long email, as it tries to 
explain how it's (currently) used, as well as my reasoning. If you have 
questions, or suggestions for changing something, please do. :)


1. What and why

In recent commits with OpenAL Soft, I've been adding support for a MIDI 
extension (tentatively called ALC_SOFT_midi_interface). I realize the 
majority of games do and will use things like mp3 or ogg, however I feel 
there is potential utility in making MIDI available for games and apps.

Modern ports/remakes of old game engines would obviously have some 
benefit of built-in MIDI support, but I also feel even new games could 
utilize MIDI for a much more dynamic music system compared to what you 
could get from streaming pre-rendered audio. The amount of memory and 
CPU power available these days is more than enough to handle software 
MIDI synths with quality soundfonts, too. As well, a good soundfont can 
be smaller than a game's collection of compressed music (MIDI files 
themselves are of course insanely small, so there's little size overhead 
to adding more music).

I'll start off by saying the MIDI extension is based on the SF2 spec. It 
allows apps to specify their own soundfont so it can have some assurance 
about the sound it gets, rather than being at the whim of the device or 
system configuration. The actual quality of the sound may differ some, 
but that's how it is with standard OpenAL too (different methods for 
resampling, effects, filters, etc). By having a base spec to work with, 
I hope to avoid problems many old games would run into with MIDI.

Currently OpenAL Soft implements MIDI using FluidSynth, however I hope 
to be able to handle it all internally (a lot of what's needed is 
already there, but some more groundwork is still needed).


2. API basics

The extension API is designed in a way that tries to avoid messing about 
with file formats (with one exception, see Soundfonts). So instead of 
loading a MIDI file into a buffer and playing it via a source or 
something, you specify timestamped MIDI events. This allows apps 
flexibility in how they wish to store the sequence data, and the 
capabilities of the sequence data (loops, dynamic volume/instrument 
alterations, etc).

void alMidiEventSOFT(ALuint64SOFT time, ALenum event, ALsizei channel,
                      ALsizei param1, ALsizei param2);
void alMidiSysExSOFT(ALuint64SOFT time, const ALbyte *data,
                      ALsizei size);

The time is based on a microsecond clock. Because a 32-bit value would 
overflow relatively quickly, I decided to make it 64-bit. The current 
clock time can be retrieved with passing AL_MIDI_CLOCK_SOFT to:

ALint64SOFT alGetInteger64SOFT(ALenum pname);
void alGetInteger64vSOFT(ALenum pname, ALint64SOFT *values);

So for example, if you want to play a note a half-second after the 
current clock time, for one second, you could do:

ALuint64SOFT tval = alGetInteger64SOFT(AL_MIDI_CLOCK_SOFT);
alMidiEventSOFT(tval +  500000, AL_NOTEON_SOFT, 0, 60, 64);
alMidiEventSOFT(tval + 1500000, AL_NOTEOFF_SOFT, 0, 60, 0);

Specifying multiple events with the same clock time processes them in 
the order specified. An application could also execute events in 
"real-time" by always passing a timestamp of 0 (any timestamp before the 
current clock will execute ASAP; it won't try to pretend the event 
actually happened at the time specified). Though this obviously limits 
the event granularity to however often OpenAL processes audio updates -- 
specifying timestamps after the current clock allows for more precise 
event processing.

By default, the MIDI engine is not in a playing state. The means the 
clock will not be incrementing and events will not be processed. To 
control MIDI processing, you have the functions:

void alMidiPlaySOFT(void);
void alMidiPauseSOFT(void);
void alMidiStopSOFT(void);
void alMidiResetSOFT(void);

alMidiPlaySOFT will start/resume MIDI processing, and the clock will 
start incrementing.

alMidiPauseSOFT will pause MIDI processing, and the clock will stop 
incrementing. Any currently playing notes will stay on.

alMidiStopSOFT will stop MIDI processing, and the clock will reset to 0. 
Any pending MIDI events before the current clock (as of the time it as 
called) will be processed, and all channels will get an all-notes-off 
event (which will put any playing notes into a release phase). All other 
events are flushed.

alMidiResetSOFT will stop MIDI processing, and the clock will reset to 
0. All MIDI events will be flushed, and the MIDI engine will be reset to 
power-on status.


3. Soundfonts

This one may be difficult to properly explain. The structure used to 
handle soundfonts is heavily distilled from the HYDRA structure 
described in the SF2 spec. The spec itself mentions that the structure 
is not optimized for run-time synthesis or on-the-fly editing, so I 
tried to cut out a lot of the stuff that's unneeded for, or 
unnecessarily complicates, run-time synthesis. Basically a lot of things 
got merged, so there's a bit more load-time work for better run-time 
processing (or so I hope).

Soundfonts are broken up into 3 objects. Soundfont, Preset, and 
Fontsound (name is debatable, but I'm not sure what else to call it to 
avoid confusion or clashes). A soundfont contains PCM samples and a 
collection of presets, which contain a collection of fontsounds, which 
contain generator properties, modulators, and sample info. These objects 
have the standard alGen*, alDelete*, and alIs* functions.

In a break from standard AL API design, a function is provided to load 
an SF2 format soundfont into a soundfont object. I decided to do this 
because the SF2 format can be rather difficult to parse, and even more 
difficult to properly process and load into AL objects. And it gets even 
more difficult if you want proper error checking. I think it's a bit 
unfair for apps to rely on 3rd party libs (like Alure) or to create a 
loader themselves to properly load a soundfont.

void alLoadSoundfontSOFT(ALuint id,
                          size_t(*cb)(ALvoid*,size_t,ALvoid*),
                          ALvoid *user);

Instead of taking a filename, it's given a read callback. This allows 
apps to store soundfonts however they want and not be restricted to 
having them on disk for standard IO access (i.e. they can be stored in a 
resource archive, a zip file or whatever). For example:

size_t read_func(ALvoid *buffer, suze_t len, ALvoid *user)
{
     return fread(buffer, 1, len, (FILE*)user);
}
...
FILE *file = fopen("my-soundfont.sf2", "rb");

ALuint sfont;
alGenSoundfontsSOFT(1, &sfont);
alLoadSoundfontSOFT(sfont, read_func, file);
fclose(file);

Selecting a soundfont to use is done with the functions:

void alMidiSoundfontSOFT(ALuint id);
void alMidiSoundfontvSOFT(ALsizei count, const ALuint *ids);

These can only be called when MIDI processing is stopped or reset. 
Selecting a soundfont will deselect any previously selected. To deselect 
all soundfonts without selecting a new one, call alMidiSoundfontvSOFT 
with a count of 0. A soundfont cannot be deleted if it's currently selected.

A default soundfont can be accessed using soundfont ID 0. This soundfont 
can be whatever the user or system may have configured. Obviously this 
isn't something that can be relied on, but it can be a useful option if 
a user has a preferred soundfont and your music uses standard MIDI 
functionality (i.e. doesn't rely on a non-standard instrument set, or on 
modulators that significantly alter the behavior of MIDI controllers).


This email is getting a bit long, so I'll probably end it here. There's 
obviously more to it, and I can attempt to explain the rest of the 
soundfont stuff if anyone's curious. But for now, I wouldn't mind some 
feedback on what I've written about so far. :)

Thanks!