[openal] MIDI support

[Chris Robinson]

Just pinging this again for anyone that newly registered, in case they 
missed it. Or is interest in it just that low?

On 01/08/2014 02:07 PM, Chris Robinson wrote:
> Not sure who all is here yet, but I suppose it's a good a time as any to
> get a conversation going. This will be a long email, as it tries to
> explain how it's (currently) used, as well as my reasoning. If you have
> questions, or suggestions for changing something, please do. :)
>
>
> 1. What and why
>
> In recent commits with OpenAL Soft, I've been adding support for a MIDI
> extension (tentatively called ALC_SOFT_midi_interface). I realize the
> majority of games do and will use things like mp3 or ogg, however I feel
> there is potential utility in making MIDI available for games and apps.
>
> Modern ports/remakes of old game engines would obviously have some
> benefit of built-in MIDI support, but I also feel even new games could
> utilize MIDI for a much more dynamic music system compared to what you
> could get from streaming pre-rendered audio. The amount of memory and
> CPU power available these days is more than enough to handle software
> MIDI synths with quality soundfonts, too. As well, a good soundfont can
> be smaller than a game's collection of compressed music (MIDI files
> themselves are of course insanely small, so there's little size overhead
> to adding more music).
>
> I'll start off by saying the MIDI extension is based on the SF2 spec. It
> allows apps to specify their own soundfont so it can have some assurance
> about the sound it gets, rather than being at the whim of the device or
> system configuration. The actual quality of the sound may differ some,
> but that's how it is with standard OpenAL too (different methods for
> resampling, effects, filters, etc). By having a base spec to work with,
> I hope to avoid problems many old games would run into with MIDI.
>
> Currently OpenAL Soft implements MIDI using FluidSynth, however I hope
> to be able to handle it all internally (a lot of what's needed is
> already there, but some more groundwork is still needed).
>
>
> 2. API basics
>
> The extension API is designed in a way that tries to avoid messing about
> with file formats (with one exception, see Soundfonts). So instead of
> loading a MIDI file into a buffer and playing it via a source or
> something, you specify timestamped MIDI events. This allows apps
> flexibility in how they wish to store the sequence data, and the
> capabilities of the sequence data (loops, dynamic volume/instrument
> alterations, etc).
>
> void alMidiEventSOFT(ALuint64SOFT time, ALenum event, ALsizei channel,
>                       ALsizei param1, ALsizei param2);
> void alMidiSysExSOFT(ALuint64SOFT time, const ALbyte *data,
>                       ALsizei size);
>
> The time is based on a microsecond clock. Because a 32-bit value would
> overflow relatively quickly, I decided to make it 64-bit. The current
> clock time can be retrieved with passing AL_MIDI_CLOCK_SOFT to:
>
> ALint64SOFT alGetInteger64SOFT(ALenum pname);
> void alGetInteger64vSOFT(ALenum pname, ALint64SOFT *values);
>
> So for example, if you want to play a note a half-second after the
> current clock time, for one second, you could do:
>
> ALuint64SOFT tval = alGetInteger64SOFT(AL_MIDI_CLOCK_SOFT);
> alMidiEventSOFT(tval +  500000, AL_NOTEON_SOFT, 0, 60, 64);
> alMidiEventSOFT(tval + 1500000, AL_NOTEOFF_SOFT, 0, 60, 0);
>
> Specifying multiple events with the same clock time processes them in
> the order specified. An application could also execute events in
> "real-time" by always passing a timestamp of 0 (any timestamp before the
> current clock will execute ASAP; it won't try to pretend the event
> actually happened at the time specified). Though this obviously limits
> the event granularity to however often OpenAL processes audio updates --
> specifying timestamps after the current clock allows for more precise
> event processing.
>
> By default, the MIDI engine is not in a playing state. The means the
> clock will not be incrementing and events will not be processed. To
> control MIDI processing, you have the functions:
>
> void alMidiPlaySOFT(void);
> void alMidiPauseSOFT(void);
> void alMidiStopSOFT(void);
> void alMidiResetSOFT(void);
>
> alMidiPlaySOFT will start/resume MIDI processing, and the clock will
> start incrementing.
>
> alMidiPauseSOFT will pause MIDI processing, and the clock will stop
> incrementing. Any currently playing notes will stay on.
>
> alMidiStopSOFT will stop MIDI processing, and the clock will reset to 0.
> Any pending MIDI events before the current clock (as of the time it as
> called) will be processed, and all channels will get an all-notes-off
> event (which will put any playing notes into a release phase). All other
> events are flushed.
>
> alMidiResetSOFT will stop MIDI processing, and the clock will reset to
> 0. All MIDI events will be flushed, and the MIDI engine will be reset to
> power-on status.
>
>
> 3. Soundfonts
>
> This one may be difficult to properly explain. The structure used to
> handle soundfonts is heavily distilled from the HYDRA structure
> described in the SF2 spec. The spec itself mentions that the structure
> is not optimized for run-time synthesis or on-the-fly editing, so I
> tried to cut out a lot of the stuff that's unneeded for, or
> unnecessarily complicates, run-time synthesis. Basically a lot of things
> got merged, so there's a bit more load-time work for better run-time
> processing (or so I hope).
>
> Soundfonts are broken up into 3 objects. Soundfont, Preset, and
> Fontsound (name is debatable, but I'm not sure what else to call it to
> avoid confusion or clashes). A soundfont contains PCM samples and a
> collection of presets, which contain a collection of fontsounds, which
> contain generator properties, modulators, and sample info. These objects
> have the standard alGen*, alDelete*, and alIs* functions.
>
> In a break from standard AL API design, a function is provided to load
> an SF2 format soundfont into a soundfont object. I decided to do this
> because the SF2 format can be rather difficult to parse, and even more
> difficult to properly process and load into AL objects. And it gets even
> more difficult if you want proper error checking. I think it's a bit
> unfair for apps to rely on 3rd party libs (like Alure) or to create a
> loader themselves to properly load a soundfont.
>
> void alLoadSoundfontSOFT(ALuint id,
>                           size_t(*cb)(ALvoid*,size_t,ALvoid*),
>                           ALvoid *user);
>
> Instead of taking a filename, it's given a read callback. This allows
> apps to store soundfonts however they want and not be restricted to
> having them on disk for standard IO access (i.e. they can be stored in a
> resource archive, a zip file or whatever). For example:
>
> size_t read_func(ALvoid *buffer, suze_t len, ALvoid *user)
> {
>      return fread(buffer, 1, len, (FILE*)user);
> }
> ...
> FILE *file = fopen("my-soundfont.sf2", "rb");
>
> ALuint sfont;
> alGenSoundfontsSOFT(1, &sfont);
> alLoadSoundfontSOFT(sfont, read_func, file);
> fclose(file);
>
> Selecting a soundfont to use is done with the functions:
>
> void alMidiSoundfontSOFT(ALuint id);
> void alMidiSoundfontvSOFT(ALsizei count, const ALuint *ids);
>
> These can only be called when MIDI processing is stopped or reset.
> Selecting a soundfont will deselect any previously selected. To deselect
> all soundfonts without selecting a new one, call alMidiSoundfontvSOFT
> with a count of 0. A soundfont cannot be deleted if it's currently
> selected.
>
> A default soundfont can be accessed using soundfont ID 0. This soundfont
> can be whatever the user or system may have configured. Obviously this
> isn't something that can be relied on, but it can be a useful option if
> a user has a preferred soundfont and your music uses standard MIDI
> functionality (i.e. doesn't rely on a non-standard instrument set, or on
> modulators that significantly alter the behavior of MIDI controllers).
>
>
> This email is getting a bit long, so I'll probably end it here. There's
> obviously more to it, and I can attempt to explain the rest of the
> soundfont stuff if anyone's curious. But for now, I wouldn't mind some
> feedback on what I've written about so far. :)
>
> Thanks!