[openal] MIDI support

Mon Mar 24 11:13:13 EDT 2014

> On 01/08/2014 02:07 PM, Chris Robinson wrote:
>
> Not sure who all is here yet, but I suppose it's a good a time as any to 
> get a conversation going. This will be a long email, as it tries to 
> explain how it's (currently) used, as well as my reasoning. If you have 
> questions, or suggestions for changing something, please do. :)

Thanks Chris and Ryan for keeping the OpenAL.org and OpenAL spec community driven and vibrant.

I am working on an enhancement bugreport for JogAmp JOAL to support your OpenAL-soft MIDI extensions from Java.
https://jogamp.org/bugzilla/show_bug.cgi?id=1002

Comments to the OpenAL-soft MIDI specification inline below:

> 1. What and why
>
> In recent commits with OpenAL Soft, I've been adding support for a MIDI 
> extension (tentatively called ALC_SOFT_midi_interface). I realize the 
> majority of games do and will use things like mp3 or ogg, however I feel 
> there is potential utility in making MIDI available for games and apps.
>
> Modern ports/remakes of old game engines would obviously have some 
> benefit of built-in MIDI support, but I also feel even new games could 
> utilize MIDI for a much more dynamic music system compared to what you 
> could get from streaming pre-rendered audio. The amount of memory and 
> CPU power available these days is more than enough to handle software 
> MIDI synths with quality soundfonts, too. As well, a good soundfont can 
> be smaller than a game's collection of compressed music (MIDI files 
> themselves are of course insanely small, so there's little size overhead 
> to adding more music).
>
> I'll start off by saying the MIDI extension is based on the SF2 spec. It 
> allows apps to specify their own soundfont so it can have some assurance 
> about the sound it gets, rather than being at the whim of the device or 
> system configuration. The actual quality of the sound may differ some, 
> but that's how it is with standard OpenAL too (different methods for 
> resampling, effects, filters, etc). By having a base spec to work with, 
> I hope to avoid problems many old games would run into with MIDI.
>
> Currently OpenAL Soft implements MIDI using FluidSynth, however I hope 
> to be able to handle it all internally (a lot of what's needed is 
> already there, but some more groundwork is still needed).

It would be great if the OpenAL-soft MIDI extension can handle external general MIDI input and output devices as well.
For many games and apps it would be good to use all kinds of MIDI devices ranging all from keyboards to guitars and "3D" multibutton DJ equipment for input.
Using OpenAL to orchestrate hardware synthesizers with spatialisation is all welcome as well.

It would be awesome if the OpenAL-soft MIDI extension had a callback api for real time audio generation for use to add a custom MIDI synth implemented using software or hardware DSP.
http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing

> 2. API basics
>
> The extension API is designed in a way that tries to avoid messing about 
> with file formats (with one exception, see Soundfonts). So instead of 
> loading a MIDI file into a buffer and playing it via a source or 
> something, you specify timestamped MIDI events. This allows apps 
> flexibility in how they wish to store the sequence data, and the 
> capabilities of the sequence data (loops, dynamic volume/instrument 
> alterations, etc).
>
> void alMidiEventSOFT(ALuint64SOFT time, ALenum event, ALsizei channel,
>                       ALsizei param1, ALsizei param2);
> void alMidiSysExSOFT(ALuint64SOFT time, const ALbyte *data,
>                       ALsizei size);
>
> The time is based on a microsecond clock. Because a 32-bit value would 
> overflow relatively quickly, I decided to make it 64-bit. The current 
> clock time can be retrieved with passing AL_MIDI_CLOCK_SOFT to:
>
> ALint64SOFT alGetInteger64SOFT(ALenum pname);
> void alGetInteger64vSOFT(ALenum pname, ALint64SOFT *values);
>
> So for example, if you want to play a note a half-second after the 
> current clock time, for one second, you could do:
>
> ALuint64SOFT tval = alGetInteger64SOFT(AL_MIDI_CLOCK_SOFT);
> alMidiEventSOFT(tval +  500000, AL_NOTEON_SOFT, 0, 60, 64);
> alMidiEventSOFT(tval + 1500000, AL_NOTEOFF_SOFT, 0, 60, 0);
>
> Specifying multiple events with the same clock time processes them in 
> the order specified. An application could also execute events in 
> "real-time" by always passing a timestamp of 0 (any timestamp before the 
> current clock will execute ASAP; it won't try to pretend the event 
> actually happened at the time specified). Though this obviously limits 
> the event granularity to however often OpenAL processes audio updates -- 
> specifying timestamps after the current clock allows for more precise 
> event processing.
>
> By default, the MIDI engine is not in a playing state. The means the 
> clock will not be incrementing and events will not be processed. To 
> control MIDI processing, you have the functions:
>
> void alMidiPlaySOFT(void);
> void alMidiPauseSOFT(void);
> void alMidiStopSOFT(void);
> void alMidiResetSOFT(void);
>
> alMidiPlaySOFT will start/resume MIDI processing, and the clock will 
> start incrementing.
>
> alMidiPauseSOFT will pause MIDI processing, and the clock will stop 
> incrementing. Any currently playing notes will stay on.
>
> alMidiStopSOFT will stop MIDI processing, and the clock will reset to 0. 
> Any pending MIDI events before the current clock (as of the time it as 
> called) will be processed, and all channels will get an all-notes-off 
> event (which will put any playing notes into a release phase). All other 
> events are flushed.
>
> alMidiResetSOFT will stop MIDI processing, and the clock will reset to 
> 0. All MIDI events will be flushed, and the MIDI engine will be reset to 
> power-on status.

Is the alMidi* operating on the current bound OpenAL context?
How can I manipulate the OpenAL Source for the Midi engine?
How can I use multiple Midi engines?

> 3. Soundfonts
>
> This one may be difficult to properly explain. The structure used to 
> handle soundfonts is heavily distilled from the HYDRA structure 
> described in the SF2 spec. The spec itself mentions that the structure 
> is not optimized for run-time synthesis or on-the-fly editing, so I 
> tried to cut out a lot of the stuff that's unneeded for, or 
> unnecessarily complicates, run-time synthesis. Basically a lot of things 
> got merged, so there's a bit more load-time work for better run-time 
> processing (or so I hope).
>
> Soundfonts are broken up into 3 objects. Soundfont, Preset, and 
> Fontsound (name is debatable, but I'm not sure what else to call it to 
> avoid confusion or clashes). A soundfont contains PCM samples and a 
> collection of presets, which contain a collection of fontsounds, which 
> contain generator properties, modulators, and sample info. These objects 
> have the standard alGen*, alDelete*, and alIs* functions.
>
> In a break from standard AL API design, a function is provided to load 
> an SF2 format soundfont into a soundfont object. I decided to do this 
> because the SF2 format can be rather difficult to parse, and even more 
> difficult to properly process and load into AL objects. And it gets even 
> more difficult if you want proper error checking. I think it's a bit 
> unfair for apps to rely on 3rd party libs (like Alure) or to create a 
> loader themselves to properly load a soundfont.
>
> void alLoadSoundfontSOFT(ALuint id,
>                           size_t(*cb)(ALvoid*,size_t,ALvoid*),
>                           ALvoid *user);
>
> Instead of taking a filename, it's given a read callback. This allows 
> apps to store soundfonts however they want and not be restricted to 
> having them on disk for standard IO access (i.e. they can be stored in a 
> resource archive, a zip file or whatever). For example:
>
> size_t read_func(ALvoid *buffer, suze_t len, ALvoid *user)
> {
>      return fread(buffer, 1, len, (FILE*)user);
> }
> ...
> FILE *file = fopen("my-soundfont.sf2", "rb");
>
> ALuint sfont;
> alGenSoundfontsSOFT(1, &sfont);
> alLoadSoundfontSOFT(sfont, read_func, file);
> fclose(file);
>
> Selecting a soundfont to use is done with the functions:
>
> void alMidiSoundfontSOFT(ALuint id);
> void alMidiSoundfontvSOFT(ALsizei count, const ALuint *ids);
>
> These can only be called when MIDI processing is stopped or reset. 
> Selecting a soundfont will deselect any previously selected. To deselect 
> all soundfonts without selecting a new one, call alMidiSoundfontvSOFT 
> with a count of 0. A soundfont cannot be deleted if it's currently selected.
>
> A default soundfont can be accessed using soundfont ID 0. This soundfont 
> can be whatever the user or system may have configured. Obviously this 
> isn't something that can be relied on, but it can be a useful option if 
> a user has a preferred soundfont and your music uses standard MIDI 
> functionality (i.e. doesn't rely on a non-standard instrument set, or on 
> modulators that significantly alter the behavior of MIDI controllers).
Is there a function to un-load the loaded soundfonts?

> This email is getting a bit long, so I'll probably end it here. There's 
> obviously more to it, and I can attempt to explain the rest of the 
> soundfont stuff if anyone's curious. But for now, I wouldn't mind some 
> feedback on what I've written about so far. :)
>
> Thanks!
Cheers
Xerxes