From petrenko at usc.edu  Sat Jul 18 04:58:04 2020
From: petrenko at usc.edu (Aleksei Petrenko)
Date: Sat, 18 Jul 2020 01:58:04 -0700
Subject: [openal] Programmatic access to sounds played through OpenAL
Message-ID: <CAFh6kTeBB+WXSj191HfP8AURx9JJA8Bin21A4dHRB+4J_ZQyWA@mail.gmail.com>

Hi everyone,

I am looking for help with a rather unusual problem.
For a reinforcement learning project, we want to run a computer game much
faster than realtime, and on top of capturing the rendered output, we also
want to capture the sound that the player perceives.

Naturally, the game uses OpenAL, which I have very little experience with.
I understand that the game defines multiple emitters, filters, there's
Doppler shift depending on object velocities, etc. So, the sound
pipeline is rather complicated and the final sound is highly non-trivial. I
cannot just rewrite everything without using OpenAL.

My question is, is it possible to get programmatic access to the sounds
played during every particular frame of the game?

An example to be more concrete. Let's say the normal framerate of the game
is 60FPS. I am running the game 10 times faster than realtime, so at 600FPS
(10x is not important, I want to run the game as fast as hardware allows,
the exact speedup is not known). At some point, a game event triggers a
sound that under normal circumstances would play for 1 second, so for 60
game ticks. In my fast simulation it only lasts 0.1 seconds, and the same
60 game ticks.

Can I record the sound samples that would have been played during every
game tick if it ran normally, such that if I record everything and play it
back at the normal speed, it would be indistinguishable from normal sounds?

This question on SO didn't get much attention:
https://stackoverflow.com/questions/62945506/programmatic-access-to-a-sound-played-through-openal
<https://urldefense.com/v3/__https://stackoverflow.com/questions/62945506/programmatic-access-to-a-sound-played-through-openal__;!!LIr3w8kk_Xxm!5zD0XPJENUmu2H1nDIKTLu7b6izQvainqM5bIciOFWAaCUHNL4r6xTAc2XleU4g$>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openal.org/pipermail/openal/attachments/20200718/4dd50a36/attachment.htm>

From chris.kcat at gmail.com  Sat Jul 18 06:32:49 2020
From: chris.kcat at gmail.com (Chris Robinson)
Date: Sat, 18 Jul 2020 03:32:49 -0700
Subject: [openal] Programmatic access to sounds played through OpenAL
In-Reply-To: <CAFh6kTeBB+WXSj191HfP8AURx9JJA8Bin21A4dHRB+4J_ZQyWA@mail.gmail.com>
References: <CAFh6kTeBB+WXSj191HfP8AURx9JJA8Bin21A4dHRB+4J_ZQyWA@mail.gmail.com>
Message-ID: <20200718033249.7e2cac81@kittycat.default>

On Sat, 18 Jul 2020 01:58:04 -0700
Aleksei Petrenko <petrenko at usc.edu> wrote:

> My question is, is it possible to get programmatic access to the
> sounds played during every particular frame of the game?
> 
> An example to be more concrete. Let's say the normal framerate of the
> game is 60FPS. I am running the game 10 times faster than realtime,
> so at 600FPS (10x is not important, I want to run the game as fast as
> hardware allows, the exact speedup is not known). At some point, a
> game event triggers a sound that under normal circumstances would
> play for 1 second, so for 60 game ticks. In my fast simulation it
> only lasts 0.1 seconds, and the same 60 game ticks.

Hi.

With OpenAL Soft's ALC_SOFT_loopback extension, it should be possible
to do this, as long as you're able to modify your program's source. If
the ALC_SOFT_loopback extension is supported, you can get the
alcLoopbackOpenDeviceSOFT function (with alcGetProcAddress), and create
an ALCdevice using that. Specify the render format you want when calling
alcCreateContext, and the normal AL functions can be used as normal when
the context is made current.

At that point, you're in control of when and how fast samples are
mixed. Call alcRenderSamplesSOFT to render samples and fill in your
provided buffer. Ideally you'd render ~20ms worth of samples per call
(512 to 1024 sample frames at 44100hz), but any size can work and it
can be called as often as your CPU allows. So for example:

#include "alext.h"

LPALCLOOPBACKOPENDEVICESOFT alcLoopbackOpenDeviceSOFT;
LPALCRENDERSAMPLESSOFT alcRenderSamplesSOFT;

...sound initialization...
if(!alcIsExtensionPresent(NULL, "ALC_SOFT_loopback"))
{
    error("Loopback not supported");
    abort();
}

alcLoopbackOpenDeviceSOFT = alcGetProcAddress(NULL,
    "alcLoopbackOpenDeviceSOFT");
alcRenderSamplesSOFT = alcGetProcAddress(NULL, "alcRenderSamplesSOFT");

ALCdevice *device = alcLoopbackOpenDeviceSOFT(NULL);
ALCint attrs[] = {
    /* Standard 16-bit stereo 44.1khz. Can change as desired. */
    ALC_FORMAT_TYPE_SOFT, ALC_SHORT_SOFT,
    ALC_FORMAT_CHANNELS_SOFT, ALC_STEREO_SOFT,
    ALC_FREQUENCY, 44100,

    /* end-of-list */
    0
};
ALCcontext *context = alcCreateContext(device, attrs);
alcMakeContextCurrent(context);

...in main loop...
    ..make normal AL update calls..

    /* Ensure 'buffer' can hold 1024 sample frames when calling (4096
     * bytes for 16-bit stereo). */
    alcRenderSamplesSOFT(device, buffer, 1024);

    ..1024 sample frames are now in 'buffer'..
...

The above main loop can run faster than real-time. An AL call made after
rendering 44100 sample frames would act as if it was made after one
second of playback. If all the resulting samples are concatenated and
played back at normal speed, it would sound indistinguishable from
normal device playback. See
<https://openal-soft.org/openal-extensions/SOFT_loopback.txt> for more
information about the extension.

From petrenko at usc.edu  Sat Jul 18 17:55:25 2020
From: petrenko at usc.edu (Aleksei Petrenko)
Date: Sat, 18 Jul 2020 14:55:25 -0700
Subject: [openal] Programmatic access to sounds played through OpenAL
In-Reply-To: <20200718033249.7e2cac81@kittycat.default>
References: <CAFh6kTeBB+WXSj191HfP8AURx9JJA8Bin21A4dHRB+4J_ZQyWA@mail.gmail.com>
 <20200718033249.7e2cac81@kittycat.default>
Message-ID: <CAFh6kTfggbBRMURydAD8sMqCOm0NkoifmoN9vq5GR3V3Wdr_8g@mail.gmail.com>

Hi Chris,

this is fantastic! Exactly the kind of advice I was looking for.
We'll try to make this work, I'll let you know how it goes!


Best regards,
Aleksei

On Sat, Jul 18, 2020 at 3:30 AM Chris Robinson <chris.kcat at gmail.com> wrote:

> On Sat, 18 Jul 2020 01:58:04 -0700
> Aleksei Petrenko <petrenko at usc.edu> wrote:
>
> > My question is, is it possible to get programmatic access to the
> > sounds played during every particular frame of the game?
> >
> > An example to be more concrete. Let's say the normal framerate of the
> > game is 60FPS. I am running the game 10 times faster than realtime,
> > so at 600FPS (10x is not important, I want to run the game as fast as
> > hardware allows, the exact speedup is not known). At some point, a
> > game event triggers a sound that under normal circumstances would
> > play for 1 second, so for 60 game ticks. In my fast simulation it
> > only lasts 0.1 seconds, and the same 60 game ticks.
>
> Hi.
>
> With OpenAL Soft's ALC_SOFT_loopback extension, it should be possible
> to do this, as long as you're able to modify your program's source. If
> the ALC_SOFT_loopback extension is supported, you can get the
> alcLoopbackOpenDeviceSOFT function (with alcGetProcAddress), and create
> an ALCdevice using that. Specify the render format you want when calling
> alcCreateContext, and the normal AL functions can be used as normal when
> the context is made current.
>
> At that point, you're in control of when and how fast samples are
> mixed. Call alcRenderSamplesSOFT to render samples and fill in your
> provided buffer. Ideally you'd render ~20ms worth of samples per call
> (512 to 1024 sample frames at 44100hz), but any size can work and it
> can be called as often as your CPU allows. So for example:
>
> #include "alext.h"
>
> LPALCLOOPBACKOPENDEVICESOFT alcLoopbackOpenDeviceSOFT;
> LPALCRENDERSAMPLESSOFT alcRenderSamplesSOFT;
>
> ...sound initialization...
> if(!alcIsExtensionPresent(NULL, "ALC_SOFT_loopback"))
> {
>     error("Loopback not supported");
>     abort();
> }
>
> alcLoopbackOpenDeviceSOFT = alcGetProcAddress(NULL,
>     "alcLoopbackOpenDeviceSOFT");
> alcRenderSamplesSOFT = alcGetProcAddress(NULL, "alcRenderSamplesSOFT");
>
> ALCdevice *device = alcLoopbackOpenDeviceSOFT(NULL);
> ALCint attrs[] = {
>     /* Standard 16-bit stereo 44.1khz. Can change as desired. */
>     ALC_FORMAT_TYPE_SOFT, ALC_SHORT_SOFT,
>     ALC_FORMAT_CHANNELS_SOFT, ALC_STEREO_SOFT,
>     ALC_FREQUENCY, 44100,
>
>     /* end-of-list */
>     0
> };
> ALCcontext *context = alcCreateContext(device, attrs);
> alcMakeContextCurrent(context);
>
> ...in main loop...
>     ..make normal AL update calls..
>
>     /* Ensure 'buffer' can hold 1024 sample frames when calling (4096
>      * bytes for 16-bit stereo). */
>     alcRenderSamplesSOFT(device, buffer, 1024);
>
>     ..1024 sample frames are now in 'buffer'..
> ...
>
> The above main loop can run faster than real-time. An AL call made after
> rendering 44100 sample frames would act as if it was made after one
> second of playback. If all the resulting samples are concatenated and
> played back at normal speed, it would sound indistinguishable from
> normal device playback. See
> <
> https://urldefense.com/v3/__https://openal-soft.org/openal-extensions/SOFT_loopback.txt__;!!LIr3w8kk_Xxm!83zmI7TW0wMP1j4irCuIr_UIc3DS4dOu99HuEnONDlUCVywFLZHZZskL-XP7CqY$
> > for more
> information about the extension.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openal.org/pipermail/openal/attachments/20200718/988952a0/attachment.htm>