salzian / bevy_fmod Goto Github PK

Right now, we use .unwrap()a lot 😨 I'm sure there is a better way in many situations. We should resort to logging and discuss what to do in case of an unrecoverable failure. Audio is important, but running into an unwrap would crash the game.

This is because there is no unpause trait in AudioSinkPlayback. When the FMOD event is paused toggle() will call play() from AudioSinkPlayback but this is not sufficient.

When adding RapierPhysicsPlugin from bevy_rapier3d to the spatial example you will get compilation errors saying Velocity is ambiguous. But in my own project I've also had a case where it silently broke the spatial audio (resulting in just 2d audio).

I think it would be a good time to also think about the following question: Would it be beneficial if a physics engine can provide the velocity instead?

1. Avoids redundant calculation of velocity, physics engines will calculate this anyway.
2. More accurate velocity if it's calculated by a physics engine?
3. Avoids worry about ambiguous imports.

Let's also include bevy_xpbd when we come to a solution.

According to the readme the demo_project needs to be in the root folder, but for the examples you need to place it in the examples folder currently.

Also include that crates.io badge everyone has.

Evaluate and apply if necessary
https://github.com/tbillington/bevy_best_practices

At low speeds it sounds good (although I don't really have a reference so far) but at high speeds the doppler effect starts to become wobbly. Best to demonstrate: https://youtu.be/WKCePPvmycs

I haven't investigated so far, open to suggestions. This issue assumes #28 is fixed.

https://github.com/Salzian/bevy_fmod/blob/9a4d23427f446f99f435c6542df5b956bcc9360d/src/components/velocity.rs#L27C23-L27C23

Just noticed this. We could maybe use time.period.as_secs_f32()

It would be nice to play a spatial sound from any location without necessarily having to attach it to an entity.

So for example, I have a shotgun with some reverb, I want it playing from the location of the shot, not attached to the weapon entity, because I don't want the sound following the weapon around (my game is fast, you'd notice it).

It seems that right now I would have to create an entity, attach a SpatialAudioBundle to it, track it's playback state, then delete the entity.

Cloned the repo and got the following:

Updating files: 100% (42/42), done.
Downloading examples/demo_project/Assets/return.wav (58 MB)
Error downloading object: examples/demo_project/Assets/return.wav (a00f615): Smudge error: Error downloading examples/demo_project/Assets/return.wav (a00f615dd4b2ea19ed71134825409865ac4b8ae16be21d2b23f812abdc13650f): batch response: This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access.

Errors logged to 'C:\Rust\fmod\bevy_fmod\.git\lfs\logs\20230911T174432.0371003.log'.
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: examples/demo_project/Assets/return.wav: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

Maybe it would be good to replace this with a compressed file, or is there a reason for such a large audio file?

If you make your audio source a child, then spatial sounds come out from 0,0,0 globally.

You can test with:

//! Spatial audio:
//! The spatial audio bundles provide all the components necessary for spatial audio.
//! Make sure your sound has a spatializer assigned to it in FMOD Studio.
//!
//! Controls:
//! Use the arrow keys to move around.

use bevy::prelude::*;
use bevy_fmod::prelude::AudioSource;
use bevy_fmod::prelude::*;

fn main() {
    App::new()
        .add_plugins((
            DefaultPlugins,
            FmodPlugin {
                audio_banks_paths: &[
                    "./assets/audio/demo_project/Build/Desktop/Master.bank",
                    "./assets/audio/demo_project/Build/Desktop/Master.strings.bank",
                    "./assets/audio/demo_project/Build/Desktop/Music.bank",
                ],
            },
        ))
        .add_systems(Startup, setup_scene)
        .add_systems(PostStartup, play_music)
        .add_systems(Update, orbit_audio_source)
        .add_systems(Update, update_listener)
        .run();
}

#[derive(Component)]
struct Cube;

fn setup_scene(
    mut commands: Commands,
    mut meshes: ResMut<Assets<Mesh>>,
    mut materials: ResMut<Assets<StandardMaterial>>,
    studio: Res<FmodStudio>,
) {
    // Plane
    commands.spawn(PbrBundle {
        mesh: meshes.add(Plane3d::default().mesh().size(5.0, 5.0)),
        material: materials.add(Color::rgb(0.3, 0.5, 0.3)),
        transform: Transform::from_xyz(0.0, -1.0, 0.0),
        ..default()
    });

    // Light
    commands.spawn(PointLightBundle {
        point_light: PointLight {
            shadows_enabled: true,
            ..default()
        },
        transform: Transform::from_xyz(4.0, 8.0, 4.0),
        ..default()
    });

    // Camera
    commands
        .spawn(SpatialListenerBundle::default())
        .insert(Camera3dBundle {
            transform: Transform::from_xyz(0.0, 0.0, 4.0),
            ..default()
        });

    // Audio source: Orbiting cube
    let event_description = studio.0.get_event("event:/Music/Radio Station").unwrap();

    commands
        .spawn((
            Cube,
            PbrBundle {
            mesh: meshes.add(Cuboid::default()),
            material: materials.add(Color::rgb(0.8, 0.7, 0.6)),
            transform: Transform::from_scale(Vec3::splat(0.2)),
            ..default()
        })).with_children(|parent| {
            parent.spawn(SpatialAudioBundle::new(event_description));
        });
}

fn play_music(mut audio_sources: Query<&AudioSource>) {
    audio_sources.single_mut().play();
}

fn orbit_audio_source(
    time: Res<Time>,
    mut audio_sources: Query<&mut Transform, With<Cube>>,
) {
    for mut audio_source in audio_sources.iter_mut() {
        audio_source.translation.x = time.elapsed_seconds().sin() * 2.0;
        audio_source.translation.z = time.elapsed_seconds().cos() * 2.0;
    }
}

fn update_listener(
    keyboard: Res<ButtonInput<KeyCode>>,
    time: Res<Time>,
    mut listeners: Query<&mut Transform, With<AudioListener>>,
) {
    let mut transform = listeners.single_mut();

    let speed = 4.;

    if keyboard.pressed(KeyCode::ArrowRight) {
        transform.translation.x += speed * time.delta_seconds();
    }
    if keyboard.pressed(KeyCode::ArrowLeft) {
        transform.translation.x -= speed * time.delta_seconds();
    }
    if keyboard.pressed(KeyCode::ArrowDown) {
        transform.translation.z += speed * time.delta_seconds();
    }
    if keyboard.pressed(KeyCode::ArrowUp) {
        transform.translation.z -= speed * time.delta_seconds();
    }
}

FMOD exports multiple files per bank & localization. We basically just need a system that takes in an array of bank paths and loads the one matching the client's host. Might want to investigate if there are already other localization crates for bevy that we want to integrate with. Would be nice if there was a plugin that already exposes the preferred locale of the user.

See https://www.fmod.com/docs/2.00/api/studio-guide.html#sample-data-loading

Currently we load samples when we call Studio::EventDescription::createInstance, but we probably want to support loading all of the sounds for one level for example.

This way we can avoid linking to a cloud storage link to an FMOD project in the repo

One of the more advanced features of FMOD allows pulling information back out of FMOD at runtime.
I've been trying to figure out how to do so with bevy_fmod recently. While I can register callbacks, capturing bevy context to trigger Events for example has been proving... a bit more challenging.

It seems useful enough to me to consider adding an example for such useage (if possible at all).

Has anyone attempted to do this before with bevy? I'll share some code snippets as examples for what I mean.

basic callback example:

use bevy::prelude::*;
use bevy_fmod::fmod_studio::FmodStudio;
use libfmod::ffi::{FMOD_OK, FMOD_RESULT, FMOD_STUDIO_EVENTINSTANCE, FMOD_STUDIO_EVENT_CALLBACK_TYPE};

// Implement a function with the expected FFI signature that calls the closure
extern "C" fn callback_bridge<F>(
    param1: u32,
    param2: *mut FMOD_STUDIO_EVENTINSTANCE,
    param3: *mut std::ffi::c_void,
) -> i32
where
    F: FnMut(u32, *mut FMOD_STUDIO_EVENTINSTANCE, *mut std::ffi::c_void) -> i32,
{
    unsafe {
        // Get the closure from the context data
        let callback_wrapper: &mut CallbackWrapper<F> = &mut *(param3 as *mut CallbackWrapper<F>);

        // Call the closure with the provided parameters
        (callback_wrapper.callback)(param1, param2, param3)
    }
}

impl<F> CallbackWrapper<F>
where
    F: FnMut(u32, *mut FMOD_STUDIO_EVENTINSTANCE, *mut std::ffi::c_void) -> i32,
{
    // Function to create a new CallbackWrapper
    fn new(callback: F) -> CallbackWrapper<F> {
        CallbackWrapper { callback }
    }

    // Function to get the function pointer to the bridge function
    fn get_callback_bridge(&self) -> libfmod::ffi::FMOD_STUDIO_EVENT_CALLBACK {
        Some(callback_bridge::<F>)
    }
}

pub fn setup_fmod_callbacks_system(
    studio: Res<FmodStudio>
) {
   let mut my_closure = move |param1: u32,
                               param2: *mut FMOD_STUDIO_EVENTINSTANCE,
                               param3: *mut std::ffi::c_void| {
        print!("triggered by FMOD event");
        FMOD_OK
    };

    let callback_wrapper = CallbackWrapper::new(my_closure);
    let callback_bridge = callback_wrapper.get_callback_bridge();

    let _ = studio
        .0
        .get_event("event:/my-event")
        .unwrap()
        .set_callback(callback_bridge, 0xFFFF_FFFFu32);
}

This reliably triggers when the appropriate FMOD event triggers, multiple times due to the catch_all mask FFFF_FFFF.

But as for relaying this information to bevy... I've tried an approach using crossbeam-channel.

FMOD (Studio) API provides a lot of functionality for diagnostics like CPU usage, memory usage, etc.
Maybe we can start thinking about how we can nicely integrate this and use this issue as a tracker.

Given that this project is now on crates.io, we should put a bigger focus on documenting everything.

• Test on smartphone
• Test on Quest 3

Just realized that the spatial example is actually showing two things: Spatial audio and audio control.
Let's split these and also make sure that the spatial example starts playing audio immediately.

FMOD may also be configured to use a right-handed coordinate system by passing FMOD_INIT_3D_RIGHTHANDED to System::init. This will make it so that we don't have to do any of our own conversions.

Everyone with FMOD Studio installed should have the FMOD demo project installed. No need for Google Drive. Also that demo project includes more diverse examples, useful for our examples.

Investigate and evaluate support for web / WASM. I personally have not much experience with WASM, but maybe simply following the FMOD and bevy docs will already be sufficient.

If someone has some WASM experience, especially with external, non-rust libraries, feel free to pick this up.

Fetching the event every time it's played seems inefficient. We should fetch the event references on startup (maybe even some fancy compile-time validation) and only pass references to the play_incoming_events system.

https://rust-lang.github.io/api-guidelines/

https://github.com/bevyengine/bevy_github_ci_template

This apparently handles bundling for different platforms. Maybe it fixes our dynamic linking issue.

Test it out and mention it in our documentation if it's a viable alternative to frankensteining your own build.rs.