scratchfoundation / scratch-audio Goto Github PK
View Code? Open in Web Editor NEWWeb Audio-based audio engine for Scratch 3.0
License: BSD 3-Clause "New" or "Revised" License
Web Audio-based audio engine for Scratch 3.0
License: BSD 3-Clause "New" or "Revised" License
Clicking the stop sign twice with a sound playing, should stop the sound with no errors.
The first click stops the sound, the second click throws an exception and stuff stops working
Add a click-flag-hat and a play-sound-until-done under it. Click the green flag, click the stop button twice.
macOS and Safari 11
This can be fixed by changing the check in https://github.com/LLK/scratch-audio/blob/develop/src/SoundPlayer.js#L51 to if (this.bufferSource && this.isPlaying)
.
If possible I guess it should check if the buffer source is playing, but I'm not sure that's possible.
We'll need a set of 18 drum sounds to match the scratch 2.0 play drum block. We can start with the existing sounds:
https://github.com/LLK/scratch-flash/tree/master/src/soundbank/drums
And optionally work on replacing them with higher quality sounds.
Pressing mode button in https://llk.github.io/scratch-gui/#141331752/ should play a brief sound and switch mode.
Open https://llk.github.io/scratch-gui/#141331752/ , press green flag, press mode (may have to press for a while to trigger toggle mode).
Ubuntu 16.04, chrome 56.0.2924.87
We should create a small set of scratch projects that we can use to test audio performance.
This library has a problem of browser compatibility. audio.createStereoPanner()
is not surpported by safari.
In addition to the built-in instruments (such as piano, flute, etc), the "set instrument" block could allow you to choose one of the sounds in the sprite. Then, the "play note" block would play this sound at different pitches by changing its playback rate. "Play note 60" (middle C) would play the sound unchanged, "play note 61" would shift it up a semitone, etc. The duration set by the "play note" block would stop the sound if necessary, with a short fade out to prevent clicks. So, for example, you could record your dog barking, and make it bark a melody, like happy birthday.
Early experiments with this idea are promising. Seems like fun.
One concern is that the pitch of the sounds produced in this way will depend on the pitch of the recorded sound, which could be confusing, especially if you are trying to use multiple sounds together, or in combination with the built in instruments.
Design questions:
If you use several loudness blocks, they should not cause the loudness analysis code to run multiple times in a single scratch animation frame.
Each time a loudness reporter is called it runs the loudness analysis.
This program uses a run-without-screen-refresh custom block to check the loudness 100,000 times, and measure the time it takes:
On Scratch 2.0 on my machine, this reports 0.08 seconds, and on Scratch 3.0 it reports about 0.45 seconds.
One strategy for improving this would be store a timestamp each time the loudness is requested, and also store the loudness value itself. Then, each time, if the current time is long enough after the timestamp (e.g. one frame at 30fps- since we are not measuring the platform's audio buffer duration), then make a new measurement- otherwise just return the stored value. For the program above, this should result in the loudness analysis being done once each time the custom block is run, instead of 100,000 times.
Mac OS 10.12.6 Chrome 63
I should be able to set an audio effect on a sprite, then clone the sprite, and the clone should have the same audio effect setting.
Clones don't inherit audio effect values from their parent sprite.
Create this code and press the green flag:
Clicking on the original sprite in the center will the meow with a high pitch, but clicking on the clone will play the sound without the pitch effect applied.
Mac OS 10.11.6 Chrome 59.0
Instantiating a new AudioEngine should not create a new audio context, because it causes problems like this scratchfoundation/scratch-gui#556
You can something similar to the shared audio context technique that the GUI uses.
Each time you play a sound, a webaudio buffersource node is created. After the sound ends, the node should be garbage collected.
I'm not totally sure but I think that they can not be garbage collected. We store a reference to each buffer source node in order to set the playback rate while the sound plays (for the pitch effect), so at the very least these references need to be deleted after the sound ends.
Note that we cannot debug this using the Chrome Web Audio Inspector, because its own operation prevents garbage collection of nodes (see here)
Mac OS 10.11.6 Chrome 59.0
from @SillyInventor:
Pitch and pan effects should change the frequency and speaker side intensity of instruments respectively.
No noticeable change.
Play noise or note, change pitch, replay noise or note.
Operating System and Browser
Ubuntu 16.0.4, Chrome 56.0.2924.87
Reported during Extensions Internal Playtest (6/16/17).
The Speech extension should have working speech synthesis and recognition.
@thisandagain's speech recognition never worked:
Speech synthesis worked, but speech recognition never worked
I am not sure.
We were all looking at https://ericrosenbaum.github.io/scratch-gui/ via Chrome, Mac. This behaviour was seen on @thisandagain's computer.
When you change the pan effect, and then play a sound, the sound should start playing at the new pan position.
The pan position sounds like it is sliding from the old to the new position at the beginning of the sound, even if the pan value was set before the sound started.
Open https://llk.github.io/scratch-gui/
Put on headphones, so you can hear the pan effect clearly.
Drag out three separate blocks:
Play sound 1 (should play the meow sound)
Set effect pan left/right 100
Set effect pan left/right -100
Click on pan 100, then play sound, then pan -100, then play sound.
You should hear the sound play in your right ear, then in your left ear, and so on.
Instead, after the pan value has been changed, at the start of the sound you hear it slide quickly from its previous pan position to the new position.
Mac OS 10.11.6 Chrome 56.0.2924.87
A playground which would allow you to test scratch-audio
.
So you can test scratch-audio
without relying on other modules to be working.
The green flag and stop button both clear the audio effects to 0. If you do these things, for example:
click [set pitch effect to 100]
press green flag
click [change pitch effect by 10]
click [play sound meow]
The meow sound should play with pitch effect set to 10.
The meow sound plays with pitch effect set to 110. Apparently, the green flag and stop buttons are not completely clearing the sound effect values.
Not that the problem appears only after using "change effect." If you press the green flag and then play the sound, it correctly has the pitch effect set to 0. So the problem is with the stored value used by "change effect" to update it.
See above.
Mac OS 10.11.6 Chrome 57
Currently we cannot load wav files from existing Scratch projects when they have the adpcm format. Looks like the relevant decoder code from Scratch 2.0 is here
We will probably need to re-design the audio engine. This is mainly to separate the life cycle of a SoundPlayer from that of a clone. In 2.0, a sound started by a clone continues after the clone has been deleted (see scratchfoundation/scratch-vm#949), so the re-design is required to support that behavior.
@thisandagain and I discussed a possible new design for the audio engine, which I wanted to document here, because this work is likely to be deferred for another few months.
TL;DR: AudioEngine manages a set of persistent SoundPlayers, one for each sound
Requirements
A target (sprite or clone) needs to be able to:
The requirements above are currently being met, but this one is not:
Current Audio Engine Design
Here's the current (April 2018) setup of the audio engine:
A sprite and its clone share a single activeSoundPlayers list, which is used to
Proposed Re-design
The idea is to reduce the system down to just an AudioEngine that manages a set of SoundPlayers. Each sound created in Scratch is associated with a single persistent SoundPlayer object.
The Audio Engine creates and deletes these SoundPlayers to stay in sync with the VM.
Each SoundPlayer would know if it is currently playing, and when played, restart itself if it is already playing.
The Audio Engine can update a SoundPlayer's parameters (volume, pitch, pan) when the sound starts or while it plays.
When a sound is started by a target, the VM would use the parameters for that target to update the SoundPlayer.
Questions
When the VM updates sound parameters for a target (e.g. using a "set volume" block), the AudioEngine will need a way to update those parameters on all the SoundPlayers currently being played by that target. How should this work? Two ideas:
The other big question is how this will affect other users of the AudioEngine, such as the sound library. Hopefully the changes to keep things working would be small.
We plan to move the instrument and drum blocks to an official scratch extension. Specifically, it will include (at a minimum) these blocks:
play drum
rest
play note
set instrument
change tempo
set tempo
tempo
Existing projects that use these blocks will load and function as before (they will load the extension automatically).
When you click a "play sound until done" block, it should highlight during the duration of the sound.
The block does have the correct timing behavior, waiting until the sound completes to continue to execution of the next block in the stack (so, for example, this stack highlights correctly: repeat 10 [play sound and wait]).
No highlight appears.
Drag out a "play sound until done" block and click on it
Mac OS 10.11.6 Chrome 56.0.2924.87
The "set effect fuzz to 100" block should cause sounds to play with a nice fuzzy distortion.
The "set effect robot to 100" block should cause sounds to play with a buzzy robotic quality.
These blocks have no effect.
Go to https://llk.github.io/scratch-gui/ and create these stacks:
clear audio effects
set effect fuzz to 100
play sound 1
clear audio effects
set effect robot to 100
play sound 1
they should play the meow sound with fuzz and robot effect applied, respectively, but they don't.
Mac OS 10.11.6 Chome 56.0.2924.87
The drum sounds should be loaded like other static assets, using scratch-storage.
I got them working in a hacky way by loading them from the github repo itself. See here.
The play sound block has a menu that should list the names of the sounds in the sprite.
The menu contains a list of numbers 1 to 10.
Click on the menu on the play sound block.
Mac OS 10.11.6 Chrome 56.0.2924.87
When the loudness block is called and the microphone is connected, a value is returned.
Nothing is returned, and an error is thrown: TypeError: this.mic.mediaStream is undefined
It looks like the error is coming from here:
It looks like this error is spawning from this line:
// If the microphone is set up and active, measure the loudness
if (this.mic && this.mic.mediaStream.active) {
The microphone itself is set shortly before:
navigator.mediaDevices.getUserMedia({audio: true}).then(stream => {
this.mic = this.audioContext.createMediaStreamSource(stream);
This is definitely weird, since mediaStream
is defined in the WebAudio spec. Apparently a bug in Firefox. (I tested a comparable demo which uses createMediaStreamSource
to take a look - yep, the mediaStream
property is actually missing.)
So, browser specific. But worth looking into. I think you could just store the media stream in a separate property from this.mic
:
+ this.audioStream = stream;
this.mic = this.audioContext.createMediaStreamSource(stream);
~ if (this.mic && this.audioStream.active) {}
Firefox 58, 59. Debian (Testing).
Just started getting this deprecation warning
[Deprecation] GainNode.gain.value setter smoothing is deprecated and will be removed in M64, around January 2018. Please use setTargetAtTime() instead if smoothing is needed. See https://www.chromestatus.com/features/5287995770929152 for more details.
@ericrosenbaum can you determine if we want the instantaneous behavior (in which case I guess we keep the setter the way it is) or if we want a smoothed behavior (in which case we should switch to using setTargetAtTime)?
I'm adding this to the December milestone just to stay ahead of it, but I do not think (judging from the warning) that it would cause any buggy behavior if we let it be. But I'd like to get a decision on the books this month just so we know where we stand.
We are missing our required TRADEMARK
file as per the boilerplate:
https://github.com/LLK/scratch-boilerplate
If the microphone input for the loudness block is connected and then disconnected, it should automatically attempt to reconnect.
The mic input stays disconnected.
Mac OS 10.12.6 Chrome 61
Some menus on blocks allow a reporter to be dropped onto them, to select an item for example with a variable or a "pick random" block. We could allow an effect to be selected by numeric index.
To match the behavior of Scratch 2.0, the green flag and stop button should not reset the volume for a target.
Green flag and stop button currently do reset the volume.
The issue is described with repro here: scratchfoundation/scratch-vm#927
It should be as simple as removing this line:
https://github.com/LLK/scratch-audio/blob/84fdf5c76a41a4960e34282ea96687d9ef19d6aa/src/index.js#L125
Generally, when you run the "play sound" block multiple times with the same sound selected, it stops and restarts the sound. For example this stack plays a stuttering sound, restarting the sound many times per second, rather than playing many overlapping copies of the sound:
In Scratch 2.0, clones of a sprite share their sounds as if they were all being played by the parent sprite, so that if one clone starts a sound, and another clone then starts it, the sound restarts. So this stack also creates a stuttering sound:
In the current implementation for 3.0, each clone gets its own audioPlayer, so many clones can all play the same sound without restarting each other. But this probably creates performance and memory issues (I think this is the cause of #27, for example).
So probably we should change it so that clones don't have a full audioPlayer. Instead, each sprite has an audioPlayer, and clones would only get a reference to it.
If we do that, how do we handle the audio effects?
Generally, how would we implement this?
Currently we have a temporary placeholder InstrumentPlayer, which plays notes using an existing soundfont library, soundfont-player. These soundfonts are high quality, but far too large in file size for our purposes, and for the moment they load on demand (i.e. at the moment the 'play note' block or 'set instrument' block runs, if the soundfont has not yet loaded, it loads, causing a delay of a few seconds). Other limitations of this approach include the fact that we cannot route these sounds through all of the audio effects (currently, they are routed through echo, reverb, fuzz and robot, but not pitch or pan), we cannot set the volume, and we cannot sustain the sound longer than the sample duration. All this will change once we decide on a new approach that gives us high quality sounds with a reasonable file size.
There's an old Scratch 1.4 block – I'll call it "MIDI instrument" since that's its op code (midiInstrument:
). It's Scratch 1.4's version of the "set instrument" block.
In Scratch 2.0, the block looks like this:
The only visible difference is that one block has the obsolete colour and the other has the Sounds category colour, but they are stored differently in the project files and they do behave differently.
Scratch 2.0 has very few instruments in comparison to 1.4 – only 21 vs. 128! (No factorial.) So, to account for the limited sound options, Scratch 2.0 uses a mapping of 1.4 instrument IDs to 2.0 instrument IDs. (That's defined here.)
However, I think it's not very debatable to say that this isn't the right thing to do. Any old 1.4 projects that used instrument blocks are basically broken by this, and any future (well, current) musical projects are relatively more limited now.
The simple way to handle this would be to use the very same map along with a new sound primitive definition in the VM, but that's only a semi-fix. On the one hand, that'd make those 1.4 projects behave the same way they have for the last three-ish years. On the other hand, the projects still "don't work" – they still don't behave in the same way as their old user-created versions.
Arguably, this isn't even a problem – Scratch 2.0 has already been the current version for three and a half years (even longer when 3.0 is released!). Chances are very few people, if any, still care about the old versions of the projects, or the old instrument options. (I was hardly even around then – I certainly didn't care about those projects!)
But the real best solution is likely to come from the official reason these sounds aren't available anymore. Here's some text from an old post by Lightnin:
Unfortunately, for various reasons, Scratch 2.0 can't use those [MIDI] synthesizers, so we have to build our own synthesizer and musical instrument library.
Here's the limitations he's talking about:
We can only use SoundFonts that are public domain or available under a license that allows us to distribute them freely, such as a Creative Commons license.
We're limited in size. The entire library should be no more than two megabytes. Assuming we include 25 instruments, each instrument should be about 80k in size.
So one problem is file size. At this point I think it's alright to sacrifice a bit more data – remember, connections are always improving and browsers do cache files well!
So the real issue here is licensing, and a lack of the sounds.
I'd be interested in seeing how collecting those sounds might go. At this point, the Scratch community is larger than ever, so gathering the sounds might be easier than it was back in 2013 (there's now 10 times more active users!). I guess one of the problems might be consistency of quality, though – obviously sounds from different people are likely to have subtle recording/accuracy differences. The real best bet would be to have one person, or an organized group of people, record every sample, but 128 instruments (of which quite a few may be more difficult to get! There's quite a lot of synthesizers and such..) is quite a daunting number..
I wonder if there's some MIDI libraries already created online that have licenses that would be compatible with Scratch? That'd probably be the best way to get 128 accurate MIDI instrument samples.. :)
TL;DR revisiting the old MIDI sounds might be nice.
But I'm not the professional audio person here, nor a lawyer, so.. I'm really just bringing this up for discussion!
(edit: I'm assuming scratch-audio is the best place to put this!)
(edit2: This also applies to old drum sounds, I think.)
Please describe what should happen
The stack should un-highlight after running
Describe what actually happens
The stack stays highlighted after the sound has completed
Explain what someone needs to do in order to see what's described in Actual behavior above
At https://llk.github.io/scratch-gui/ drag out a "move 10 steps" and attach "play sound meow until done" to it.
Run the stack
The cat moves 10 steps and "meow" plays, however the stack remains highlighted after playing
Chrome 57, macOS 10.12
I think we can use a vocoder to create a robotic-sounding effect, perhaps by modifying the implementation here.
The "play sound until done" block should resolve and de-highlight at the time the sound completes. Currently it predicts the end time of the sound using the duration and the playback rate at the time the sound is started. If you use a "set pitch effect" block to change the playback rate after starting the sound, this timing is incorrect.
I'm hoping we will be able to use the onended event of the AudioBufferSourceNode, though this is not directly exposed in the way that we are using ToneJs.
In readme.md:
npm install https://github.com/LLK/scratch-audioengine.git
Now it should be:
npm install https://github.com/LLK/scratch-audio.git
Work from develop
as per our Git Flow workflow in other 3.0 repositories.
Ideally, all the audio effects would be applied at the level of each sprite or clone. In audio terms, each sprite or clone would have its own audio effects chain that includes all of the active audio effects, with its own settings for its own sounds.
Unfortunately, some of the audio effects are processor intensive: echo, reverb, fuzz, and robot. Instantiating them all for each of a potentially large number of sprites or clones would quickly cause audio dropouts or other glitches.
Right now, as a workaround, the AudioEngine creates a single instance of each of these four effects, through which all sounds are played. Any sprite or clone can set, change, or clear them, but the results affect all sounds.
(The pitch effect and pan effect are less processor intensive and so are instantiated by the AudioPlayer for each sprite or clone).
Some questions:
Are there ways to re-implement the echo, reverb, fuzz and robot effects to make them less processor intensive?
Should we dynamically create and destroy audio effects nodes when they are enabled and disabled by blocks?
Should we dynamically create and destroy audio effects when a sprite or clone starts or stops playing sounds?
Should we try a hybrid strategy that simulates having individual effects chains for sprites, using a single one? In that case, the single global effect would be set to the value requested by the sprite that most recently played a sound. I think this would require a data structure to track sounds as they start and end. For example, sprite 1 sets the reverb to 10, and starts a sound, causing global reverb to be set to 10; sprite 2 sets reverb to 50 and starts a sound, causing the global reverb to be set to 50; sprite 2's sound finishes but sprite 1's sound is still playing, so the global reverb is set to 10.
The sound blocks should play sounds on an iPhone.
No sound plays.
Visit https://llk.github.io/scratch-gui/ on an iPhone and click on a play sound, play note, or play drum block.
iPhone 6, iOS 10.2, Chrome for iOS
Currently, we have a prototype implementation that uses the drum sounds from Scratch 2.0. We should consider creating a set of new, higher quality drum sounds.
It should be possible to stop a sound while it's playing without causing a click.
We're getting click sounds in several situations when sounds are stopped. This includes stopping sounds using the stop button, using the "stop all sounds" block, stopping and restarting a sound by playing it in a loop, and while navigating the sound library by hovering over sounds.
Click the sounds tab
Click "Add Sound"
Add the sound "C Bass"
Run the following stack:
Fortunately, in all these clicky situations, the same stop() function is called, here. We should be able to add a gain node to each bufferSourceNode and schedule a very short ramp down before stopping it.
Mac OS 10.11.6 Chrome 58.0
Loudness works consistently on all supported platforms
From feedback form:
On an iPad microphone seems to either detect noise in the loudness block or it doesn't.
iPad
We might want to have a global compressor at the end of the effects chain, where the sound from all the sprites gets combined. This would help prevent clipping by reducing the volume if the sounds get too loud.
Another idea for making the input to the "play note" more intuitive and musical: support both numeric input of MIDI note numbers (as in Scratch 2.0) and also note name/octave notation like "C4" or "G#3" or "Eb2" (similar to "scientific pitch notation). We could also automatically convert valid note names without an octave to the default octave. So these would all play the same note:
Play note 60
Play note C4
Play note C
See scratchfoundation/scratch-gui#257
Even if we use scratch-storage for the actual loading, there will still need to be something to indicate that the sounds are decoded and ready for playing.
Mac OS 10.12.2 Chrome Version 56.0.2924.87 (64-bit)
We are using the Tone.js library for several things, but we don't need most of it. Eventually we should transition away from it, which will require recreating some parts of it, including:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.