interactiveaudiolab / wut Goto Github PK

What it says on the tin.

This should just grab models from nussl, but this issue only refers to the WUT infrastructure part of this problem.

Avoid repeatedly running deep clusterer by pickling state on back-end

Apply algorithms to uploaded audio then give back results along with algorithm picker recommendations

Encapsulate each track and it's control in a class that dictates interaction within the track

Like zoom, and selection methods. Such that it's possible to yolk the waveform into zoom, and add in multiple selection

The latest version of wut doesn't let you download anything. It would be good to have a download button next to each of the "blue selection" and the "orange selection" stop buttons.

Avoid hardcoding size. Stretch goal here is clean, responsive resizing of tracks.

When paused/stopped, if you drag the playhead to the middle of the track (say 10" into a 30" song) the audio will start from that position, but the playhead will bounce back to the 0" tick and crawl from there (instead of the selected position).

Right now the Mixture and Result waveforms are set up by hand, but it would be great to have an object that would be instantiatable to use.

No mobile support. Figure out which browsers are compatible. Throw up warning if we can't serve that browser.

Send the browser the spectrogram data made from an STFT that has much lower resolution (i.e., bigger hop/window size).

Fix redis bug that leads to simultaneous sessions sharing data

Explore doing the STFT in javascript. Is possible to get the same results as the nussl version? Is this more efficient? Will we need a web worker?

Selecting the same region twice is a problem. It causes a much larger region, never selected, to appear selected.

When soloing and muting tracks, it is easy to get into a bad state (i.e., mute button is unselected but the track IS muted, etc.)

In the spectrogram display, when the user makes a selection, we change the actual values of the heatmap (-50 or so) to change the colors. Is there a better way to do this that doesn't distort the heatmap scale?

Add user confirmation of loss of old data (suggest saving data before continuing)

nginx?

In plotly_plot.js look at this.DOMObject.on('plotly_selected', ...) and updatePlotWithSelection() for help.

If you load an audio file, then hit "begin" again and try and load that same audio file a 2nd time, you don't get the file play interface in the dialog box that 2nd time. "OnChange" event doesn't fire because it is the same file. Figure out an alternative?

Alternately: the reason I did this was to change models for separation, as the only way to change models is to start over with a new file upload. If we separate those two steps, then maybe this problem becomes a non-issue.

Give user option to select which algorithms they'd like loaded

There is some non-deterministic behavior when adding dots onto the waveform UI in waves-ui.js: sometimes some dots disappear when adding many dots. Not sure what's up here...

Right now, all elements span the entire width of the screen, the top toolbar is half built, status messages are sporadically displayed. Once the functionality is in place, clean the entire product up to make it feel cohesive.

Right now, the gains for each track in the results page get set once when the page is generated, but the gains should be set when the user selects "Play".

So things like scaling the audio for each track can react to when a track is muted.

Offer T-SNE as dimensional reduction alternative to PCA

• Unique ID for each track (see #24)
• Add new tracks at top

Can we make it more efficient?

example of Librosa's license - https://github.com/librosa/librosa/blob/master/LICENSE.md
choose a license - https://choosealicense.com/

2018-11-29 17:06:31] [ INFO] --- Max freq = 10206.73828125 Hz (general_audio.py:85)
127.0.0.1 - - [2018-11-29 17:06:32] "GET /socket.io/?EIO=3&transport=polling&t=MTXkH3r&sid=dc35f25d806f4835ac02e826e9648a56 HTTP/1.1" 200 223 14.658554
Matching file found at /tmp/.nussl/models/deep_clustering_vocals_44k_long.model, skipping download.
message handler error
Traceback (most recent call last):
File "/anaconda3/lib/python3.7/site-packages/engineio/server.py", line 423, in _trigger_event
return self.handlersevent
File "/anaconda3/lib/python3.7/site-packages/socketio/server.py", line 512, in _handle_eio_message
self._handle_event(sid, pkt.namespace, pkt.id, pkt.data)
File "/anaconda3/lib/python3.7/site-packages/socketio/server.py", line 458, in _handle_event
self._handle_event_internal(self, sid, data, namespace, id)
File "/anaconda3/lib/python3.7/site-packages/socketio/server.py", line 461, in _handle_event_internal
r = server._trigger_event(data[0], namespace, sid, *data[1:])
File "/anaconda3/lib/python3.7/site-packages/socketio/server.py", line 490, in _trigger_event
return self.handlers[namespace]event
File "/anaconda3/lib/python3.7/site-packages/flask_socketio/init.py", line 251, in _handler
*args)
File "/anaconda3/lib/python3.7/site-packages/flask_socketio/init.py", line 634, in _handle_event
ret = handler(*args)
File "/Users/bryan/Documents/GitHub/WUT/app/app/views.py", line 115, in initialize
dc = audio_processing.DeepClusteringWUT(separation_sess.user_signal, separation_sess.user_original_file_folder, model_path)
File "/Users/bryan/Documents/GitHub/WUT/app/app/audio_processing/deep_clustering.py", line 41, in init
use_librosa_stft=True)
TypeError: init() takes 1 positional argument but 2 were given
[2018-11-29 17:06:32] [ ERROR] --- message handler error (server.py:425)
Traceback (most recent call last):
File "/anaconda3/lib/python3.7/site-packages/engineio/server.py", line 423, in _trigger_event
return self.handlersevent
File "/anaconda3/lib/python3.7/site-packages/socketio/server.py", line 512, in _handle_eio_message
self._handle_event(sid, pkt.namespace, pkt.id, pkt.data)
File "/anaconda3/lib/python3.7/site-packages/socketio/server.py", line 458, in _handle_event
self._handle_event_internal(self, sid, data, namespace, id)
File "/anaconda3/lib/python3.7/site-packages/socketio/server.py", line 461, in _handle_event_internal
r = server._trigger_event(data[0], namespace, sid, *data[1:])
File "/anaconda3/lib/python3.7/site-packages/socketio/server.py", line 490, in _trigger_event
return self.handlers[namespace]event
File "/anaconda3/lib/python3.7/site-packages/flask_socketio/init.py", line 251, in _handler
*args)
File "/anaconda3/lib/python3.7/site-packages/flask_socketio/init.py", line 634, in _handle_event
ret = handler(*args)
File "/Users/bryan/Documents/GitHub/WUT/app/app/views.py", line 115, in initialize
dc = audio_processing.DeepClusteringWUT(separation_sess.user_signal, separation_sess.user_original_file_folder, model_path)
File "/Users/bryan/Documents/GitHub/WUT/app/app/audio_processing/deep_clustering.py", line 41, in init
use_librosa_stft=True)
TypeError: init() takes 1 positional argument but 2 were given
127.0.0.1 - - [2018-11-29 17:06:32] "POST /socket.io/?EIO=3&transport=polling&t=MTXkKRM&sid=dc35f25d806f4835ac02e826e9648a56 HTTP/1.1" 200 195 1.238359
^CKeyboardInterrupt
2018-11-29T23:06:38Z
KeyboardInterrupt
2018-11-29T23:06:38Z

We will need to add zoom functionality for:

• Mixture spectrogram
• Results tracks
• Mel spectrogram

We could consider adding zoom functionality for:

• Mixture waveform at the top
• Deep clustering PCA

• When you open up the dialog box, the existing selection of eigenvectors should be displayed.
• When you close the dialog box, the number of the eigenvector selected should display in the 2-d display
• you say 'percent' explained, but you display a number between 0 and 1.
• when you change the set of principal components displayed, the old selection box that was on the previous set of components remains.
• you should really buy Doctor Professor Pardo a jet ski. In purple.

Allow users to selectively remove tracks from results page