group-24 / palpitate Goto Github PK

Hi, I'm trying to run Palpitate in Docker on Mac however after uploading the video and waiting for some time I'm getting this traceback

Traceback (most recent call last):
File "heart_rate_server.py", line 9, in
from server.heart_rate_imposer import HeartRateImposer
ImportError: No module named server.heart_rate_imposer

I've tried to use your Bill Clinton example and that also gives the same error.

@eadward I've now done the CNN-RNN approach discussed in the meeting.

It is behaving very similarly to the subject independent RNN aproach, meaning it is not learning anything.

For example here is a training trace:

Train on 10721 samples, validate on 10410 samples
Epoch 1/15
10721/10721 [==============================] - 56s - loss: 0.9948 - val_loss: 1.5181
Epoch 2/15
10721/10721 [==============================] - 56s - loss: 0.9000 - val_loss: 1.4525
Epoch 3/15
10721/10721 [==============================] - 55s - loss: 0.8459 - val_loss: 1.4802
Epoch 4/15
10721/10721 [==============================] - 55s - loss: 0.7673 - val_loss: 1.4254
Epoch 5/15
10721/10721 [==============================] - 55s - loss: 0.7109 - val_loss: 1.4720
Epoch 6/15
10721/10721 [==============================] - 55s - loss: 0.7307 - val_loss: 1.4763
Epoch 7/15
10721/10721 [==============================] - 55s - loss: 0.6144 - val_loss: 1.4643
Epoch 8/15
10721/10721 [==============================] - 54s - loss: 0.5660 - val_loss: 1.4857

#Final result on validation set: 
Model r:  (0.044959106007683075, 4.4554423862127374e-06)
Model rmse:  1.2189017562850681


#Final result on test set:
Model r:  (-0.0059427638053850106, 0.48934621677575696)
Model rmse:  1.3999194428042123

The loss (root mean square error in this case) keeps decreasing, but the training has no effect on the validation set performance. This is exactly the same thing as it is happening with the direct RNN approach.

The architecture of the network is (tinkered keras syntax for clarity), shape comments represent the shape of the network at that point:

    # shape (n_images, frequencies, time)
    # shape (1,200,158)

    model.add(Convolution2D(16,in_shape[1],5))  #doing 1d convolutions on the time asix with filter size 5
    model.add(MaxPooling2D((1,2)))

    # shape (16,1,77)
    model.add(Convolution2D(32,1,10))
    model.add(MaxPooling2D((1,2)))


    # shape (32,1,34)
    model.add(Reshape(dims=(...)))
    #shape (32,34)
    model.add(Permute((2,1)))
    #shape (time, features)
    #(34,32)
    model.add(LSTM(200)) 

    model.add(Dense(1))
    model.add(Activation('linear'))

For the video we will be looking at colour changes in specific areas of the face (do a bit of literature review on that to see what you can learn from previous attempts).

This program must be able to take a video as an input and return relevant information about the facial color change

Look into how OpenSMILE can help us. You don't need to actually put it in our build system. I'll deal with that at some point.

Using WEKA, we must have a comprehensive machine learning model.

For each configuration of the network, let the model be trained with 10 different parameters (10 iterations of the loop in rnnAproach) for 20 epochs with early stopping of 2.

Each configuration should be tested on both user dependant and user independent data set.

Network Configurations:

• 2 layers stacked LSTM, 1 MLP at output, with dropout and Gaussian noise
• 1 layers LSTM, 1 MLP at output, with dropout and Gaussian noise
• 2 layers stacked LSTM, 1 MLP at output, with dropout and Gaussian noise, with MLP also before LSTMs
• 2 layers stacked LSTM, 1 MLP at output, with dropout and Gaussian noise with MLP between LSTMs
• 2 layers stacked LSTM, 1 MLP at output, with dropout and Gaussian noise with MLP between LSTMs and before LSTMs
• all of the above with l2 regularization
• all of the above with GRUs instead of LSTMs (if this one performs really badly, you don't have to do all of them)

So create a google docs spreadsheet. For each configuration you will have two tables to fill in, one for dependant and one for independent user data. Tables should have the following columns:
(param1, param2, param3, ... , r, rmse)

You should also copy the get_RNN_model function for each configuration, so there is no ambiguity as to what was used and can be easily reproduced. Also record the best parameters and the r and rmse on the test set.

For example the first configuration should look something like:

2 layers stacked LSTM, 1 MLP at output, with dropout and Gaussian noise - [link to get_RNN_model function]

results for dependant data:
(lstm_hidden, mlp_hidden, dropout1, dropout2, r, rmse)
(256,100,0.5,0.5, 0.63, 0.75)
.
.
.
(450,150,0.4,0.7, 0.59, 0.77)

best parameters: (256,100,0.5,0.5)
performance on test set r=0.65, rmse=0.75

results for independant data:
(lstm_hidden, mlp_hidden, dropout1, dropout2, r, rmse)
(256,100,0.5,0.5, 0.03, 1.2)
.
.
.
(450,150,0.4,0.7, 0.03, 1.2)

best parameters: (256,100,0.5,0.5)
performance on test set r=0.10, rmse=1.2

The way you change between dependant data and independent data is to change the NormalizedSubjectSplitSpectrograms class. The current versions gets the independent one. There is some commented code that can get the dependant one. The main idea is to merge train and validation set shuffle them and then split them again to get the dependant data. Same can be done with the test set but there might be no need.

There needs to be a script which returns a dicitionary of <subjectID>_<state> -> [[description, time, bpm]], from the data in the excel spreadsheets in the HeartAV data

Here are some suggestions, feel free to add/remove/change them, as we did not discuss these at the meeting:

• Website can upload video to be processed by the server
• Video is shown on the page.

We must decide on an angle we are pursuing for software. An android app, or a website?

By the end of the sprint, we will need to have a prototype/mock of the decided upon software

We need a program to preprocess all of the data we have.

this will require an agreed upon input/output for both the audio and video extraction.

This program must be able to traverse the database(es) we have and return a directory/other database full of preprocessed data from the audio and video extractor.

Incorporate a machine learning library in our build system. Make a test case that uses it to learn something trivial to illustrate how to work with it. It would also be nice to have examples for different ML tecniques.

The tests need to be run in a docker context to ensure that the dependencies are all correct.

This is also assigned to @kren1 , but github only allows one person assigned to each thing

We need a script that given a list of (subjectID; audio_filepath) paris. Inserts a new batch into the database describing it. Then extract audio features from the audio_file (and adds those features to the database if appropriate). Then for each feature, for all timeslices insert the tuple
(subjectID, featuerID, batchID, featureValue, timeslice) into the database.

My current idea is to make something like this:
http://deepdreamgenerator.com/

Where the first tile is a black background, with the video centred in the middle, then scroll down to post a url for us to analyse

This is script should read the files (either xls or the tab seperated ones) and extract the bpm for subjects in the appropriate slice for given batch.

inputs: batchID, files(either an xls file or a the root of dataset), timeslice.

outputs: SQL statements (or direct insert into the database) that inserts the tuples
(subjectID; bpmFeatureID, batchID, bpm in that timeslice, timeslice)

To check previous approaches using audio check this (and related) papers:

You will note that the voice recording can be sustained vowels, reading sentences and other types of activities. Some work better than others (see papers that cite the MBC corpus), and it is your goal also to determine which ones work better.

This program should be able to take audio as input and return the features extracted from it using a config file (I assume that OpenSMILE already has some form of configuration)