We could setup a testing pipeline using googletest. But, we should keep things simple for "fast implementation" if this testing pipeline will only turn things more complex it shouldn't be implemented. However, in a mediapipe application we've identified a lot of boilerplate code, so automated tests could come in handy.

This issue aims at providing a convention (for coding, file structure...) for everyone to follow. The follow rules are just a suggestion, we should further discuss about them.

Coding Style

General rules:

• Indentation: 4 spaces;
• File/Folder names: snake_case (MediaPipe already adopts this pattern, so we must follow 😢);

C++

Follow the Google C++ Style Guide.

Description

The face mesh calculator must be explored to latter implementation and new features extraction.

Relevant sources:

• MediaPipe Face Mesh;
• readminds-features-study/mediapipe/graphs/face_mesh;

Deadline

• 27/11/2020

Deadline: 08/01/2020.

The report should discuss the following topics:

• 1. Thesis overview, explaining our part of the problem;
  • 1.1 Focus on the part about emotions;
• 2. Explain the process of reimplementing the facial features;

Other relevant topics should be argued on this issue.

Note: We may start this report by December 26th.

We must now implement the work by Swedish team, to test our features against original implementation.

This is part of #30.

We should provide documentation on how to run feature extractors and models. Also, we should adapt the code, if needed, to be easy-to-use with external scripts (readmind's scripts).

The calculator must be able to read from devices and from video files. (Names can be changed, these are just suggestions).

Name: VideoCaptureCalculator;
Input: device (Input device, or video file path);
Output: RGB ImageFrame;

ICA remote photoplethysmography method - by Poh, M. Z., McDuff, D. J., & Picard, R. W. (2010), available at github.com/phuselab/pyVHR, should be implemented. Later, we'll expand implementation to benchmark all available methods.

This issue aims at providing a convention (for coding, file structure...) for everyone to follow. The follow rules are just a suggestion, we should further discuss about them.

Coding Style

General rules

• Indentation: 4 spaces;
• File/Folder names: snake_case (MediaPipe already adopts this pattern, so we must follow 😢);
• Line length (max): 79 characters;
• Quotes: Double for string and single for char;
• File sufixes: MediaPipe defines some file suffixes given the type file, e.g. if the file implements a calculator it's name must ends with _calculator;

C++

Follow the Google C++ Style Guide.

For documentation, use only single line comments: //. Check existing code for examples on how to document files, classes and functions.

Python

Code must follow PEP 8, and documentation must follow PEP 257.

Every .py that don't follow theses PEPs won't be accepted in pull request. For checking this standards, we'll use pycodestyle.

Exceptions

For docstring documentation, PEP 257 may not be strictly used. Every docstring must specify the behavior of functions, classes, etc such that others can "easily understand".

As for PEP 8, some exceptions are acceptable:

• (List exceptions here, when we detect them);

Styles defined in code reviews

• Blank line between code and documentation ─ PR #35;

Bash

Follow Google Bash Style Guide. Everything defined there is ok to be used.

When running the application using the current mediapipe version some errors popped up. Updating the mediapipe version might solve them. If any API call being used have changed we may need to refactor some code 💀.

This issue concerns three topics:

1. General Code/Structure Refactor

We should perform a refactor under whole repository, following strictly the conventions defined in issue #2. Also, we should set up an easy way for updating mediapipe to latest version (which is basically a clone from current MediaPipe's master).

2. Conventions Overview

We've stated that Google C++ Style Guide would be used, but we haven't followed it strictly. Thus, after the refactor, we should be more severe on code reviews. Also we must define an convention for documentation (we may discuss this in issue #2), providing at least a minimal API.

3. Report I

Now, we should be starting Report I as defined in issue #16. But this should be done after repository refactor.

#10 is required;

Deadline: 02/12/2020.

In order to implement these features we need to know in advance what are the landmark points related (and relevant) to them. Thus, using the images (within data/faces/) produced in issue #10, we must first identify such points. The division (generated by a random algorithm) of who gets which feature mapping is the following:

• @arufonsekun: F3 and F7;
• @jeanchilger: F5 and F6;
• @henriqueherrera: F2 and F4;

The feature F1 was left aside, so the first to finish their mapping gets it.

The objective of this stage is to obtain all the landmark points related to each feature.

All the mappings should be reported on the document shared. Later, it will be better developed into a formal report.

I've detected a possible bug with F3 feature. When running feature_extractor_video, the value of the feature keeps increasing every time (press Enter to get feature values). Not sure if this is a bug with feature_extractor_video or with F3, we should check this before running experiments.

Milestone 2 should be finished by February 1st but it couldn't be accomplished. In March 1st we started another step in the project, but Milestone 2 is still not ready.

This issue aims at reporting this delay. We are late for a month and MUST work this in the further stages.

We should improve repository structure following tips from this repository about releasing research code.

Also we should improve our documentation.

With this same refactor, we should provide scripts for updating mediapipe.

The calculator must be able to convert an image to gray scale format. (Names can be changed, these are just suggestions).

Name: GrayScaleConverterCalculator;
Input: image (RGB ImageFrame);
Output: image (RGB ImageFrame converted to gray scale);

The calculator must be able to write image frames into a video. (Names can be changed, these are just suggestions).

Name: VideoWrittingCalculator;
Input: frame (RGB ImageFrame);
Output: video (Video gathering the frames);

A graph gathering the calculators defined in #5, #6, #7 and #8 must be implemented. All contributors may implement this functionality on separate branches. This graph won't be practically used.

The calculator must be able to apply the gaussian blur into an image. (Names can be changed, these are just suggestions).

Name: GaussianBlurCalculator;
Input: image (RGB ImageFrame);
Output: image (RGB ImageFrame);

That could be accomplished by two approaches:

1. Taking source code directly from MediaPipe repository; or
2. Using bazel to get a instance of MediaPipe in a "indenpendent" way (like this example).

deadline: 01/02/2020

depends on #20.

We've implemented most of our features descriptors. Now we must assemble them in order to generate the features files and
corresponding datasets.

We should have a single script accomplishing this task (in bash probably). It should call the c++ script to generate a brand new csv file with the facial features (a.k.a. F1 to F7). After, the python script should be used to generate the HR signals and append them to the existing csv. This final csv file is our feature file.

Important to note: We should extract the existing features in the same timestamp they were extracted in Bevilacqua's original implementation. Thus, we may insert them in the original csv files, and we won't need to implement features described in #24 for now. These features don't seems to be so relevant, in the model implementation from #26 their performance was worse than the others features.

We were saving the "generic model" (trained on all subjects) mistakenly. The result of that is the "subject-level" model wasn't beginning with any weights, therefore the training was starting fresh for each individual.

The fix is pretty simple and is already under implementation.

#14 is required

Deadline*: 05/12/2020.

Based on the exploration of the face mesh functionality (#10), the already existing facial features must be implemented using MediaPipe. Thus, features F1 to F7 will be reimplemented.

*Note: We should try to end this issue at 3rd December, when we'll meet with swedish team.

Deadline: 30/01/2021.

The experiments will be executed using a helper program (preferably a CLI-like). This program must provide following functionalities:

1. It must accept an argument for number of executions that will be computed (e.g. we'll run the algorithms N times and compute the final result as the mean of the N individual results taken);
2. It must save both individual results for each execution performed and the final result;

As we'll inevitably use python for issue #20, the csv generation must have two parts: one in C++, for things related to mediapipe and another for python, for things related to HR and any additional implementation in the language.

Since anything in python is simpler, the c++ csv generation we'll just write to a file, and in python we'll add content to this written csv.

Deadline: 02/02/2021.

Description

Implement features present in the gux-deeplearning readminds dataset however not highlighted in Fernando's paper.

Features

• face_blinks_1s
• face_blinks_5s
• face_motion_error
• face_motion_instability
• face_mouth_edges_change
• face_mouth_histogram_change
• face_reference_scale

Sources

Check this doc for a basic understanding of each feature.

MediaPipe has some papers related to it, regarding machile learning architectures used by the framework, among other things. Here are some (maybe) interesting for our study:

• MediaPipe: A Framework for Building Perception Pipelines - Presents the framework;
• Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs - Used in face mesh;
• Real-time Pupil Tracking from Monocular Video for Digital Puppetry - Used in iris tracking;
• MediaPipe Hands: On-device Real-time Hand Tracking - Used in hand pose/palm detection;