emkor / audiopyle Goto Github PK

Coordinator may store its config in Redis. Example config is backblaze bucket access parameters. Also, it should read config stored in Redis at startup

Implement client for storing config in Redis

The next component is converter. Service should convert a given mp3 file in local filesystem to the mono wav file with 16 bit and 44.1 kHz spec, without splitting file. It should be configurable (dir with input files, dir for output files) and it should take filename as an argument. Also, during conversion, it should indicate that output file is still in writing phase - maybe append _conversion to its filename and remove it after process ends.

add some tests to verify AudioConverter from #7 is actually working

Add option of passing config (at container creation) to coordinator with multiple backblaze buckets access parameters

Some of the features may not be as usable as the other ones. Research which one are time-consuming in analysis and may not be worth computing.

Since we have some complex objects (containing other, nested objects) in our project and we need to serialize them in orders to store object instances in Redis etc, we need to find a way to do it effectively. One way is to implement to_json() in each of model-classes, other way would be to create some service to do it in one place, other one - create method which will map objects to JSONs recursively.

Most of constants currently are under classes (AudiopyleConst, for example). Time to get rid of those classes and define constants on the top level of a file, not inside class, and also fix calls to those constants.

Do some googling and find what system we could use for monitoring performance / metrics of our system. Of course, we are aiming at something with docker support while being free.

Read some docs on Docker if my assumption is correct and COPY is more relevant than ADD, do changes accordingly and check the results with building docker images.

Xtracter should read config on container startup. Config may contain blacklisted vamp plugins, available Redis binds etc

Add some methods for handling files other than just mp3, just as in this comment:
#60 (comment)

Xtracter containers should look at tasks in given Redis queue and start popping them and doing actual analysis

Create basic persister docker image

Coordinator may store it's state in Redis as timestamp indicating what files were already added to task queue

implement client for redis queues

While building on travis, few warnings come up about lack of GLIBC_2.14 and GLIBC_2.15. Although tests are fine, some of the plugins may not work properly. Try to find out how to update GLIBC or find more suitable docker image.

For now we are using only simple print() to log some messages and events, but it is highly unprofessional and we should introduce some proper logging (python has this built-in). So the task is to replace print with proper logging and add more logs where its necessary.

Our docker build looks like this:
-- docker-base-image
-----commons
-----------coordinator
-----------xtracter
-----------persister

So, the coordinator, xtracter and persister are all based on commons image, which is based on docker-base-image. Building commons image is time-consuming, so maybe it would be better to actually just install commons python package while building coordinator, xtracter and persister. It would look like this:
-- docker-base-image
-----commons+coordinator
-----commons+xtracter
-----commons+persister

Scraper is the next app for audiopyle system.
Base idea is to download song information (like album, artist etc., maybe tags?) from internet sources, like MusicBrainz, Last.fm, Spotify etc and store it in specified format. It would be useful for ML and for feature analysis.

Find out which python framework would be OK to use while implementing access do DB (MySQL)

Add ability to select track from DB and draw waveform for it

Since our project is growing quite rapidly, soon we will feel the pain of Python's dynamic types. The fix would be to add type annotations in each public method and function - this way, IntelliJ can infer types of arguments and outputs.

With Visualizer it should be easy to determine how precise should feature extraction be. By default, most of features are extracted from blocks size of 1024 frames (about 1/40 of a second of track), which gives us ~12000 feature values per 5-minute track, and its obvious we do not need so many for most of the features.
Select the right precision for each of the track.

Prepare xtracter - complete service which reads local .wav file and extracts features using vamp and store them in json file in preconfigured directory.

liquibase is a system for managing database schemas it would be nice to integrate it into project since schemas may evolve quickly in early development phases

Change models in Xtracter module so they better reflect actual results from VAMP plugins extraction

Current build/test process is highly inefficient. It goes like this:

• build commons docker image locally basing on pulled base debian 8 image
• build xtracter docker image from DOWNLOADED commons docker image

We could make it more efficient by reusing locally built commons image.

Second thing is: lets assume in one push someone made changes to commons AND xtracter. During build process, commons docker image would be affected by just pushed changes, but xtracter image would be based on commons image downloaded from docker hub - so it would be outdated.

For now, xtracter can analyze only wav files, despite converter is ready. Time to make use of it.

Prepare script to run docker locally with MySQL db instance accessible from outside the docker.

Protip:
docker run -p 3306:3306 --name some-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:latest

Xtracter containers should send analysis results to results queue (Redis) immediately after results are available

We should have some GUI tool for visualizing extracted features, since there will be lot of work with them. The tool may not be perfect, since it would be our internal, but should have some basic features:

• choosing track from db
• show its waveform on timeline (load this as feature from DB and draw with html5)
• allow user to select which feature to show for track and draw them accordingly on top of the waveform
• maybe allow to play the track somehow(?)

Create more logical resources bucket structure (having test resources and vamp plugins)

This is temporary service (will be used only on non-scalable installation) which should select next audio file to analyze, based on list of bucket files and list of already analyzed files in MySQL

Write some integration level tests to for Redis client #15

First concept assumes that once we run any docker container, we map its port with host port, and then pass that port to next docker container to make communication possible. For example, we map Redis Task Queue port to XXXX and Redis Results Queue port to YYYY and MySQL port to ZZZZ, and pass that ports to coordinator (XXXX), xtracters (XXXX to get tasks, YYYY to enable results submitting) and persister (YYYY to read results from cache, and ZZZZ to store results in mysql).

But maybe theres simpler way with usage of docker linking mechanisms?

Create basic docker image for coordinator.

Create proof of concept showing centralized logs on graylog2 docker container.
https://www.graylog.org/

work on connection between DB and the app itself

Create SQL DDL queries that will create tables for AudioFiles, Plugins, PluginOutputs, Analysis and Features. Also,extend script from Persistence #1 with executing the DDL query.

test out some frameworks for Visualizer and choose the one that fits our needs. The framework should allow to connect do DB, load features and draw them for user with HTML5.

Try to integrate code climate into Travis build:
https://github.com/codeclimate/codeclimate#code-climate-cli

Add ability to actually select features from DB for given track and draw them on top of the waveform

Create simple B2 client (b2api wrapper) in commons module; xtracter and coordinator will make use of it

Xtracter should download and cache audio files for analysis from backblaze using given access parameters

Implement simple DAO service which will translate python Feature-objects and store them in database

Add necessary services so that coordinator can read files from hardcoded backblaze bucket.

Persister should read Features stored in Redis as JSONs and convert them into Python objects. Then it should proceed to store them in DB with already implemented DAO.

Coordinator should store info about audio files from backblaze in Redis task queue in chunks (50).

Add logo and basic description of what this project actually do