Giter VIP home page Giter VIP logo

nephos's Introduction

logo

Project Nephos forthebadge made-with-python GPLv3 license

Google Summer Of Code 2018 (Open Source Love png3) Project under CCExtractor; Project Nephos

CodeFactor Requirements Status Build Status codecov

Abstract

Project Nephos aims at simplifying the process of moving samples from local storage to cloud for Universities by automating, almost, all the steps involved. It consists of three independent modules; recording module, processing module, and uploading module.

The recording module is responsible for managing the addition of channel lists, set up of recording jobs and saving the recorded streams. The processing module parses saved samples, associates tags, extracts subtitles and converts the video files to MP4 to reduce the file size. The uploading module uploads the processed stream files, and also shares samples with other universities if required.

Nephos is developed, using Python and few other open source projects. To accomplish all the above mentioned tasks with cent-percent reliability and zero failures (unless wrong data is input, which gets logged), testing and logging is an integral part of Nephos development and running cycle, respectively.

Project Link


Installation (And Setup)

  1. Nephos requires following environment dependencies:
    • Python 3
    • Pip
    • pipenv
  2. Nephos requires following third party libraries:
    • FFmpeg and FFprobe
    • Multicat
    • CCExtractor (requires tesseract-ocr-dev and leptonica-dev)
    1. Nephos uses mail client to send email notifications, please configure it before running Nephos.
    2. You'll be asked to enter the email address(es) of recipient(s) of critical mails at initialisation of Nephos. It is only asked on first launch, to edit it later:
      • Go to Nephos directory, default is $HOME/Nephos
      • Edit the hidden file ".critical_mail_addrs", multiple addresses separated by a single whitespace
      • Restart Nephos after editing the file
      • In case of any email address fails at RegEx match, it'll be listed in info logs, ignored by config handler and you can correct it in the same file

Install Using Git Clone

    1. Clone the repository, git clone https://github.com/thealphadollar/nephos.git && cd nephos
    2. Run the script, sudo ./install.sh
      The script will install all the aforementioned system dependencies and python libraries for Nephos.
      NOTE: Running pip3 install . Will automatically determine your distribution and install the dependencies for you
    1. Observe and modify configurations available in $HOME/Nephos/config (especially maintenance and module configurations, and processing script)
      • Update path to various libraries, in the config file, being used; fatal error might error in case where the path to binaries (soft links work) is not appropriate.
    2. Add nephos_start.sh as a cron job to be executed at startup in root crontab.
      @restart /path/to/nephos_start.sh 2&> ~/Nephos/boot_start.log
    3. Run nephos using sudo ./nephos_start.sh. This command runs Nephos under a new screen session. Press ctrl + a and then 'd' to detach from the session while it keeps running in the background.

To Add Channels And Jobs

Please go to Nephos Config repository to modify the channels, jobs and share lists that Nephos works on.

Running the Web Interface

Nephos has a Web Interface that you can see how everything works. To get started set the enviorment FLASK_APP to the path where webServer.py is and run it with flask run and you can see the database entries in an instant!

Currently the web app offers for you to see which channels are jobs in the database and can be accessed by visiting the url flask provides with the following paths: /channels and /jobs

Developer Documentation

Developers can view the documentation that is present for users since it is detailed and one needs to read it in order to understand how Nephos functions. Along with that, docstrings have been placed in HTML format in docs/DevDocs and can be accessed in a systematic manner by opening docs/DevDocs/nephos.html in a browser.

You can also go here to access it online.

Tests

In order for all code to be successful and well maintanable we run tests to test out the code. To do so run pipenv run py.test

More Info

For more information regarding using Nephos and how it works, visit the wiki

Reporting Bugs

Bugs should be reported in the issue tracker. Security issues must be reported at [email protected] to avoid exploitation.

nephos's People

Contributors

matejmecka avatar t1dus avatar thealphadollar avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

nephos's Issues

Create maintenance directory and functions.

Taking inspiration from the current Nephos, create a class of maintenance jobs and make it so that it is easy to add new maintenance jobs in future.

Possibly...
Separate classes for each type of job, and then run command of all jobs in a single maintenance.run()

Create a derived SMTP Handler

We can improvise on the current mail sending model by making a derived class from SMTPHandler of the logging module.

It can have multiple functions and can be done at the end and embedded into the system seamlessly.

Convert WIKI to PDF

Store Wiki as PDF in the docs directory. Currently, the docs are available as md files.

Allow adding multiple run times for uploader

Currently the uploader is run only once each day; this should be changed to
allow user to add as many times as they want nephos to run.

This can be done by allow entries under time and then parsing them and adding
as many scheduler jobs for uploader as the time.

Check Nephos Initiated

Currently we do not check if nephos has been initiated before the user gives
start command. A check should be established for this.

[DOUBT] Should we allow different roles while sharing?

Drive provides allows different levels of access to emails the data is shared
with. For Nephos, editing access doesn't look good since once the file is
uploaded we barely need to edit it.

Currently, we give all access to read and download the file. Should the user be
allowed to set this while adding share entities?

[DOUBT] Should I store logs in a DB

I don't see a point in storing logs in a database but if this is ever required, it can simply be done by making a class of SQLLogHandler in the logger.py file.

[DOUBT] Should we have `verbose` option

We currently do not display debug logs, but store them in the log file. Maybe we should provide the user with an option to allow viewing these in the terminal itself.

[TODO] Create Error handler for each module

Error handling is an important part and hence each module will have a errors.py which will be called in cases of errors such as not able to authenticate with google or low storage.

They in trun will log the error and even push email if the error is critical.

Use server account for authentication

Currently we depend on a user for authentication; this may be removed by using
a service account and then authentication can be done using email and pass.

Add logs for separate files

We should have separate logs for each recording with the output of CCExtractor,
ffmpeg etc and it should be uploaded to drive along with the other processed
files.

Catch more exceptions and error in user input

Currently, user input can break the program, and hence these errors need to be caught at the moment.

Also, more exception handling is a requirement and should be completed before final submission.

[DOUBT] Should there be an argument to just init nephos

Currently we load nephos files directly and also all the configuration files
are copied only at the runtime. This poses the problem that user can't update
config without running it the first time.

We can change this by separating first_time() from other function.

[DOUBT] Should we pool preprocessing size wise

Currently we do not store the size of the file (can be easily queried using
ffprobe, which we already use) to be processed and hence it has no factor in
how the files go into preprocessing pool. There is no apparent benefit of
taking file size into consideration but it's worth a try to see if there is
improvement in speed.

Remove .aux files

Currently, after recording a channel and processing it, only .ts file is removed. We should remove .aux files as well, preferable post recording since they aren't used anywhere in pipeline.

[TODO] Create separate CLI commands for different functioning

This should be done in order to make separate parts of code independent. For eg., we can have a command to edit basic config, and another command to add a job.

A main command will init the main application and keep it running while the other commands will interact with database and other side stuff.

Emails failing behind proxy!

Currently, nephos fails to send critical emails behind a proxy.

Currently This can be seen on the redhen server. This needs to be corrected to work

[BUG] Tags not set properly

The command for setting tag to "processing" and, in exceptions, to reverting it
back to "processed" is not working.

[TODO] Remove independent recorder configuration loading

Currently, the recording module loads it's configuration independent of the
Config handler. This is not an issue but to main uniformity, refactoring should
be done and a class should be made to incorporate loading by Confighandler.

[BUG] Folders not being removed on preprocessing failure

On the failure of a task, task is removed but the folder created for it's storage is not removed. Best would be to make sure that folder is created when the processing script is invoked and it is deleted afterwards on failure.

Complete the wiki

The wiki has been started and shall be maintained every week, writing down the working of the Nephos.

Refactor imports

Currently the imports are not grouped according to their types, we should be
doing it.

[DOUBT] Should we allow adding single time jobs?

Currently, the module doesn't allow addition of single time jobs and repetition
days must be specified for the jobs.

We can allow adding single time jobs, if need be, by changing the insert_jobs
function.

Post record work

Just after recording a stream, check if it's size is greater than zero, if it's not then make that channel down and remove it's recording instead of adding it to task queue.

Check and catch exit codes for subprocess commands

Currently we rely on the failure of the subprocess command to check if the
recording was successful or not. Instead, we should check the exit code
returned from the command and unless it is 0, do not move further in the
particular method.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.