pypyr-scheduler / pypyr-scheduler-server Goto Github PK

View Code? Open in Web Editor NEW

4.0 1.0 2.0 386 KB

Schedule pypyr pipelines with apscheduler

License: MIT License

Python 99.24% Shell 0.76%

pypyr scheduler restful-api apscheduler pipelines

pypyr-scheduler-server's Introduction

Schedule pypyr pipelines with apscheduler and control them via REST. The API interface is provided by connexion.

Documentation status

This documentation is in a very early stage and many things could be missing or wrong. Please rely on the code for now.

Install

Make a new venv, activate it, clone the repo, run pip install. No pypi release yet. We recommend using pipenv, which makes it easy to run commands inside a virtual env without the need to create or activate it:

pip install --user pipenv  # only needed once per python install
pipenv install
pipenv run server

Usage

pypyr-scheduler needs a shared secret between the server and the client. It reads it from the environment variable PYRSCHED_SECRET. If the server does not see a secret it generates it and writes it to its logfile for later use. If there is a secret set in the environment, it is used.

A client needs the same shared secret.

Development / Testing

Run pipenv run server. Now you can connect to the server with a suitable client. Currently available are pypyr-scheduler-cli and pypyr-scheduler-rpc-client with the first one depending on the latter.

Production deployment

TBD.

Documentation

Detailed documentaion is available on ReadTheDocs. It is generated from the docs/source folder in this repository. Feel free to send a PR is you find any typos.

Development

Run tests with pipen run tests.

Quick links

Related projects

pypyr is the workhorse underlying pypyr-scheduler. It runs pipelines defines as .yaml file and has many different pipeline steps included. Check it out, if you need a simple task automation for one-shot execution.

Flask-APScheduler provides a similar way to run job within flask as server. It even provides a REST-API. Try this if you don't need the functionality of pypyr.

pypyr-scheduler-server's People

Contributors

Stargazers

Watchers

Forkers

sturmianseq yaythomas

pypyr-scheduler-server's Issues

Descriptions in command line help are not consistent.

Describe the bug
Descriptions in command line help are not consistent. Sometimes, they end with a ".", sometimes not

To Reproduce
run python -m pyrsched -h

Expected behavior
The descriptions should either end with a period or not, but all the same.

pipeline logs get created but have no content

Describe the bug
Logfiles for pipelines are created but contain no log entries

To Reproduce
Steps to reproduce the behavior:

Upload a pipeline
Create a job
Start the job

Expected behavior
The pipeline logs to its own log

Start development server with --enable-upload
Upload a pipeline file via the swagger ui console
Look at the (default) pipelinesdirectory for the uploaded file.

Expected behavior
Pipeline was uploaded and saved to the file system.

DEBUG:connexion.decorators.validation:http://localhost:5000/pipelines/pipeline_now.yaml validating parameters...
DEBUG:connexion.decorators.parameter:Function Arguments: ['filename', 'body']
INFO:pyrsched.api.pipelines:POST /pipelines/pipeline_now.yaml
INFO:pyrsched.api.pipelines:saving file contents to /home/dzerrenn/projects/pypyr-scheduler/pipelines/pipeline_now.yaml
ERROR:pyrsched.app:Exception on /pipelines/pipeline_now.yaml [POST]
Traceback (most recent call last):
  File "/home/dzerrenn/.local/share/virtualenvs/pypyr-scheduler-UBNwg3Am/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/dzerrenn/.local/share/virtualenvs/pypyr-scheduler-UBNwg3Am/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/dzerrenn/.local/share/virtualenvs/pypyr-scheduler-UBNwg3Am/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/dzerrenn/.local/share/virtualenvs/pypyr-scheduler-UBNwg3Am/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/dzerrenn/.local/share/virtualenvs/pypyr-scheduler-UBNwg3Am/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/dzerrenn/.local/share/virtualenvs/pypyr-scheduler-UBNwg3Am/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/dzerrenn/.local/share/virtualenvs/pypyr-scheduler-UBNwg3Am/lib/python3.6/site-packages/connexion/decorators/decorator.py", line 73, in wrapper
    response = function(request)
  File "/home/dzerrenn/.local/share/virtualenvs/pypyr-scheduler-UBNwg3Am/lib/python3.6/site-packages/connexion/decorators/uri_parsing.py", line 132, in wrapper
    response = function(request)
  File "/home/dzerrenn/.local/share/virtualenvs/pypyr-scheduler-UBNwg3Am/lib/python3.6/site-packages/connexion/decorators/validation.py", line 165, in wrapper
    response = function(request)
  File "/home/dzerrenn/.local/share/virtualenvs/pypyr-scheduler-UBNwg3Am/lib/python3.6/site-packages/connexion/decorators/validation.py", line 339, in wrapper
    return function(request)
  File "/home/dzerrenn/.local/share/virtualenvs/pypyr-scheduler-UBNwg3Am/lib/python3.6/site-packages/connexion/decorators/decorator.py", line 44, in wrapper
    response = function(request)
  File "/home/dzerrenn/.local/share/virtualenvs/pypyr-scheduler-UBNwg3Am/lib/python3.6/site-packages/connexion/decorators/parameter.py", line 126, in wrapper
    return function(**kwargs)
  File "/home/dzerrenn/projects/pypyr-scheduler/pyrsched/api/pipelines.py", line 55, in create
    return _save_file(filename, body)
  File "/home/dzerrenn/projects/pypyr-scheduler/pyrsched/api/pipelines.py", line 44, in _save_file
    content.save(str(target))
AttributeError: 'bytes' object has no attribute 'save'
DEBUG:connexion.apis.flask_api:Getting data and status code
DEBUG:connexion.apis.flask_api:Got data and status code (500)
INFO:werkzeug:127.0.0.1 - - [12/Dec/2019 08:43:51] "POST /pipelines/pipeline_now.yaml HTTP/1.1" 500 -

example configuration files are not packaged together with pyrsched

Describe the bug
If the package is installed via pip install, there are no default config files.