panoptes-organization / panoptes Goto Github PK

Hi,

Learned about Panoptes recently, very interesting project!

I'm working on an interface for a workflow scheduler/manager. I'm watching the project for releases, and will check again its status after the initial work with snakemake workflow has been done/released.

I couldn't find a ticket for the documentation, not sure if that's too early for that? But what I am more interested is in learning how you guys are planning to get other workflow systems to work with Panoptes.

The sooner that is defined, the easier it would be for developers involved in these projects to assess how hard it would be to create the integration for Panoptes :)

Cheers
Bruno

Create public gitter channel.

Add video on youtube with biohackathon accomplishments.

Split index.html in smaller components. E.g. header, footer, navbars e.t.c.

Add route job status that shows job-specific information.

mockups here: link

when a local run (from snakemake-example-workflow) is performed and the results folder is already present, the badge "workflows now running" in panoptes main page always shows 1 and the badge "workflows executed" never gets populated.

Expected behaviour: even in that corner case, the main page should populate the badges correctly.

Write API tests for our current code

Describe the bug
On the front page in the second box (total workflows), it does not show the total number of workflows, but the total number of workflows that are not complete (error, running, No Execution, etc.)

Expected behavior
The box should show the total number of workflows.

• Investigate cloud solution
• Deploy our docker image there
• Travis to cloud solution
• 2 instances (master & develop)

Describe the bug
The hamburger button is not working

Expected behavior
The button should open/close the menu bar on the left

Add setup.py in order to install and run panoptes easily

We have many components from coreui. We should try and simplify!!!

• Extract job execution information (start time, end time)
• Add them in the database
• Create UI to visualize the information

Is your feature request related to a problem? Please describe.
The current API is having the following endpoints:

An endpoint to delete workflows and jobs is missing

Describe the solution you'd like

Develop an API endpoint that deletes workflows
/api/delete/<workflow-id>

The endpoint should delete both the workflow and the corresponding jobs

Add option in the backend to delete workflows from the database

Update tests to use the master version of snakemake.

Currently, we use a custom branch. We need to rely on the latest master from snakemake.

In the workflows route, show the progress of each workflow and the status.

At the moment it's fine the way we use bootstrap/coreui.io with jinja in flask, but before the project grows too much, we have to think of splitting the front-end from the back-end. We could use coreui.io with React, Angular or Vue.js. The front end server can be node.js.

Add pagination in the in the /workflows page when the number of workflows exceeds a certain number.

Is your feature request related to a problem? Please describe.
At the moment we support only sqlite3 database. This is fine for small applications, but it should not scale well with bigger apps.

Describe the solution you'd like
Test different databases (e.g. mysql) to check if SQL alchemy (that we use) works out of the box, or further changes are required.

Dockerize app...

In the workflow_status page show apart from the progress bar (percent of steps that are executed):

• the jobs that are currently running
• the jobs that are complete

When panoptes is loaded we get the following 404 errors. It does not affect the result, but it's important to fix this. Some import in flask is not set proply.

127.0.0.1 - - [16/Nov/2019 13:41:29] "GET /node_modules/flag-icon-css/css/flag-icon.min.css HTTP/1.1" 404 -
127.0.0.1 - - [16/Nov/2019 13:41:29] "GET /node_modules/@coreui/icons/css/coreui-icons.min.css HTTP/1.1" 404 -
127.0.0.1 - - [16/Nov/2019 13:41:29] "GET /node_modules/font-awesome/css/font-awesome.min.css HTTP/1.1" 404 -
127.0.0.1 - - [16/Nov/2019 13:41:29] "GET /node_modules/simple-line-icons/css/simple-line-icons.css HTTP/1.1" 404 -
127.0.0.1 - - [16/Nov/2019 13:41:29] "GET /node_modules/jquery/dist/jquery.min.js HTTP/1.1" 404 -
127.0.0.1 - - [16/Nov/2019 13:41:29] "GET /node_modules/popper.js/dist/umd/popper.min.js HTTP/1.1" 404 -
127.0.0.1 - - [16/Nov/2019 13:41:29] "GET /node_modules/bootstrap/dist/js/bootstrap.min.js HTTP/1.1" 404 -
127.0.0.1 - - [16/Nov/2019 13:41:29] "GET /node_modules/pace-progress/pace.min.js HTTP/1.1" 404 -
127.0.0.1 - - [16/Nov/2019 13:41:29] "GET /node_modules/perfect-scrollbar/dist/perfect-scrollbar.min.js HTTP/1.1" 404 -
127.0.0.1 - - [16/Nov/2019 13:41:29] "GET /node_modules/@coreui/coreui/dist/js/coreui.min.js HTTP/1.1" 404 -

We can start here

https://github.com/DataBiosphere/toil/blob/master/src/toil/cwl/cwltoil.py#L613

Release version 0.1

Based on the idea introduced from Yohannes Koster we need to create a custom logger in snakemake.

We need to implement a custom logger, that sends updates to your server via HTTP/HTTPS. The user can invoke Snakemake with a special argument, say --wms-monitor, which would trigger the custom logger to be used in addition to console logging. This way, Snakemake could be started from the server process OR via the console, and in both cases, the server process would see the updates.

For example, snakemake workflows can start with sn

Generate UML diagram for server/models.py

• Add utils directory with a script that creates a UML diagram
• Automate this via GitHub action or similar

When a job crashes the database should be updated to failed or error. We get two logs. An example of an error message for the job looks like:

{'msg': "{'name': 'HTSeq_count', 'jobid': 3, 'output': ['results/control_rep2.htseq.tsv'], 'log': ['logs/local_log/HTSeq_count_control_rep2.log'], 'conda_env': '/Users/foivosgypas/Desktop/snakemake_example_workflow/.snakemake/conda/d9cacd2d', 'aux': {}, 'indent': False, 'shellcmd': '(htseq-countasdasd         --format bam         --order pos         --stranded yes         --type exon         --idattr gene_id         --mode union         results/control_rep2.sorted.bam samples/Homo_sapiens.GRCh37.85.chr14.gtf > results/control_rep2.htseq.tsv) &> logs/local_log/HTSeq_count_control_rep2.log', 'level': 'job_error'}", 'timestamp': 'Sun Nov 24 10:31:19 2019', 'id': 3}

The second log we have is the following:

{'msg': "{'level': 'error', 'msg': 'Exiting because a job execution failed. Look above for error message'}", 'timestamp': 'Sun Nov 24 10:31:19 2019', 'id': 3}

We need the following endpoints:

• /api/workflows
• /api/<workflow_id>
• /api/jobs/<workflow_id>
• /api/job/<workflow_id>/<job_id>

Describe the bug
Snakemake crashes when ip/port of panoptes is not correct

To Reproduce

• Install and run panoptes
• Add flag with wrong flag (e.g. --wms-monitor "http://127.0.0.1:5001")

Expected behavior
The correct behavior is to exit properly with a warning message

Screenshots or logs

Traceback (most recent call last):
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/urllib3/connection.py", line 157, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/urllib3/connectionpool.py", line 387, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/Users/foivosgypas/miniconda3/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Users/foivosgypas/miniconda3/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Users/foivosgypas/miniconda3/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Users/foivosgypas/miniconda3/lib/python3.7/http/client.py", line 1016, in _send_output
self.send(msg)
File "/Users/foivosgypas/miniconda3/lib/python3.7/http/client.py", line 956, in send
self.connect()
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/urllib3/connection.py", line 184, in connect
conn = self._new_conn()
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/urllib3/connection.py", line 169, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x10d080828>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=5001): Max retries exceeded with url: /create_workflow (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x10d080828>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/snakemake/logging.py", line 455, in setup_logger
r = requests.get(wms_monitor + "/create_workflow")
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=5001): Max retries exceeded with url: /create_workflow (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x10d080828>: Failed to establish a new connection: [Errno 61] Connection refused'))
Building DAG of jobs...
Traceback (most recent call last):
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/snakemake/init.py", line 627, in snakemake
batch=batch,
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/snakemake/workflow.py", line 548, in execute
logger.info("Building DAG of jobs...")
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/snakemake/logging.py", line 157, in info
self.handler(dict(level="info", msg=msg, indent=indent))
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/snakemake/logging.py", line 132, in handler
handler(msg)
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/snakemake/logging.py", line 228, in custom_server_handler
server_info = {'msg': repr(msg), 'timestamp': time.asctime(), 'id': self.server["id"]}
AttributeError: 'Logger' object has no attribute 'server'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/bin/snakemake", line 8, in
sys.exit(main())
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/snakemake/init.py", line 2029, in main
wms_monitor=args.wms_monitor
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/snakemake/init.py", line 636, in snakemake
print_exception(ex, workflow.linemaps)
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/snakemake/exceptions.py", line 73, in print_exception
log_verbose_traceback(ex)
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/snakemake/exceptions.py", line 61, in log_verbose_traceback
logger.debug(tb)
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/snakemake/logging.py", line 163, in debug
self.handler(dict(level="debug", msg=msg))
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/snakemake/logging.py", line 132, in handler
handler(msg)
File "/Users/foivosgypas/Desktop/snakemake/venv_snakemake_develop/lib/python3.7/site-packages/snakemake/logging.py", line 228, in custom_server_handler
server_info = {'msg': repr(msg), 'timestamp': time.asctime(), 'id': self.server["id"]}
AttributeError: 'Logger' object has no attribute 'server'

Update info on the following repo:

https://github.com/elixir-europe/BioHackathon-projects-2019
https://github.com/elixir-europe/BioHackathon-projects-2019/tree/master/projects/14

Describe the bug
status colors between /workflows and /workflow/ do not match

• Write up a manuscript draft
• Share it with contributors
• Submit to bioRxiv or arXiv

Automate the build and test of the service using CI/CD

• CI
• CI-docker build
• CD

Create a SPA application with react.js

@agardelakos: I think we need help from @venson7 for this. Let's have a skype call soon...

• Handshake between snakemake and the server
• Store level info (progress) in the database
• Work with cases where there is no level info (progress) available
• Make proper visualization

Maybe we need contributors to sign some kind of EULA agreement before we start reviewing their PR (or even at the creation of a ticket). Real life example of the agreement is this and an example use case is this.

This ticket is not about writing code, only to track and document the investigation of the above

Do we need to install the coreui components via npm or can we avoid it?

Update README with running example workflow (https://github.com/fgypas/snakemake_example_workflow)

Add also some instructions in the index.html page.

Add general config file for the flask server with info such as IP, port, path, run mode (develop/production), db path e.t.c.

We need to dynamically create a dependency graph based on the input and output files to visualize the progress of the workflows.

Some options we should consider from snakemake
--d3dag, --dag, --rulegraph, --filegraph

The users should be able to login to the server to get a JSON token and export it in the environment. The HTTP requests from snakemake to panoptes should be decorated with the token.

Make a 2-minute presentation for Biohackathon 2019.

• Add conda instructions and package it for bioconda

• Add pypi package

Is your feature request related to a problem? Please describe.
When a big number of workflows is running the website further down.

Describe the solution you'd like
We need to add a pagination option that shows only a specific number of workflows (the user can select e.g. max 50 workflows per page).

Add button/route under "Show all workflows" that cleans up the database. It would be nice if we can to ask for confirmation, as people might press it by accident.