swissdatasciencecenter / renku Goto Github PK

Renku provides a platform and tools for reproducible and collaborative data analysis.

License: Apache License 2.0

Shell 1.58% Python 13.62% Dockerfile 1.15% Scala 46.31% Mustache 6.22% Smarty 1.49% TypeScript 21.70% Nix 0.78% JavaScript 7.15%

datascience collaboration reproducibility renku knowledge-graph

renku's Introduction

Renku

Renku is a platform that bundles together various tools for reproducible and collaborative data analysis projects. It is aimed at independent researchers and data scientists as well as labs, collaborations, and courses and workshops. Renku can be used by anyone who deals with data, whether they are a researcher, data analyst, project owner, or data provider.

Renku promotes reproducibility by providing tools to track your analysis workflows and save them together with your versioned data, code, and environment specification. Every result can be replayed either to repeat a calculation or to re-execute on new data or with a different choice of parameters.

Renku encourages reusability by storing and querying the connections between datasets, code executions, and results in a Knowledge Graph. Producers and consumers of analysis artifacts can always recover the full provenance of a result, establishing trust and reducing boilerplate.

Renku stimulates collaboration among peers and across disciplines by guaranteeing that a media-rich discussion space and fully configured, shareable interactive computational environments are always just a click away. Collaborators can easily work on projects together or in parallel, combining their work in a systematic and safe manner.

Getting Started

Renku consists of RenkuLab, a web-based application and Renku Client, a command-line tool for managing code, data, workflows and making practical use of the Knowledge Graph.

A public instance of RenkuLab is available at https://renkulab.io, and there are several other deployments at various institutions. To start exploring Renku, feel free to make an account and try it out! You can follow the first steps tutorial or continue reading about the Renku project.

Documentation

Tutorials: how to get your Renku work off the ground
Topic Guides: discussions about concepts central to Renku
How-to Guides: recipes for common use-cases with Renku for users and administrators
Reference: syntax, structure, architecture, etc.

Contributing

We're happy to receive contributions of all kinds, whether it is an idea for a new feature, a bug report or a pull request!

Please review our contributing guidelines before submitting a pull request.

Getting in touch

There are several channels you can use to communicate with us; we monitor all of them, so your messages will always get to us, but communication will be slightly more streamlined if you pick a channel that most suits your purpose and needs.

discourse: questions concerning Renkulab or Renku CLI usage, release notes
github & renku (CLI) <https://github.com/SwissDataScienceCenter/renku-python>: create platform-usability and software-bug issues
gitter: communicate with the team

Renku is developed as an open source project by the Swiss Data Science Center in a team split between EPFL and ETHZ.

Project structure

Renku consists of several sub-repositories:

renku-gateway: a simple API gateway
renku-graph: Knowledge Graph services
renku-notebooks: a lightweight service for handling interactive notebooks through Jupyter servers and a k8s operator
amalthea: k8s operator for Jupyter servers.
renkulab-docker: base images for interactive sessions
renku-python: the Python CLI, SDK, and core backend service
renku-ui: web front-end
renku-project-templates: base templates used for instantiating renku projects.
renkulab-docker: docker images used for renku sessions.

renku's People

Contributors

Stargazers

Watchers

renku's Issues

Define client for CLI in Keycloak

Probably with publicClient: true.

apispec doesn't work when logging in with ui

Issue with url: http://localhost/api/swagger.json
Logs:

apispec_1           | invalid request block size: 5225 (max 4096)...skip
apispec_1           | [uwsgi-http key: localhost client_addr: 172.18.0.5 client_port: 58040] hr_instance_read(): Connection reset by peer [plugins/http/http.c line 646]

Probable cause: server rejects the request headers due to size with cookies set by ui after login.

Reproducing:

Log in to http://localhost/ui/
Try to load http://localhost/admin/swagger/ -> fails

remove references to internal.datascience.ch

several scala packages reference internal.datascience.ch -- these should probably be removed
To do:

Support deployment on different domain than localhost

configurable docker-compose.yml
generate certificates using letsencrypt

Supporting the simple(st) user story

This is a meta-issue to keep track of work done to support the basic data science user story.

Please add a link to the relevant issue once it's created in the corresponding repository. Track the fine-grained issues in sprint project.

0. Housekeeping

1. Initiate a project, import a dataset

Finalize dataset import to work with renga repos (SwissDataScienceCenter/renku-python#99)
update .gitattributes to track outputs of renga run and datasets in LFS (SwissDataScienceCenter/renku-python#104)

metadata
- document the ontology for projects, datasets, and CWL steps/workflows (https://github.com/SwissDataScienceCenter/renga-ontology/issues/3)
- include JSON-LD context with all metadata (SwissDataScienceCenter/renku-python#105) (@jirikuncar)

2. Explore the dataset using a notebook (isolated environment)

Look into capturing data artifacts the same way as with renga run

3. Run code on data, recording data provenance for reproducibility

write user stories as test cases for renga run
Fix bugs with provenance calculation (SwissDataScienceCenter/renku-python#106)
Resolve lineage in submodules (SwissDataScienceCenter/renku-python#120)
Implement renga status that checks the validity of data provenance (SwissDataScienceCenter/renku-python#107)
Write out workflows, fix problem with relative paths (SwissDataScienceCenter/renku-python#98)

(Dockerization & python environments -- for future sprint)

4. Push project to a server for sharing

Finalize storage service
- decouple from graph (SwissDataScienceCenter/renku-storage#44)
- update swagger and doc (SwissDataScienceCenter/renku-storage#50)
- define bucket/backend for LFS (SwissDataScienceCenter/renku-storage#51)
Memory provisioning for gitlab on testing.datascience.ch (@ebiiii / @leafty)
Configure gitlab runners on k8s
- Set up the ingress for notebook servers @leafty
- Look into resource limitations of runners
UI running on testing.datascience.ch
- Resolve the token issue (reuse flask proxy from old UI? -> @3C111 / @leafty / @ebiiii) - authenticating the UI with keycloak done in SwissDataScienceCenter/renku-ui#118
- Configure deployment
Manipulate project access settings
Initialize a project in UI (templates? Gitlab-ci, .renga metadata etc.)
renga push/sync (index the provenance information)
staging environment (https://github.com/SwissDataScienceCenter/incubator-renga-k8s/issues/1)

5. Validate results/workflows

Troubleshoot the CI configuration -- related to SwissDataScienceCenter/renku-python#102

6. Initiate discussions (ku) on the work done

create/participate in discussions (SwissDataScienceCenter/renku-ui#108)
* Markdown editors
File browser for linking to entities in the repo
Connect notebooks to live notebook server (SwissDataScienceCenter/renku-ui#114)
* Dataset view, data preview

cannot run deployer with FLASK_DEBUG: 1

when adding FLASK_DEBUG: 1 to docker-compose.yml the deployer fails to start

Docs on https://renga.readthedocs.io/

Now they are on https://renga-documentation.readthedocs.io/.
Needs to be working on https://renga.readthedocs.io/.

Merge swagger definitions from all services

Naming conventions for environmental variables

Proposal

service related: RENGA_{SERVICE}_{WHATEVER_WITH_UNDERSCORES}
global config for multiple services RENGA_SERVICES_{...}

Acceptance

--
cc @erbou @leafty @rokroskar @ebiiii

Clean renga branches

Unify trailing slashes in APIs

client_id default to renga-client

I would propose to change the default Keycloak client_id to renga-client from demo-client.

Documentation: RENGA or Renga?

Decide of the proper / consistent spelling for Renga/RENGA and apply everywhere in the docs.

CI: use CLI and Python SDK for integration tests

error during renga init --autosync

Following tutorial I got this error:

$ renga init --autosync

Traceback (most recent call last):
  File "/Users/andriineronov/anaconda/envs/python3/bin/renga", line 11, in <module>
    sys.exit(cli())
  File "/Users/andriineronov/anaconda/envs/python3/lib/python3.5/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/andriineronov/anaconda/envs/python3/lib/python3.5/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/andriineronov/anaconda/envs/python3/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/andriineronov/anaconda/envs/python3/lib/python3.5/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/andriineronov/anaconda/envs/python3/lib/python3.5/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/andriineronov/anaconda/envs/python3/lib/python3.5/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/andriineronov/anaconda/envs/python3/lib/python3.5/site-packages/renga/cli/_config.py", line 82, in new_func
    result = ctx.invoke(f, config, *args, **kwargs)
  File "/Users/andriineronov/anaconda/envs/python3/lib/python3.5/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/andriineronov/anaconda/envs/python3/lib/python3.5/site-packages/renga/cli/init.py", line 74, in init
    project = client.projects.create(name=name)
  File "/Users/andriineronov/anaconda/envs/python3/lib/python3.5/site-packages/renga/models/projects.py", line 42, in create
    data = self._client.api.create_project({'name': name})
  File "/Users/andriineronov/anaconda/envs/python3/lib/python3.5/site-packages/renga/api/projects.py", line 40, in create_project
    json=project, )
  File "/Users/andriineronov/anaconda/envs/python3/lib/python3.5/site-packages/renga/api/client.py", line 43, in decorator
    f(*args, **kwargs), expected_status_code)
  File "/Users/andriineronov/anaconda/envs/python3/lib/python3.5/site-packages/renga/errors.py", line 66, in return_or_raise
    raise cls(response)
renga.errors.UnexpectedStatusCode: Unexpected status code: 500

fix failing travis builds

the renga builds are failing on travis for some time -- looks like it could be due to the xenial image: travis-ci/travis-ci#9080 (comment)

Documentation: developer docs for the graph components

set up a travis build

we need a travis build for the "umbrella" repo -- for the moment, would be enough to just run make all

make start fails

Following instructions at http://renga.readthedocs.io/en/latest/user/setup.html#setup. Calling make start fails:

...
srcrenga_explorer_1 is up-to-date
srcrenga_storage_1 is up-to-date
Starting srcrenga_graph-init_1 ... done
docker: Error response from daemon: No such network: renga_default.
make: *** [start] Error 125

Not clear to me how to solve this.

Running on macOS 10.12. Docker 17.09.0-ce-mac35 (19611) is installed, memory increased as recommended.

`docker-compose up` in master fails

two probably related problems:

keycloak:

21:34:12,080 ERROR [org.jboss.resteasy.resteasy_jaxrs.i18n] (default task-8) RESTEASY002010: Failed to execute: javax.ws.rs.NotFoundException: Realm does not exist

resource-manager:

resource-manager_1  | 2017-08-27T21:35:00.464543016Z 4) Error injecting constructor, com.fasterxml.jackson.databind.JsonMappingException: No content to map due to end-of-input
resource-manager_1  | 2017-08-27T21:35:00.464548110Z  at [Source: [B@86733; line: 1, column: 0]
resource-manager_1  | 2017-08-27T21:35:00.464554203Z   at controllers.PermissionController.<init>(PermissionController.scala:26)
resource-manager_1  | 2017-08-27T21:35:00.464560796Z   at controllers.PermissionController.class(PermissionController.scala:24)
resource-manager_1  | 2017-08-27T21:35:00.464567188Z   while locating controllers.PermissionController
resource-manager_1  | 2017-08-27T21:35:00.464573581Z     for parameter 2 at router.Routes.<init>(Routes.scala:31)
resource-manager_1  | 2017-08-27T21:35:00.464580573Z   while locating router.Routes
resource-manager_1  | 2017-08-27T21:35:00.464587565Z   while locating play.api.inject.RoutesProvider
resource-manager_1  | 2017-08-27T21:35:00.464598452Z   while locating play.api.routing.Router
resource-manager_1  | 2017-08-27T21:35:00.464603846Z     for parameter 0 at play.api.http.JavaCompatibleHttpRequestHandler.<init>(HttpRequestHandler.scala:201)
resource-manager_1  | 2017-08-27T21:35:00.464611138Z   while locating play.api.http.JavaCompatibleHttpRequestHandler
resource-manager_1  | 2017-08-27T21:35:00.464617930Z   while locating play.api.http.HttpRequestHandler
resource-manager_1  | 2017-08-27T21:35:00.464625022Z     for parameter 4 at play.api.DefaultApplication.<init>(Application.scala:221)
resource-manager_1  | 2017-08-27T21:35:00.464633012Z   at play.api.DefaultApplication.class(Application.scala:221)
resource-manager_1  | 2017-08-27T21:35:00.464640404Z   while locating play.api.DefaultApplication
resource-manager_1  | 2017-08-27T21:35:00.464647096Z Caused by: com.fasterxml.jackson.databind.JsonMappingException: No content to map due to end-of-input
resource-manager_1  | 2017-08-27T21:35:00.464653988Z  at [Source: [B@86733; line: 1, column: 0]
resource-manager_1  | 2017-08-27T21:35:00.464659282Z 	at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:270)
resource-manager_1  | 2017-08-27T21:35:00.464664176Z 	at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3838)
resource-manager_1  | 2017-08-27T21:35:00.464668771Z 	at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3756)
resource-manager_1  | 2017-08-27T21:35:00.464675264Z 	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2099)
resource-manager_1  | 2017-08-27T21:35:00.464682156Z 	at play.api.libs.json.jackson.JacksonJson$.parseJsValue(JacksonJson.scala:235)
resource-manager_1  | 2017-08-27T21:35:00.464688548Z 	at play.api.libs.json.Json$.parse(Json.scala:40)
resource-manager_1  | 2017-08-27T21:35:00.464694941Z 	at play.api.libs.ws.ahc.AhcWSResponse.json$lzycompute(AhcWS.scala:538)
resource-manager_1  | 2017-08-27T21:35:00.464701234Z 	at play.api.libs.ws.ahc.AhcWSResponse.json(AhcWS.scala:538)
resource-manager_1  | 2017-08-27T21:35:00.464708825Z 	at ch.datascience.service.security.PublicKeyReader$$anonfun$fetchRSAPublicKey$1.apply(PublicKeyReader.scala:50)
resource-manager_1  | 2017-08-27T21:35:00.464715717Z 	at ch.datascience.service.security.PublicKeyReader$$anonfun$fetchRSAPublicKey$1.apply(PublicKeyReader.scala:48)
resource-manager_1  | 2017-08-27T21:35:00.464732298Z 	at scala.util.Success$$anonfun$map$1.apply(Try.scala:237)
resource-manager_1  | 2017-08-27T21:35:00.464740588Z 	at scala.util.Try$.apply(Try.scala:192)
resource-manager_1  | 2017-08-27T21:35:00.464747580Z 	at scala.util.Success.map(Try.scala:237)
resource-manager_1  | 2017-08-27T21:35:00.464753973Z 	at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:237)
resource-manager_1  | 2017-08-27T21:35:00.464763262Z 	at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:237)
resource-manager_1  | 2017-08-27T21:35:00.464769955Z 	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
resource-manager_1  | 2017-08-27T21:35:00.464781241Z 	at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
resource-manager_1  | 2017-08-27T21:35:00.464787135Z 	at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
resource-manager_1  | 2017-08-27T21:35:00.464792229Z 	at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
resource-manager_1  | 2017-08-27T21:35:00.464802217Z 	at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
resource-manager_1  | 2017-08-27T21:35:00.464809409Z 	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
resource-manager_1  | 2017-08-27T21:35:00.464816001Z 	at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
resource-manager_1  | 2017-08-27T21:35:00.464822594Z 	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
resource-manager_1  | 2017-08-27T21:35:00.464829786Z 	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
resource-manager_1  | 2017-08-27T21:35:00.464834980Z 	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
resource-manager_1  | 2017-08-27T21:35:00.464839674Z 	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
resource-manager_1  | 2017-08-27T21:35:00.464844269Z 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
resource-manager_1  | 2017-08-27T21:35:00.464848764Z 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

cc @jirikuncar @ebiiii @leafty -- this needs to be fixed asap please...

docs: standardize docs sources to have 80 character width

~~add "RST" linter~~ (will not do now)

document best api practices

create a best api practices reference do in Renga's developer documentation

based on https://docs.google.com/presentation/d/1SGlkK0UXeXcbseHaRy1tq3p1cwQMxDSTgVoXgu22uLE/edit#slide=id.p

Integrate pieces of old Makefile that builds images across all repos with the new Makefile in development

addresses #117

moved over pieces of the old makefile to build docker images across the repos
build specific branches

Generate the target list in wait-for-services.sh

Probably by using docker labels or parsing docker-compose.yml.

Development branches

To do for each repository with an incubator or development branch:

Migrate "incubator" repos to "development" branches of renga-* repositories, when ready
At the same time, add a disclaimer/cautionary note to the README in the master and development branches.
Update the README in the development branch to reflect the new component
Protect the development branch.

renga
renga-python (add a note/disclaimer to README)
renga-ui
~~- [ ] incubator-renga-graph~~ (not working on it this sprint)
renga-storage (create the development branch)

running renga-ui image built with Makefile fails

the Makefile target for renga-ui should run the npm commands before building the docker image

docs: add description about how keycloak is configured and how to change it

travis master branch build errors

the build errors because it tries to clone the repos over ssh without a key -- switch to https

[doc] dead link

In page https://renga.readthedocs.io/en/latest/developer/code_repo_structure.html

the link https://github.com/SwissDataScienceCenter/renga-tutorials is dead.

Fix API version to 0.1.0

Use volumes for db and cassandra?

Documentation: first steps/installation of renga

verify that the `example.sh` script works

doesn't seem to work at the moment.

needs to be updated to point to http://localhost instead of port 9000 for services and port 8080 for keycloak
update API calls to storage

Setup user support system

Beyond git issues, how we are going to assist users.

quickstart: login yields internal server error

I've been following the instructions to install renga locally on
http://renga.readthedocs.io/en/latest/user/setup.html#getting-the-code-and-images

Everything works fine and I can navigate to http://localhost/ui
Then, when I click on the "login" button below, I receive an internal server error (500).

The URL http://localhost/admin/swagger/ works fine.

I cannot connect via the python client,though:

$ renga login http://localhost
Username: demo
Password:
/Users/leopold/Applications/miniconda3/lib/python3.6/site-packages/renga/api/client.py:83: UserWarning: Using insecure trasnport protocol, use HTTPS
  warnings.warn('Using insecure trasnport protocol, use HTTPS')
Traceback (most recent call last):
  File "/Users/leopold/Applications/miniconda3/bin/renga", line 11, in <module>
    sys.exit(cli())
  File "/Users/leopold/Applications/miniconda3/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/leopold/Applications/miniconda3/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/leopold/Applications/miniconda3/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/leopold/Applications/miniconda3/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/leopold/Applications/miniconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/leopold/Applications/miniconda3/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/leopold/Applications/miniconda3/lib/python3.6/site-packages/renga/cli/_config.py", line 91, in new_func
    result = ctx.invoke(f, config, *args, **kwargs)
  File "/Users/leopold/Applications/miniconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/leopold/Applications/miniconda3/lib/python3.6/site-packages/renga/cli/login.py", line 58, in login
    scope=scope)
  File "/Users/leopold/Applications/miniconda3/lib/python3.6/site-packages/renga/api/authorization.py", line 62, in fetch_token
    return super(AuthorizationMixin, self).fetch_token(token_url, **kwargs)
  File "/Users/leopold/Applications/miniconda3/lib/python3.6/site-packages/requests_oauthlib/oauth2_session.py", line 221, in fetch_token
    verify=verify, proxies=proxies)
  File "/Users/leopold/Applications/miniconda3/lib/python3.6/site-packages/renga/api/client.py", line 43, in decorator
    f(*args, **kwargs), expected_status_code)
  File "/Users/leopold/Applications/miniconda3/lib/python3.6/site-packages/renga/errors.py", line 66, in return_or_raise
    raise cls(response)
renga.errors.UnexpectedStatusCode: Unexpected status code: 404

I'm running docker 17.09.0-ce-mac35 (19611) on MacOS 10.12.6 and am using the master branch.

cc/ @leafty @erbou @3C111 @jirikuncar

Sentry for Renga-Deployer @jirikuncar

Clean .ivy2 directories

In link to: sbt/sbt#2687, I'd suggest to have:

rm -rf ~/.ivy2/cache/ch.datascience/<artifact>

after the corresponding artifact is produced by the build.