Review docker-compose `links` and `depends_on` about renku HOT 16 CLOSED

• graph-*: looks ok
• resource-manager should be:

links:
- cassandra
- reverse-proxy
depends_on:
- graph-typesystem
- keycloak

• storage should be:

links:
- cassandra
- reverse-proxy
depends_on:
- graph-typesystem
- graph-mutation
- resource-manager
- keycloak

• explorer should be:

links:
- cassandra
- reverse-proxy
depends_on:
- graph-typesystem
- keycloak

from renku.

Should depend-on specify services that are needed for successful execution of function or simply what is needed to start the service? I think the latter... and in this case:

• resource-manager and storage don't depend on keycloak -- they will start and work without it (or they should, anyway)
• does storage need to see the other services to start?

from renku.

resource-manager won't start without keycloak, storage will, but will 500 at every request.
Also, what's the point of starting without ensuring functional behavior?

from renku.

Well, take the deployer as an example. If it can't add vertices to the knowledge graph, should it refuse to create and execute contexts? I would say no -- it should just acknowledge that the KG is unavailable and that the state of the executions will be synced later. It doesn't actually do this now, but it probably should eventually. I'm not sure this pattern makes sense for other services that's why I asked the question.

from renku.

the storage uses keycloak to get the public key to validate the incoming tokens. (as discussed with leafty, this could be made optional as the user credentials are anyway checked by the RM)
the storage uses the graph mutation to populate the graph when reading/writing files/buckets. This is currently mandatory as it checks if it was correctly added before returning the permission token.
the storage uses the RM, obviously for managing the permissions.
the storage doesn't use the type-system as far as I know? or is it a transitive dependency?

from renku.

@ebiiii typesystem is needed when you use vertexReader or EdgeReader.
As it needs datatypes and such for reconstruction objects from janusgraph.

from renku.

does the RM need KC because it fetches the keys? If the idea is that the RM should work with other identity providers, should it not just be given the public keys to check the signatures?

For other services, they should just rely on the RM to check the tokens IMHO

from renku.

@rokroskar

it should just acknowledge that the KG is unavailable and that the state of the executions will be synced later.

This is contrary to the current vision of the platform. Please synchronise with @erbou on this topic. The deployer behaving like this will break lineage and linkage done by other services when accessed from within the deployed application.

from renku.

I understand that -- we've had these discussions before. I'm just trying to clarify what actually depends on what and why.

from renku.

In my view, depends_on specifies what the service needs to start. In many cases, the hard dependency is the db because the internally-used tables have to be initialized. Apart from that, the rest are soft dependencies. Yes, there will be runtime errors but the service won't crash. The runtime errors should be handled gracefully giving an informative message and let the service continue to run.

If we make soft dependencies into hard dependencies, I think it will make it really hard later on to decentralize/distribute and scale.

from renku.

As a user I want to be able to do

docker-compose up -d storage

and use it when it's ready. I DO NOT want to see any error message.

The docker-compose does absolutely not describe a production deployment, for which it is good/necessary to try failure scenarios where services have to deny access due to missing dependents.

from renku.

As a user I want to be able to do

docker-compose up -d storage

If you want to run the service separately then we should encourage people to use Dockerfile and docker-compose.yml from each repo respectively.

from renku.

ok I think we're splitting hairs here. Practically, there is no difference between typing docker-compose up -d and docker-compose up -d storage. However, by eliminating redundant links in the docker-compose.yml we can force ourselves to write the services in a way that minimizes their interdependence. Therefore my insistence on figuring out the answer to e.g. "does storage really need KC?" Of course the docker-compose doesn't yield a production environment, but it's our primary means of R&D so if we do it poorly here there is no reason to believe we'll do it better in production.

from renku.

Practically, there is no difference between typing docker-compose up -d and docker-compose up -d storage.

There is one if you don't bring up dependencies for docker-compose up storage, as it won't work in that case.

from renku.

that's what I meant -- why would you want to bring up specifically just storage? You would just type docker-compose up and be done with it.

from renku.

Dead conversation

from renku.