A new development plan is needed in order to achieve a complete version 1 of the Open Integration Hub. To have a shared approach and a common goal, it is necessary to adjust the old project plan to all new insights.

Acceptance Criteria:

• New project plan for AP 4 is created with updated time scope and resource availabilities
• Plan is communicated across all development teams

As a OIH user I want to attach my secrets to any step of an integration flow.

The container executing the flow step should have a read access to the secret or receive the secrets as env vars.
Users with access to the flow can see the name of the secret but not the contents.

As mentioned in #26 some of our services are operating on Flow CRD. We should replace the elastic.io by openintegrationhub.org. The following CRD example demonstrates that metadata.name and spec.group.

{
    "apiVersion": "apiextensions.k8s.io/v1beta1",
    "kind": "CustomResourceDefinition",
    "metadata": {
        "name": "flows.elastic.io"
    },
    "spec": {
        "group": "elastic.io",
        "names": {
            "kind": "Flow",
            "listKind": "FlowList",
            "plural": "flows",
            "shortNames": [],
            "singular": "flow"
        },
        "scope": "Namespaced",
        "version": "v1"
    }
}

The mapper component currently lies within the elastic.io repository on dockerhub.

All components should be open sourced under open integration hub.

Acceptance Criteria:

• Mapper component is moved to openintegrationhub repository on dockerhub

Provide a concept similar to that of IAM which describes the idea, architecture and usage.

Acceptance Criteria:

• Concept has been published on Github (Link to concept)

As product owner I want to have an updated development plan to create a common understanding of the time critical services.

In order to create a new overall development plan a new project plan for AP 4 is needed by each partner.

Acceptance Criteria:

• Update the old project plan including resource and time planning
• Add actual available resources (current excel sheet includes resource planning before cutback)
• Send the updated project plan to Philipp

All images should be hosted on DockerHub

Further description:
Right now there is no image for the ICR on DockerHub

Acceptance Criteria:

• Publish icr image at DockerHub OIH-Repo

As a OIH user I want to be able to define, if a secret can only be accessed by me or also by my organization. (A secret is a key value pair.)

• Secrets scope
• MultiTenancy

acceptance:

• There is a Api which Handles the Accesscontrol Management
• The Api and the concept is documented

• Define which data can and should be consumed
  • Consume event data from messaging middleware, event bus, logs, flows status, etc.
• Implement authentication & authorization mechanisms

At the moment RabbitMQ is installed into k8s cluster as a Deployment with replicas=1 which is quite nice for running the platform locally. However for a production use we need to rely on high availability. Is it better to rely on an RabbitMQ cluster outside of the k8s cluster?

This confuses k8s' distribution mechanisms, and in worst case all components may be started at same node, and overload it.

• Should have a stable API to allow a potential interchangability of underlying infrastructure/orchestrator
• Consider providing a shared library (npm module) for all service, to generate a uniform log output?

Webhooks, which the OIH provides for external triggers (conceptional elaborations), should ideally be secured with credentials/authorization mechanisms provided by IAM or with a HMAC (similar to current elastic platform) managed solely by communication router itself.

We as OIH partners need development styleguideline to have a a common code quality across all partners and services of the OIH.

Acceptance Criteria:

• Styleguide reviewed by microservice + architecture workgroup
• Styleguide is published within the monorepo

As product owner I expect related technical user stories needed to implement the described features in epic

Acceptance Criteria:

• User stories are created
• User stories are reviewed by architecture workgroup
• User stories are accepted by product owner

There are multiple services with a copy & paste for Flow CRD definition and creation in k8s.

Flow operator

CRD definition and CRD creation.

Scheduler

CRD definition and CRD creation.

Communication Router

CRD definition and CRD creation.

Integration Component Repository (CR) requires multi-tenancy support. As in the conceptional elaborations described, the Docker images referenced by the CR can either be on Dockerhub or in a private Docker repository. In both cases, the user/tenant can provide credentials (access token, etc.) to fetch a component from a remote repository.
These credentials need to be stored by OIH in context of a tenant (or even referenced by a flow?).

As product owner I expect related technical user stories needed to implement the described features in epic

Acceptance Criteria:

• User stories are created
• User stories are reviewed by architecture workgroup
• User stories are accepted by product owner

As an OIH-Admin, I want to use different storages for the ICR so that i can install it on different platforms.

Further description:
Right now, the OIH-ICR only works with MongoDB. Right now the OIH-Platform uses a custom Kubernetes resource for flow-storage.

Acceptance Criteria:

• Build abstraction layer for flow-storage
• Flows can be stored as Kubernetes custom resource

Starting the openintegrationhub instance on the kubernetes cluster results in crashing bootloops of the container "scheduler"

Probably false secretKeyRefs, the containers cant send any "isAlive" responses based on a incomplete config file

Further description:
Error message in K8: Pod errors: "0 of 1 updated replica available - CrashLoopBackOff"

Acceptance Criteria:

• Fixed Error
• Published on Github

As a User I want to use my credentials from the IAM so that i can controll the access to the flows.

Further description:
ICR has to be connected to the OIH-Heimdal-Service for user management.

Acceptance Criteria:

• User gets credentials from OIH-IAM
• Only logged in users can access flows
• User can be part of an organization

IAM Service Accounts should have a scoped access, e.g. read-only access to all tenants for a token introspection. See IAM utils for usage scenario.

• Service Accounts can be created
• Service Accounts have scoped permissions

Currently to to apply changes to a flow you need to delete and then recreate flow.

As a product owner I want an initially filled backlog to have a basis for working in an agile manner.

To start working with a shared backlog and to create an updated project/development plan, it is necessary to initially fill the backlog.

Acceptance Criteria:

• All known issues have been created including a description and acceptance criteria

As an OIH dev-op, I need an documentation of the IAM service, so that it is possible to create the secrets for the service

Further description:
There are some information fundamentally important for creating the secret - which are missing currently - for example, the specific name of the required file: "keystore.json"

Acceptance Criteria:

• All fundamentally important information for creating the secret are given
• Reviewed by the responsible dev-op and quality manager

This is not critical yet as we can work with open Docker images in the beginning but this becomes critical for private components

We as OIH partners need some DevOps guideline to have a same understanding of how service are setup, build and deployed

Acceptance Criteria:

• reviewed dokument as a Guideline for all Partners

As oih operator I want the oih services to interact with each other so that I can properly operate the system.

Further description:

Currently the scheduler fetches flows from K8s CRD. (See FlowsDao.js)

• Receive Flows from ICR instead of K8s CRD

• Flows are received from ICR

Definition of Done

When the scheduler schedules a trigger execution, it does not know if the job is already ready to run (e.g. if the queues are created by the operator). Due to this race condition it may happen that the 1st scheduler tick is lost

I, as an OIH user,
want to be able to store and manage my keys securely,
so that these are only accessible by my (or my organization) integration flows.

I also want to store other credentials, e.g. tokens for external Docker registries.

The email component currently lies within the elastic.io repository on dockerhub.

All components should be open sourced under open integration hub.

Acceptance Criteria:

• Email component is moved to openintegrationhub repository on dockerhub

The OIH might make heavy use of Kubernetes API to create and manage containers & resources.
Although currently Kubernetes provides a lot of advantages, we should consider abstracting the access to the underlying infrastructure/orchestrator e.g. using a facade. The aim is to allow others to use an older version of Kubernetes if there are breaking API changes or even use other orchestrators entirely.

The flow operator requires privileges to create queues and exchanges in the virtual host. These component needs to read and push data only

Each service should ideally provide it's own kubernetes configuration for deployment.yaml and service.yaml in a subdirectory.

Affected services:

• communication-router
• flows-operator
• integration-content-repository
• scheduler
• crud monitoring

We should also have a consistent naming and folder structure for the infrastructure, e.g.:

• Kubernetes definitions should always be in a folder named k8s whithin the project folder
• Use a consistent naming pattern: deployment.yaml and service.yaml for the definitions
• Ingress configuration should be place in a different repository (e.g. OIH-Infrastructure)
• every Microservice need to have a checkable endpoint to have a liveliness and readiness check for the K8s

As a User, I want to manage the flows of my organization so that i have control of them.

Further description:
Right now you only can manage flows of a specific user. This has to be expanded to tenants.

Acceptance Criteria:

• Manage flows from the organization of the user
• User cannot access flows from other organizations
• Admin can access all flows

Starting the openintegrationhub instance on the kubernetes cluster results in crashing bootloops of the container "flows-operator"

Probably false secretKeyRefs, the containers cant send any "isAlive" responses based on a incomplete config file

Further description:
Error message in K8: Pod errors: "0 of 1 updated replica available - CrashLoopBackOff"

Acceptance Criteria:

• Fixed Error
• Published on Github

As a User, I want to see the API in documenation in the OpenApi-Format, so that all documentations are in the same format.

Further description:
Just transfer the API-Docs from Swagger to OpenAPI-Format

Acceptance Criteria:

• API-Docs are described as OpenAPI-Resource

If component fails and k8s failed to start it within retries defined by standard restart policy, than component will be broken, and the whole flow will be broken

• Publish events for other services, e.g. through the Notification Bus for consumers like Reports & Analytics
• Monitor the CRUD operations in Data Hub / SDF

Imagine you created a new version of your flow definition which you want to deploy via kubctl, REST API or a frontent. It may happen that the flow operator will not notice the update at all. This can be easily reproduced by executing the following command for an existing flow.

kubctl delete flow $FLOW_ID && kubectl create flow -f $FLOW_JSON_FILE

The changes will not be applied to the flow as the flow operator is not detecting the changes on existing Flow CRDs. Perfectly the flow operator checks the metadata.resourceVersion of the CDRs to detect changes.

Please check the paragraph Efficient detection of changes in the k8s API Concepts document. Here is some extract:

To enable clients to build a model of the current state of a cluster, all Kubernetes object resource types are required to support consistent lists and an incremental change notification feed called a watch. Every Kubernetes object has a resourceVersion field representing the version of that resource as stored in the underlying database. When retrieving a collection of resources (either namespace or cluster scoped), the response from the server will contain a resourceVersion value that can be used to initiate a watch against the server. The server will return all changes (creates, deletes, and updates) that occur after the supplied resourceVersion. This allows a client to fetch the current state and then watch for changes without missing any updates. If the client watch is disconnected they can restart a new watch from the last returned resourceVersion, or perform a new collection request and begin again.

ISV applications are connected using a hub and spoke model. Each application is only connected to the OIH without knowledge of other connected applications. As decided in the architecture workgroup the Smart Data Framework is needed to achieve this.

Acceptance Criteria:

• Concept for Smart Data Framework
• Basic implementation of Smart Data Framework
• Decoupling and encapsulation of all components involved in a data synchronization use case
• SDF adapters for inbound and outbound communication
• Migrate existing connectors to openintegrationhub platform (poc_3_mvp)

Starting the openintegrationhub instance on the kubernetes cluster results in crashing bootloops of the container "communication-router"

Probably false secretKeyRefs, the containers cant send any "isAlive" responses based on a incomplete config file

Further description:
Error message in K8: Pod errors: "0 of 1 updated replica available - CrashLoopBackOff"

Acceptance Criteria:

• Fixed Error
• Published on Github

As an OIH operator I want a properly documented platform to understand how the platform can be deployed / operated.

Acceptance Criteria:

• Documentation of the current platform (MVP) is expanded
• New documentation is reviewed and accepted by microservice & architecture workgroups

As product owner I expect related technical user stories needed to implement the described features in epic

Acceptance Criteria:

• User stories are created
• User stories are reviewed by architecture workgroup
• User stories are accepted by product owner

The timer component currently lies within the elastic.io repository on dockerhub.

All components should be open sourced under open integration hub.

Acceptance Criteria:

• Timer component is moved to openintegrationhub repository on dockerhub

Provide a method or configurable middleware, which introspects the client token. This can be accomplished with a request to IAM API, either with the client token itself, or using a service account.

As a User, I want to manage the flows so that i can re-use them everytime.

Further description:
By now the definition of a flow is only a show case. It needs further specification

Acceptance Criteria:

• Expand flow-definition with node, edges and relationship as in the elastic.io-example
• Expand API-functions so they can deal with whole flow-definition

I as a OIH user can define the scope of a secret in order to limit who can access this secret.

This allows me to to either share a secret with my group/organization or to limit the access only to myself, e.g. if it is a private application containing my username and password as plaintext.

As a OIH product owner, I want the memwatch in the IAM component to be deleted

Acceptance Criteria:

• Deleted memwatch-module