There are two kind of market: Public Market and Private Market

• Users view apps in public market default, view apps in private market by switch market.
• Developer create private market, publish apps to the market, share the market to specific users.

We need a migration tool for initial and migrate database [db-ctrl].
This is a vote issue for choose tools.

Optional:
https://github.com/mattes/migrate

Develop app workflow:

1. Create a new app

• Click create button
• Create or select a repo (An empty repo storage without index.yaml is able to be registered to OpenPitrix)
• Upload app config package, validate automatically in the background. The package will be uploaded to the repo automatically and generate the first version of the app.
• Developer can download the package.

2. Create a new app version in an exist app.

• Show the repo which cannot be changed.
• List all the exist app version on the left side.
• Upload app config package, generate new version when validate succeeded.

Publish app workflow:

app.status: draft/active/suspended/deleted
app_version.status: draft/submitted/passed/rejected/active/suspended/deleted

• If an app_version's status is active, the app is active

active means the app is published to public or private market.

• If publish app version to public market need admin to review, else if publish app version to private market, the app version's status change to active immediately.

question: Should we make runtime plugin as a separate project or just bundle the code with openpitrix together?

When calling CreateCluster Api, params of k8s/helm plugin is different from that of qingcloud plugin
qingcloud: createCluster(app_id, app_version, appConf, routerConf, vxnetConf, eipConf)
k8s/helm: createCluster(app_id, app_version, appConf)

Maybe we could put routerConf, vxnetConf,eipConf into appConf.

Different runtime env may has different api
qingcloud: DescribeRouter, DescribeVxnet, DescribeEip, ResizeCluster, AddClusterNodes.
k8s/helm: without above-mentioned api

Maybe front end has different gui.

Be able to upload app package to repo through http/https/s3 protocol.

1. Scan runtime plugins and provide runtime basic info to end user. For instance, if OpenPitrix supports QingCloud, then runtime env manager (or by operator) registers the url of QingCloud api server.
2. End user needs to configure runtime before deploying app.
3. For system registered runtime, for instance, QingCloud runtime, end user needs to configure her/his credentials to access the api server.
4. End user can create her/his own runtime. For instance, she/he can create a k8s cluster on QingCloud, then create such runtime in OpenPitrix for her/his to use.
5. The manager is responsible to create/update(configure)/delete/read runtimes.

This system is an independent project which is used by other systems such as OpenPitrix, QingCloud AppCenter, or end user's applications that are running on VMs of any cloud platform.
For OpenPitrix, the logging system would satisfy the following requirements.

1. Collect health status of applications into OpenPitrix system
2. Collect monitoring info of applications into OpenPitrix system
3. Collect application logs into OpenPitrix system

The first requirement must be provided for the first OpenPitrix release.

"Jenkins with Blue Ocean plugin and deploy it on Kubernetes, also continuously deploy OpenPitrix on the Kubernetes cluster."
Can you please provide YAML file for above?

Advanced APIs for cluster management:

ResizeCluster
AddClusterNodes
DeleteClusterNodes

To deploy vm-based application into cloud platform such as QingCloud, AWS, OpenStack etc, we need to deploy two kinds of vm-based clusters.

• metadata cluster with metad (etcd as backend store) installed
• application cluster with confd installed

Where metadata cluster provides the meta data service for user's applications. Confd is the auto configuration daemon running in the application cluster instances and updates application configurations based on the info from metadata service.

Things need to design

1. Where to deploy metadata cluster? Is it per cloud? or per user? or per cloud and per user?
2. How OpenPitrix system, metad and confd. communicate?
3. Multi-tenancy support

There are two possible solutions for the communications among components.

1. Wrap confd and metad into a REST-based service so they can send requests back and forth. For security, the rest service requires certificate to authorise requests.
2. Generate ssh key for OpenPitrix runtime subsystem, create metadata vm and application cluster vm with the key pair, so the runtime can execute cmd through ssh without password.

A case: Create application cluster steps:

1. User starts to deploy an application.
2. Runtime service first checks if metadata service created or not, if not, creates metadata service first in the background.
3. Runtime service creates application cluster, starts confd daemon on each instance.
4. Runtime service registers the application cluster info to metadata service.
5. Confd in each cluster instance watches the changes of the metadata info and starts to refresh its configuration, and exec reload cmd if appropriate.
6. Runtime service registers application init and start cmd (the cmd to make application working) into metadata service. The cluster then executes the cmd from the metadata service. After everything is finished successfully, OpenPitrix transforms the application cluster to the status "active".

1. Setup dev env through docker compose
2. Access http://127.0.0.1:9100/swagger-ui
3. At the bottom of the page, it prompt error section, click that link and get msg as:
   "Can't read from file /sw…ger-ui/app.swagger.json"

The system supports multiple OpenPitrix deployment in hierarchy organization. Typically there are two cases:

1. The parent OpenPitrix is able to add existing one as its child.
2. The child is able to register itself to an existing one as its parent.

More details as follows.

1. Create client certificates in system B/C/D
2. Add system D as system B's sub system through system D's api-gateway address and client certificates.
3. Add system B/C as system A's sub system through system B/C's api-gateway address and client certificates.

• Only authorized user can switch to sub system.
• Admin can sync sub system's user/repo/app to current system.

'limit' attr will not take effect in 'select count(*)' operation
https://bugs.mysql.com/bug.php?id=50005

In the first version, we will use helm to manage k8s applications. So we need to figure out how to deploy helm. Helm has client (helm) and server (tiller). The server can be deployed outside of target k8s cluster. Therefore, one possible solution is package client and server into the OpenPitrix k8s runtime plugin and make the server (tiller) be able to connect multiple k8s cluster. Using the following way to differentiate target cluster.

$ KUBECONFIG=/whatever/.kube/config helm install --name myapp <app chart>

Here is the deployment diagram. We need copy the kube config used by kubectl from k8s cluster to the plugin where helm is installed. We also need to set up VPN to access k8s clusters from the plugin node.

New diagram:

link

make build

Error when performing this step:
...
@docker build -t openpitrix/openpitrix:metadata -f ./Dockerfile.metadata .
...

And the error message:
Step 1/7 : FROM openpitrix/openpitrix-builder as builder
Error parsing reference: "openpitrix/openpitrix-builder as builder" is not a valid repository/tag: invalid reference format
make: *** [build] Error 1

My docker version:

docker --version

Docker version 1.13.1, build 774336d/1.13.1

to replace swagger-go plugin

Deployments, Statefulsets, Services should be shown, together with the pods.

1. In the first release, we only provide the functionalities for end user. The developer functionalities will be provided in the second release. Here release means minor release. We use semantic version v 2.0 to manage versioning.

2. End user will be able to view apps and deploy any one she/he is interested in to selected runtime.

3. All apps info in app database is from repo storages which are indexed by repo database and accessed via repo service (api).

4. One sidecar daemon service bundled with app service will periodically check repo storage to get the updated apps info and update app database correspondingly.

5. End user (actually should be operator) is able to trigger the apps synchronisation manually.

A ticket system is needed, so that users can submit ticket to app developers when they have problems about their clusters.

Before the feature is provided, users could connect to app developers through emails.

• The document of gorm is clear.
• Gorm will do auto migration when the structure of table is changed.
• Base model definition in gorm.Model, including fields ID, CreatedAt, UpdatedAt, DeletedAt, you could embed it in your model, or only write those fields you want.
• Gorm use chainable API, *gorm.DB is the bridge of chains, for each chain API, it will create a new relation.

Selector and label are needed when create repo, label is needed when create runtime.
So a label manager is needed.

label and selector are all key value structure, like: env=staging

APIs:

• CreateLabel
• AttachLabels
• DetachLabels
• DeleteLabels
• ModifyLabel
• DescribeLabels

These handler can be implemented in runtime manager temporary.

How to create repository if run time provider is kubernetes ?
What can be the steps to achieve this.
how to resolve the error "response timeout of 5000ms exceeded"?
can you please provide any demo of dashboard operations.

There are two types of application images when developing traditional applications on OpenPitrix: virtual machine and docker. The app spec is proposed as follows.

Example of vm defined in cluster.json.mustache

      "container": {
            "type": "vm",
            "image": "img-abcdefgh",
            "zone": "pek3a",
            "credential": "crdl-abcdefgh"
        },

When type is vm, zone and credential are required.

• zone: the zone where the image exists.
• credential: the developer's credential of runtime_env, developer is the owner of the image.

If the app_version is developped by developer A, but the cluster is created by user B, OpenPitrix will migrate the image from zone defined in developer's cluster.json.mustache to user's runtime_env, and then share the image with user B, then call the api CreateCluster.

Tips and tricks

• When create cluster, take the user runtime_env's api_server and runtime as the standard.
• Migrate and share image may occur when user creates cluster.
• OpenPitrix records the migration and share info in db, if the developer migrates or shares image in runtime_env, OpenPitrix does not know. It does not matter because the APIs of the image migration and sharing are reentrant.

Example of docker defined in cluster.json.mustache

        "container": {
            "type": "docker",
            "image": "gcr.io/google_samples/k8szk:v2",
            "credential": "crdl-abcdefgh"
        },

When type is docker, credential is optional, which is used to pull image.

when task-manager create frontgate instance, the task-manager must set the pilot client tls config for frontgate. then the frontgate use the tls config to dial the pilot grpc service.

I prepare to Build the dev env in Centos7 from the step by development.md。
When I exec 'make build', it will show error as below:

real 0m 1.95s
user 0m 4.59s
sys 0m 1.12s
docker build -t openpitrix -t openpitrix:metadata -f ./Dockerfile.dev ./tmp/bin
unable to prepare context: The Dockerfile (/home/oppitrix/openpitrix/Dockerfile.dev) must be within the build context (./tmp/bin)

I'm the new of docker, I dont how to change the dockerfile for fix the problem, can somebody help?

Basic api

CreateCluster
DeleteClusters
UpgradeCluster
RollbackCluster
ResizeCluster
AddClusterNodes
DeleteClusterNodes
UpdateClusterEnv
DescribeClusters
DescribeClusterNodes
StopClusters
StartClusters
RecoverClusters
CeaseClusters

Details

"..." means any optional params

• CreateCluster
params:

  • appId: app id
  • appVersion: app version
  • conf: conf entered by user
  • advanced_params: optional json params, such as env

  response:

  • clusterId: the new create cluster id
  • jobId: the job id

• DeleteClusters
params:

  • clusterIds: the cluster ids who need to be deleted
  • advanced_params: optional json params, such as force
    response:
  • clusterIds: the to be deleted cluster ids
  • jobIds: the job ids

• UpgradeCluster
params:

  • clusterId: the cluster id who need to be upgraded
  • appVersion: the new version
  • advanced_params: optional params
    response:
  • clusterId: the new create cluster id
  • jobId: the job id

• RollbackClusters
params:

  • clusterId: the cluster id who need to be rollback
  • advanced_params: optional params
    response:
  • clusterIds: the to be rollback cluster ids
  • jobIds: the job ids

• ResizeCluster
params:

  • clusterId: the cluster id who need to be resize
  • role: the cluster role
  • cpu: the cpu resource
  • memory: the memory resource
  • advanced_params: optional params, such as gpu
    response:
  • clusterId: the resize cluster id
  • jobId: the job id

• AddClusterNodes
params:

  • clusterId: the cluster id who need to be scale
  • role: the cluster role
  • nodes: the role replica num
  • advanced_params: optional params
    response:
  • clusterId: the add nodes cluster id
  • jobId: the job id

• DeleteClusterNodes
params:

  • clusterId: the cluster id who need to be scale
  • role: the cluster role
  • nodes: the role node ids or role replica num
  • advanced_params: optional params
    response:
  • clusterId: the delete nodes cluster id
  • jobId: the job id

• UpdateClusterEnv
params:

  • clusterId: the cluster id who need to be update env
  • env: values changed from default.
  • advanced_params: optional params
    response:
  • clusterId: the update env cluster id
  • jobId: the job id

• DescribeClusters
params:

  • clusterIds: the cluster ids who need to be describe
  • advanced_params: optional params

• DescribeClusterNodes
params:

  • clusterNodeIds: the cluster node ids who need to be describe
  • advanced_params: optional params

• StopClusters
params:

  • clusterIds: the cluster ids who need to be stoped
  • advanced_params: optional params, such as force
    response:
  • clusterIds: the to be stopped cluster ids
  • jobIds: the job ids

• StartClusters
params:

  • clusterIds: the cluster ids who need to be started
  • advanced_params: optional params
    response:
  • clusterIds: the to be started cluster ids
  • jobIds: the job ids

• RecoverClusters
params:

  • clusterIds: the cluster ids who need to be recovered
  • advanced_params: optional params
    response:
  • clusterIds: the to be recovered cluster ids
  • jobIds: the job ids

• CeaseClusters
params:

  • clusterIds: the cluster ids who need to be ceased
  • advanced_params: optional params
    response:
  • clusterIds: the to be ceased cluster ids
  • jobIds: the job ids

Helm runtime Details

bold means required params, italic means optional params

• CreateCluster: use helm api: InstallRelease to implement
• DeleteClusters: use helm api: DeleteRelease to implement
• UpgradeCluster: use helm api: UpgradeRelease to implement
• RollbackClusters: use helm api: RollbackRelease to implement

  RollbackClusters can rollback all UpgradeRelease operation, such as ResizeCluster, ScaleCluster, UpgradeCluster and UpdateClusterEnv
  revision: the revision that need to be rollback(optional), default last operate revision

• ResizeCluster: use helm api: UpgradeRelease to implement

  If there are resources(may be other key) defined in Values.yaml, then this function is supported, should check from template to ensure which deployment has defined resources
  Here should change resource limit automatically.

• AddClusterNodes: use helm api: UpgradeRelease to implement

  If there are replicas(may be other key) defined in Values.yaml, then this function is supported, should check from template to ensure which deployment has defined replicas

• DeleteClusterNodes: use helm api: UpgradeRelease to implement

  If there are replicas(may be other key) defined in Values.yaml, then this function is supported, should check from template to ensure which deployment has defined replicas

• UpdateClusterEnv: use helm api: UpgradeRelease to implement

  Shows other values defined in Values.yaml except from resources and replicas.

• DescribeClusters: use kubernetes api to implement
• DescribeClusterNodes: use kubernetes api to implement
• StopClusters: use helm api: DeleteRelease to implement
• StartClusters: use helm api: InstallRelease to implement
• RecoverClusters: use helm api: InstallRelease to implement
• CeaseClusters: Update status to ceased and not able to recover.

Use travis to build image and push.

for reference;
https://docs.travis-ci.com/user/docker/#pushing-a-docker-image-to-a-registry
https://docs.travis-ci.com/user/environment-variables#Encrypting-environment-variables

Feature

1. A kind of built-in repo. App package, icon, screenshot can be uploaded to the storage.
2. The repo is created auto, visibility is public, so every developer can upload there package to the repo when create app while describe there own app only.
3. When a app version is published to public market, the app package will be copied to the built-in repo.

Db

CREATE TABLE IF NOT EXISTS attachment (
  attachment_id  VARCHAR(50)  NOT NULL,
  filename       VARCHAR(255) NOT NULL DEFAULT '',
  type           VARCHAR(50)  NOT NULL DEFAULT 'raw',
  owner          VARCHAR(50)  NOT NULL,
  digest         VARCHAR(255) NOT NULL DEFAULT '',
  create_time    TIMESTAMP    NOT NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (attachment_id)
)
CREATE INDEX attachment_attachment_id_idx
  ON attachment (attachment_id);
CREATE INDEX attachment_type_idx
  ON attachment (type);
CREATE INDEX attachment_owner_idx
  ON attachment (owner);
CREATE INDEX attachment_digest_idx
  ON attachment (digest);
CREATE INDEX attachment_create_time_idx
  ON attachment (create_time);

CREATE TABLE IF NOT EXISTS attachment_content (
  attachment_id  VARCHAR(50)  NOT NULL,
  content_key    VARCHAR(255) NOT NULL DEFAULT '',
  content        MEDIUMBLOB   NOT NULL,
  PRIMARY KEY (attachment_id, content_key)
)

CREATE INDEX attachment_content_attachment_id_idx
  ON attachment_content (attachment_id);
  
CREATE INDEX attachment_content_content_key_idx
  ON attachment_content (content_key);

Api

message GetAttachmentsRequest {
    repeated string attachment_id = 1;
    repeated string content_key = 2;
    bool ignore_content = 3;
}
message GetAttachmentsResponse {
    map<string, Attachment> attachments = 1; // attachment_id => Attachment
}
message Attachment {
    string attachment_id = 1;
    string filename = 2;
    string type = 3; // archive or raw
    map<string, bytes> attachment_content = 4; // name => content
    string digest = 5; // sha1 digest
    string owner = 6;
    google.protobuf.Timestamp create_time = 7;
}

message UploadAttachmentRequest {
    string filename = 1;
    string type = 2; // archive or raw
    bytes content = 3;
    string digest = 4; // sha1 digest
}
message UploadAttachmentResponse {
    string attachment_id = 1;
}

message CopyAttachmentsRequest {
    repeated string attachment_id = 1;
}
message CopyAttachmentsResponse {
    map<string, string> copy_attachment_map = 1; // src_id => dst_id
}

message DeleteAttachmentsRequest {
    repeated string attachment_id = 1;
}
message DeleteAttachmentsResponse {
    repeated string attachment_id = 1;
}

service AttachmentManager {
    rpc UploadAttachment(UploadAttachmentRequest) returns (UploadAttachmentResponse);
    rpc GetAttachments(GetAttachmentsRequest) returns (GetAttachmentsResponse);
    rpc CopyAttachments(CopyAttachmentsRequest) returns (CopyAttachmentsResponse);
    rpc DeleteAttachments(DeleteAttachmentsRequest) returns (DeleteAttachmentsResponse);
}

Now we only support these three types of repo:http/https/s3.
Only s3 repo can be read/write.

To make it easier for developers to use, we need to support more types of repo. Here is some plan:

• WebDAV
• SFTP
• FTP

APIs in cluster service:

UpgradeCluster
RollbackCluster

should be implemented.

There are three kind of roles: admin, developer, normal user.

Admin exist when the OpenPitrix is deployed, we can use system as the default.

Admin can create normal user and developer.

Admin can change user role from normal user to developer.

Admin can describe all users' apps, runtimes, repos and clusters.

Developer can describe his apps, runtimes, repos and clusters, also can call other CURD apis. He can also describe all the clusters deploy from the his apps.

Normal user can only describe his runtimes and clusters, also can call other CURD apis.

app service should have performance metrics to help monitoring tools collect performance data.
Available tools are as below:

http://opencensus.io/

make compose-up

Error message：

...
Creating openpitrix-repo-manager    ... error
Creating openpitrix-runtime-manager ... 
Creating openpitrix-app-manager     ... 
Creating openpitrix-cluster-manager ... 
Creating openpitrix-runtime-manager ... error
Creating openpitrix-job-manager     ... 

ERROR: for openpitrix-repo-manager  Cannot start service openpitrix-repo-manager: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"repo-manager\": executable file not found in $PATH": unknown
Creating openpitrix-app-manager     ... error
ERROR: for openpitrix-runtime-manager  Cannot start service openpitrix-runtime-manager: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"runtime-manager\": executable file not found in $PATH": unknown
Creating openpitrix-cluster-manager ... error
ERROR: for openpitrix-app-manager  Cannot start service openpitrix-app-manager: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"app-manager\": executable file not found in $PATH": unknown

ERROR: for openpitrix-cluster-manager  Cannot start service openpitrix-cluster-manager: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"cluster-manager\": executable file not found in $PATH": unkn
Creating openpitrix-job-manager     ... error

Creating openpitrix-task-manager    ... errorx.go:348: starting container process caused "exec: \"job-manager\": executable file not found in $PATH": unknown

ERROR: for openpitrix-task-manager  Cannot start service openpitrix-task-manager: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"task-manager\": executable file not found in $PATH": unknown

ERROR: for openpitrix-repo-manager  Cannot start service openpitrix-repo-manager: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"repo-manager\": executable file not found in $PATH": unknown

ERROR: for openpitrix-runtime-manager  Cannot start service openpitrix-runtime-manager: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"runtime-manager\": executable file not found in $PATH": unknown

ERROR: for openpitrix-app-manager  Cannot start service openpitrix-app-manager: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"app-manager\": executable file not found in $PATH": unknown

ERROR: for openpitrix-cluster-manager  Cannot start service openpitrix-cluster-manager: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"cluster-manager\": executable file not found in $PATH": unknown

ERROR: for openpitrix-job-manager  Cannot start service openpitrix-job-manager: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"job-manager\": executable file not found in $PATH": unknown

ERROR: for openpitrix-task-manager  Cannot start service openpitrix-task-manager: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"task-manager\": executable file not found in $PATH": unknown
ERROR: Encountered errors while bringing up the project.
make: *** [compose-up] Error 1

# docker --version
Docker version 18.03.1-ce, build 9ee9f40

# docker-compose --version
docker-compose version 1.21.0, build 5920eb0

Do I need any extra configuration?

The repo component consists of three parts: repo database, repo storage and repo service. The user of repo communicates through repo service which is the micro service of OpenPitrix. The repo service manages the lifecycle of repos such as create a repo, update a repo etc. The repo table stores the metedata of each repo. Please check the database design. The repos can be from multiple sites. Actually repo storage is not part of the repo component. We can use any repo on the internet such as Helm repo https://kubernetes-charts.storage.googleapis.com.

New diagram:

link

some 3rd party tools in docker builer image is not stable(swager/grpc-gateway).
and the output is volatile when make build.

we should use the stable version for any build tools.

How can we register runtime (Cloud Provider) OpenPitrix?
what are the steps or process to complete the above statement?
Is there any source code available, from which we can take reference for the same?

Describe in https://github.com/openpitrix/openpitrix/blob/master/docs/devops.md.
Scripts in https://github.com/openpitrix/openpitrix/tree/master/deploy/devops/scripts should be finished.

Currently OpenPitrix supports Kubernetes based container orchestration, QingCloud, AWS. Now we need to support more cloud provider.

• Aliyun
• VMware
• OpenStack
• Azure
  ...