Giter VIP home page Giter VIP logo

bioos's Introduction

About Bio-OS

Bio-OS is an open-source platform for genomics users. It provides a workspace which encapsulates data, workflows, Notebooks, job history, etc. Bio-OS provides both GUI and CLI(Command Line Interface) to quickly setup and run workflows specified by languages like WDL and notebook as well.

Bio-OS Concept


Bio-OS Workspace

Workspace is a complete encapsulation of a bioinformatics research process, including data, environment, code, operational calculation procedures, results, and dashboard as an overview. It is the basic unit that realizes executable, transportable, reproducible, shareable and publishable scientific research and biological application.


Architecture :

In Bio-OS, there are three parts in the architecture.

Control layer:

  1. Bioos-web: the front-end component of Bio-OS.
  2. Bioctl is the command-line component of Bio-OS. Users can access Bio-OS through bioctl.
  3. OpenAPI is used to describe, produce, consume, and visualize RESTful web services. It's a specification standard for REST APIs that defines the structure and syntax.
  4. Bioos-apiserver: It mainly realizes the management of workspace, workflow (conforming to WES specification), data, etc. The bioos-server will send requests to the clusters of each computing plane to implement the running workflow
  5. IAM(not supported in Version 1.0.0): service that helps you securely control user access

Workflow Layer

  1. Cromwell: open-source cromwell version v85
  2. Storage : store workflow input ,output data or log

Notebook Layer

  1. JupyterHub: a multi-user Hub that spawns, manages, and proxies multiple instances of the single-user Jupyter notebook server.
  2. Storage : Used to store data from jupyter notebook

Capabilities

There are four core capabilities which support Bio-OS.

  • Data Management

    Effectively organize massive scale sample data and easily access public data

  • Application Management

    Bring together data、workflow、Notebooks、work engine into Sharable、 Traceable and reproducible Workspace

  • Resource Management

    Autoscaling through cloud-native which supports hybrid scheduling of multi-cloud heterogeneous resources

  • Interactive Analysis

    Using Built-in Notebook to do tertiary analysis

Getting Started

Prerequisites

To run Bio-OS, the following prerequisites must be met:

  • A computer with one of the following operating systems:
    • CentOS 7.9 +
    • Ubuntu 22.04 +
  • Internet access
  • Resource Required

Minimum 8 cpu and 16G memory

Tutorial

Bio-OS Installation guide

Please look at the Bio-OS installation guide: https://bio-os.gitbook.io/userguide/bu-shu/getting-set-up

Bio-OS user guide

Please look at the Bio-OS user guide: https://bio-os.gitbook.io/userguide/

Bio-OS CLI

Please look at the CLI tutorial: https://bio-os.gitbook.io/userguide/chan-pin-zhi-nan/ming-ling-hang-gong-ju-bioctl

Help

Please read the documentation and search for existing issues that might help before you ask a question. It is also advisable to search the internet for answers first.

If you then still feel the need to ask a question and need clarification, we recommend the following:

  • Open an issue.
  • Provide as much context as you can about what you're running into.
  • Provide project and platform versions, depending on what seems relevant.

We will handle the issue carefully as soon as possible.

License

This project is licensed under the Apache-2.0 License.

bioos's People

Contributors

2uschauer avatar ealyn avatar fuxiaoyi avatar howardshaw avatar wencaiwulue avatar yifanchen90 avatar yuanminhui avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

bioos's Issues

任务挑战赛-【赛题解读】T1:Notebook插件可视化工具集成

背景介绍

Bio-OS开源创新大赛分为两种类型,分别为开发类和工具类,其中工具类赛题主要面向生信背景,对于本赛题来说,熟悉生信常用软件和工具,擅长python和R语言,可以尝试本赛题。好的工具除获得奖励之外还会被Bio-OS开源项目收录,为所有生信用户提供jupyter notebook中可以直接使用的交互式生信工具。

任务介绍

该赛题要求参赛团队基于Bio-OS提供的开源Notebook插件通过编写生信可视化工具的源代码及对应的工具所需要的参数文件,实现在jupyter Notebook插件中集成一个生信分析的工具。可视化工具需要是生物信息分析常用的工具类型,如火山图、韦恩图、circos图、聚类图等等,您也可以将您平时使用的任意工具集成进BioMate插件中。

提交内容

请将以下提交内容发送到邮箱: [email protected] 1. 一个名称为"任务挑战-T1-队伍名称"的md或word文件 a. 对工具的自由介绍,内容包括工具介绍,参数配置介绍,使用方式介绍 b. 包含一个视频链接(如上传到B站或其他网站,附上链接,如视频不大也可以直接发送邮箱) 2. 将源代码提交到https://github.com/Bio-OS/bio-mate中 多个工具需对应bio_mate/plot-defs中多个目录,每个目录中应包含以下 - 工具源代码 - 工具的页面组件ui.json文件 - 工具的输入参数配置input.json文件 - 工具的示例数据 - 工具的示例图片 - 工具的元数据文件metadata

评分标准

1. 难度基础分(权重10%)
- T1:Notebook插件可视化工具集成 --难度基础分6分
- T2:实现工作流运行任务监控 -- 难度基础分8分
- T3:命令行工具实现Bio-OS Notebook使用 -- 难度基础分8分
- T4:Bio-OS流程规范扩展 -- 难度基础分 10分
2. 工具数量:工具的数量,数量越多,评分越高,超过4个工具后此项可得满分 (权重20%)
3. Demo效果:DEMO效果展示,Demo视频中可尽量展示完整的功能以及监控效果。(权重20%)
4. 代码实现:通过提交的源代码进行code review,将从代码质量、可维护性、可读性多个维度进行评分。(权重30%)
- 代码质量:是否正确的实现了预期功能和需求,是否处理了预期的错误情况并提供了清晰地错误消息和处理方式,是否考虑可配置参数
- 可维护性:是否易于修改和维护,包括代码的模块化程度、复用性,是否易于扩展
- 可读性:代码是否易于阅读和理解,包括变量名、函数名、注释等命名是否清晰,代码结构是否合理
5. 文档编写(20%)
- 内容准确性:是否准确、完整、清晰的表达了所需信息
- 文档可读性:文档的结构是否合理,逻辑清晰,是否易于阅读和理解,是否使用了恰当的图表和图片等

Add task hit cache icon

Describe the solution you'd like
Submission have a function name CallCaching, but in the analysis page we can not distinguish which task is hit cache and which is actually run, so I hope there can be an icon to distinguish these two tasks well

说明文档需要改进啊啊啊啊啊!!!!!!!

Bio- OS的部署文档,讲到Cromwell安装 那里,提到了两种安装方式。

第一种通过helm安装,之后测试示例的CramToBam流程 就无法通过,该流程需要cromwell调用docker,但Cromwell的容器中没有docker。

那好,然后更换第二种jar包安装。但文档中只给出了jar包的配置文件,另外一个非常重要的东西,bioos-storage-pvc需要另外手工配置,然而文档里没有说明!希望文档中,按我下面回复的意思,把这段补充完整吧!

docs: bug fix at deployment userguide

Bug/PR list

There are several mistakes in the deployment userguide provided at https://bio-os.gitbook.io/userguide/bu-shu/getting-set-up/bu-shu-bioos and https://github.com/Bio-OS/helm-charts/blob/main/README.md, which will lead to small error/warnings during deployment. Since the documents hosted at Gitbook is not in integration with any public repository, I list the mistakes and fixs bellow:

1 2 3 4

Hope you fix these and merge my later PR at repo Bio-OS/helm-charts which contains more improvements with this doc.

Build apiserver failed at step 6

Describe the bug
Use docker compose to build bioos service always fail at [builder] 6/6, though seems caused by external network problem.
(objects.githubusercontent.com is observed much of this issue..?)

To Reproduce
Run docker compose up at the root of the project.

Expected behavior
A guide to move on.

Screenshots
docker-compose-apiserver-build-failed

More secure and concise stdout

Is your feature request related to a problem? Please describe.
When start an apiserver locally, it prints internal operations and states frequently to stdout. It seems dangerous for exposing so much information, most of which are unnecessary to users.

Describe the solution you'd like
Less unrelated printings, a clear and concise cli.

Screenshots
internal-log-issue

bug: toggle sidebar component flaw

Describe the bug

The toggle sidebar button couldn't wrap the items into compact counterparts, and leaves a annoying scroll bar.

To Reproduce

Click the toggle sidebar button in full-width mode.

Expected behavior

Icons or labels instead of full names in compact mode.

Screenshots

Weixin Image_20231121151332

bug: Fail to start jupyterhub during deployment

Describe the bug

Fail to start jupyterhub during deployment.

To Reproduce

Follow the deployment guide at https://bio-os.gitbook.io/userguide/bu-shu/getting-set-up/bu-shu-bioos or https://github.com/Bio-OS/helm-charts/blob/main/README.md.

$ helm install jupyterhub bioos/jupyterhub \
        --namespace bioos \
        --create-namespace \
        --set hub.db.url=mysql+pymysql://root:[email protected]:3306/bioos \
        --set hub.db.password=Bytedance2023
"bioos" has been added to your repositories
NAME: jupyterhub
LAST DEPLOYED: Sun Nov 19 10:37:46 2023
NAMESPACE: bioos
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
.      __                          __                  __  __          __
      / / __  __  ____    __  __  / /_  ___    _____  / / / / __  __  / /_
 __  / / / / / / / __ \  / / / / / __/ / _ \  / ___/ / /_/ / / / / / / __ \
/ /_/ / / /_/ / / /_/ / / /_/ / / /_  /  __/ / /    / __  / / /_/ / / /_/ /
\____/  \__,_/ / .___/  \__, /  \__/  \___/ /_/    /_/ /_/  \__,_/ /_.___/
              /_/      /____/

       You have successfully installed the official JupyterHub Helm chart!

### Installation info

  - Kubernetes namespace: bioos
  - Helm release name:    jupyterhub
  - Helm chart version:   2.0.0
  - JupyterHub version:   3.0.0
  - Hub pod packages:     See https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/2.0.0/images/hub/requirements.txt

Nothing seems wrong here, and:

$ kubectl -n bioos port-forward --address 0.0.0.0 service/hub 8081:8081
error: unable to forward port because pod is not running. Current status=Pending

The jupyterhub pod is not running. A long way debug started.


$ kubectl -n bioos get pods -o wide
NAME                   READY   STATUS             RESTARTS     AGE   IP           NODE           NOMINATED NODE   READINESS GATES
hub-5f57d5bd65-wlw6r   0/1     CrashLoopBackOff   6 (4m ago)   10m   10.244.1.3   minikube-m02   <none>           <none>
mysql-0                1/1     Running            0            47m   10.244.3.3   minikube-m04   <none>           <none>

$ kubectl -n bioos get svc -o wide
NAME                             TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE   SELECTOR
hub                              NodePort       10.255.159.107   <none>        8081:32450/TCP   34m   app=jupyterhub,component=hub,release=jupyterhub
jupyter--2fjupyterhub-2f-route   ExternalName   <none>           hub.bioos     8081/TCP         67m   <none>
mysql                            ClusterIP      10.255.80.216    <none>        3306/TCP         72m   app.kubernetes.io/component=primary,app.kubernetes.io/instance=mysql,app.kubernetes.io/name=mysql
mysql-headless                   ClusterIP      None             <none>        3306/TCP         72m   app.kubernetes.io/component=primary,app.kubernetes.io/instance=mysql,app.kubernetes.io/name=mysql

$ kubectl -n kube-system get pods,svc
NAME                                       READY   STATUS    RESTARTS         AGE
pod/coredns-7f74c56694-z87cm               1/1     Running   6 (3h40m ago)    25h
pod/csi-nfs-controller-74f4f8484-jmqnw     4/4     Running   0                3h25m
pod/csi-nfs-node-h97cs                     3/3     Running   0                3h25m
pod/csi-nfs-node-kg7j8                     3/3     Running   0                3h25m
pod/csi-nfs-node-rwbs4                     3/3     Running   0                3h25m
pod/csi-nfs-node-z4d89                     3/3     Running   0                3h25m
pod/etcd-minikube                          1/1     Running   2 (3h40m ago)    25h
pod/kube-apiserver-minikube                1/1     Running   2 (3h40m ago)    25h
pod/kube-controller-manager-minikube       1/1     Running   2 (3h40m ago)    25h
pod/kube-proxy-8dl6v                       1/1     Running   2 (3h33m ago)    25h
pod/kube-proxy-cx85g                       1/1     Running   2 (3h33m ago)    25h
pod/kube-proxy-nbvf4                       1/1     Running   2 (3h33m ago)    25h
pod/kube-proxy-sqcwc                       1/1     Running   2 (3h40m ago)    25h
pod/kube-scheduler-minikube                1/1     Running   2 (3h40m ago)    25h
pod/snapshot-controller-66746ffc86-r4w6k   1/1     Running   0                3h25m
pod/storage-provisioner                    1/1     Running   17 (3h40m ago)   25h

NAME               TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns   ClusterIP   10.255.0.10   <none>        53/UDP,53/TCP,9153/TCP   25h

$ kubectl -n ingress-nginx get pods,svc  -o wide
NAME                                            READY   STATUS              RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
pod/ingress-nginx-admission-create-z48gm        0/1     ImagePullBackOff    0          25h   10.244.0.8   minikube   <none>           <none>
pod/ingress-nginx-admission-patch-2mdgb         0/1     ImagePullBackOff    0          25h   10.244.0.9   minikube   <none>           <none>
pod/ingress-nginx-controller-684c54767f-gwtk9   0/1     ContainerCreating   0          25h   <none>       minikube   <none>           <none>

NAME                                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE   SELECTOR
service/ingress-nginx-controller             NodePort    10.255.48.183   <none>        80:30710/TCP,443:30659/TCP   25h   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
service/ingress-nginx-controller-admission   ClusterIP   10.255.61.48    <none>        443/TCP                      25h   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

$ kubectl get pods -n ingress-nginx
NAME                                        READY   STATUS              RESTARTS   AGE
ingress-nginx-admission-create-z48gm        0/1     ImagePullBackOff    0          26h
ingress-nginx-admission-patch-2mdgb         0/1     ErrImagePull        0          26h
ingress-nginx-controller-684c54767f-gwtk9   0/1     ContainerCreating   0          26h

To look deeper into the nginx pod:

$ kubectl describe pod ingress-nginx-admission-create-z48gm -n ingress-nginx
Name:         ingress-nginx-admission-create-z48gm
Namespace:    ingress-nginx
Priority:     0
Node:         minikube/192.168.49.2
Start Time:   Sat, 18 Nov 2023 09:51:46 +0800
Labels:       app.kubernetes.io/component=admission-webhook
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/name=ingress-nginx
              controller-uid=e2b83263-5092-4c77-8296-b777ed5d9705
              job-name=ingress-nginx-admission-create
Annotations:  <none>
Status:       Pending
IP:           10.244.0.8
IPs:
  IP:           10.244.0.8
Controlled By:  Job/ingress-nginx-admission-create
Containers:
  create:
    Container ID:
    Image:         registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Args:
      create
      --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
      --namespace=$(POD_NAMESPACE)
      --secret-name=ingress-nginx-admission
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wgpk6 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-wgpk6:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              kubernetes.io/os=linux
                             minikube.k8s.io/primary=true
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason   Age                    From     Message
  ----    ------   ----                   ----     -------
  Normal  BackOff  4s (x1095 over 4h10m)  kubelet  Back-off pulling image "registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80"

It seems the image is not pulled successfully. I tried to pull manully to validate:

$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80
Error response from daemon: manifest for registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80 not found: manifest unknown: manifest unknown

Manifest of the image signature failed. When I removed sha256 string:

$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20231011-8b53cabe0
v20231011-8b53cabe0: Pulling from google_containers/kube-webhook-certgen
07a64a71e011: Pulling fs layer
fe5ca62666f0: Pulling fs layer
b02a7525f878: Pulling fs layer
fcb6f6d2c998: Waiting
e8c73c638ae9: Waiting
1e3d9b7d1452: Waiting
4aa0ea1413d3: Waiting
7c881f9ab25e: Waiting
5627a970d25e: Waiting
2c4dd5b46232: Waiting

It works.

So the problem lies in image designation. A procedure is needed for manully recover the deployments. I will provide the fixed procedure in a PR so others can install this program successfully. Please check and merge it.

Expected behavior

Successful deployment of jupyterhub & bioos.

Screenshots

None.

Bio-OS Roadmap 2023

Welcome to share your ideas on the roadmap. The updated roadmap for Q3 and Q4 are shown below.

(will update later...)

Unable to access web api: 401 unauthorized

Describe the bug
Successfully starts an apiserver from build (swagger documents accessable), but functional endpoints are unreachable, returning a 401 error code. Some tokens or configs required?

To Reproduce
Start a bioos apiserver locally.

Expected behavior
Functional web service.

Screenshots
401-error-get-workspace

Re-thinking the architecture and bounded context design

Is your feature request related to a problem? Please describe.
The Bio-OS is constructed around the workspace concept, which is nicely understood. The main capability provided right now can be recapped into 3 parts: storage, workflow execution, jupyter notebook. So it confuses me all the time that there are four domain entities in the workspace bounded context, with notebook server & submission in their own context. In my consideration, the submission entity/context is tightly coupled with the workflow entity since submissions and run history are clearly only attached to the workflow (from what I've observed).

I understand that this division is based on the tech and performance consideration, so the workflow execution part should be a subdomain and therefore a microservice (just like the notebook server), but static records such as run and submission can be moved into workflow subcontext or workspace, and in web interface/cli the run history should be wrapped inside workflow as a child command.

The core dev said that cromwell and jupyterhub is in the supporting subdomains, and the core subdomain contains three bounded contexts. What I think is one bounded context is enough and the workspace could be classed as the core subdomain, while notebook server and workflow server as the supporting subdomains, with jupyterhub/cromwell as infrastruture inside. Forgive my laziness to not draw a UML to illustrate. (Maybe sometimes later😂)

Describe the solution you'd like
Explanation and/or discussion in detail about the system design, and whether we can improve it.

Add issue template

Describe the bug
We should have a issue template created rather for problem reporting or dev discussion. It's better left to you core maintainers to draft the template. The template use in this issue is a cutoff version of bug report.

To Reproduce
None.

Expected behavior
One or more issue templates filed.

Screenshots
None.

Add commit message convention check

Problem description
The team seem to have adopted AngularJS commit message convention, with however no validation on it.

Suggested solution
Tools like commitizen, commitlint, husky can be utilized. Include these node packages (along with node&npm) as dev dependency through makefile, rather than a small go-gitlint project.

Alternative solutions
None.

Additional context
None.

使用CLI工具时有时参数异常或命令执行异常时没有返回相应错误提示,希望可以优化下体验

比如导入workspace时,我没有指定必填参数挂载路径,或者挂载路径地址不符合要求时,运行命令后没有任何返回,期望可以返回对参数缺少或者不正确的提示
bioctl workspace import -y /test -t nfs

导入一个比较大的workspace,被nginx拦截无法上传成功时,运行命令后同样没有任何返回,无法得知是否上传成功及错误原因

其他一些cli命令的参数也有类似校验不完整的问题

任务挑战赛-【赛题解读】T4:Bio-OS流程规范扩展

背景介绍

Bio-OS 作为一个开源开放的生信分析平台,兼容 GA4GH 的 WES\TES 协议,为生物信息学研究提供了一个强大生物信息科研工具体系。然而,目前 Bio-OS 平台仅支持 WDL 语言作为其流程规范语言,这在一定程度上限制了用户的使用。为了进一步扩展 Bio-OS 平台的功能,本次大赛的主题是基于 WES 扩展 Bio-OS 平台支持的流程规范语言,请从 CWL、Nextflow 和 Snakemake 选择1种或者多种进行实现。

任务介绍

本次生物信息开发大赛的任务是基于GA4GH提出的WES规范(https://github.com/ga4gh/workflow-execution-service-schemas)设计并实现一个扩展方案,使 Bio-OS 平台能够支持多种流程规范语言。具体来说,参赛者需要完成以下任务:

  1. 熟悉 Bio-OS 平台的架构和工作原理,掌握其工作流管理和调度等功能。
  2. 了解 WES\TES 协议的基本原理和使用方法,掌握其与 Bio-OS 平台的集成方法。
  3. 了解 CWL、Nextflow 和 Snakemake 等流程规范语言的基本语法和语义,掌握其常用的命令和参数。
  4. 设计一个基于 WES 扩展 Bio-OS 平台支持的流程规范语言的方案,包括解析和转换流程规范语言、实现工作流管理和调度等功能。
  5. 实现扩展方案,包括编写相关的代码和配置文件、测试和调试等。
  6. 展示扩展后的 Bio-OS 平台的效果,包括使用新的流程规范语言提交工作流、监控和管理工作流等。

提交内容

请将以下提交内容发送到邮箱: [email protected]

  1. 一个名称为"生信挑战-T4-队伍名称"的md或word文件
    (示例文档https://github.com/Bio-OS/bioos/blob/main/docs/static/%E6%96%B9%E6%A1%88%E8%AE%BE%E8%AE%A1%E6%A8%A1%E7%89%88.doc仅作为参考,非必须参照)
  • 对整体流程规范扩展的方案设计介绍
  • 扩展的工作流功能使用说明
  • 包含一个视频链接(如上传到B站或其他网站,附上链接,如视频不大也可以直接发送邮箱)
  1. 将源代码提交PR到https://github.com/Bio-OS/bioos中

评分标准

本次比赛的评分标准包括以下几个方面:

  1. 难度基础分(权重10%)
  • T1:Notebook插件可视化工具集成 --难度基础分6分
  • T2:实现工作流运行任务监控 -- 难度基础分8分
  • T3:命令行工具实现Bio-OS Notebook使用 -- 难度基础分8分
  • T4:Bio-OS流程规范扩展 -- 难度基础分 10分
  1. 方案设计:监控方案的合理性、完整性和可扩展性。(权重20%) (示例文档https://github.com/Bio-OS/bioos/blob/main/docs/static/%E6%96%B9%E6%A1%88%E8%AE%BE%E8%AE%A1%E6%A8%A1%E7%89%88.doc 仅作为参考,非必须参照)
  2. Demo效果:DEMO效果展示,Demo视频中可尽量展示完整的操作及参数配置及结果生成。(权重20%)
  3. 代码实现:通过提交的源代码进行code review,将从代码质量、可维护性、可读性多个维度进行评分。(权重30%)
  • 代码质量:是否正确的实现了预期功能和需求,是否处理了预期的错误情况并提供了清晰地错误消息和处理方式,是否考虑可配置参数
  • 可维护性:是否易于修改和维护,包括代码的模块化程度、复用性,是否易于扩展
  • 可读性:代码是否易于阅读和理解,包括变量名、函数名、注释等命名是否清晰,代码结构是否合理
  1. 文档编写(20%)
  • 内容准确性:是否准确、完整、清晰的表达了所需信息
  • 文档可读性:文档的结构是否合理,逻辑清晰,是否易于阅读和理解,是否使用了恰当的图表和图片等

Reimport workflow will cause internal system error

Describe the bug
重新导入工作流的时候无论如何修改参数,点击确定后都会报系统内部错误

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior
支持工作流的重新导入

Screenshots
截屏2023-11-22 11 03 33

Desktop (please complete the following information):
开发机部署的地址
http://10.37.43.61:8888/workspace/wcle9frmi3a6rcsi7j0t0/workflow

Additional context
Add any other context about the problem here.

feat: Change workflow import method

Problem description

The current way of introducing workflows is rather unintuitive and I wonder what caused this design? In a normal user's perspective, uploading a workflow file or referencing some web link would be the simple and logical way. Any discussions?

Suggested solution

Change the method to import workflow by URL, which is clean and unified.

Alternative solutions

None.

Additional context

None.

任务挑战赛-【赛题解读】T3: 命令行工具实现Bio-OS Notebook使用

背景介绍​

Bio-OS 作为一个开源开放的生信分析平台,目前已支持使用命令行工具bioctl使用命令行运行Bio-OS, 本赛题则是基于已有的Bioctl命令行工具进行功能拓展,此赛题需参赛人员具备go语言的相关知识,并熟练掌握后端开发及调试工具。

任务介绍​

该赛题要求参赛团队基于Bio-OS现有的Bioctl命令行进一步扩展,支持通过Bioctl命令行实现Notebook的启动,运行配置修改,遍历Notebook实例及终止Notebook运行,需要参赛人员了解的内容包括

  1. 了解Bio-OS的架构和工作原理
  2. 了解Bioctl的架构和工作原理
  3. 以Bioctl为基础,开发使用Notebook相关的command
  4. 仅需支持http连接即可
  5. 需要完成命令行模式+交互模式

提交内容

请将以下提交内容发送到邮箱: [email protected]

  1. 一个名称为"任务挑战-T3-队伍名称"的md或word文件
    (示例文档https://github.com/Bio-OS/bioos/blob/main/docs/static/%E6%96%B9%E6%A1%88%E8%AE%BE%E8%AE%A1%E6%A8%A1%E7%89%88.doc仅作为参考,非必须参照)
  • 对bioctl Notebook command的方案设计介绍
  • Bioctl Notebook功能使用说明
  • 包含一个视频链接(如上传到B站或其他网站,附上链接,如视频不大也可以直接发送邮箱)
  1. 将源代码提交PR到https://github.com/Bio-OS/bioos中 ,源代码文件中应包含如下
  • main.go(Notebook command入口)
  • list.go
  • start.go
  • editNotebookSetting.go
  • Stop.go

评分标准​

本次比赛的评分标准包括以下几个方面:​

  1. 难度基础分(权重10%)
  • T1:Notebook插件可视化工具集成 --难度基础分6分
  • T2:实现工作流运行任务监控 -- 难度基础分8分
  • T3:命令行工具实现Bio-OS Notebook使用 -- 难度基础分8分
  • T4:Bio-OS流程规范扩展 -- 难度基础分 10分
  1. 方案设计:监控方案的合理性、完整性和可扩展性。(权重20%) (示例文档https://github.com/Bio-OS/bioos/blob/main/docs/static/%E6%96%B9%E6%A1%88%E8%AE%BE%E8%AE%A1%E6%A8%A1%E7%89%88.doc 仅作为参考,非必须参照)
  2. Demo效果:DEMO效果展示,Demo视频中可尽量展示完整的操作及参数配置及结果生成。(权重20%)
  3. 代码实现:通过提交的源代码进行code review,将从代码质量、可维护性、可读性多个维度进行评分。(权重30%)
  • 代码质量:是否正确的实现了预期功能和需求,是否处理了预期的错误情况并提供了清晰地错误消息和处理方式,是否考虑可配置参数
  • 可维护性:是否易于修改和维护,包括代码的模块化程度、复用性,是否易于扩展
  • 可读性:代码是否易于阅读和理解,包括变量名、函数名、注释等命名是否清晰,代码结构是否合理
  1. 文档编写(20%)
  • 内容准确性:是否准确、完整、清晰的表达了所需信息
  • 文档可读性:文档的结构是否合理,逻辑清晰,是否易于阅读和理解,是否使用了恰当的图表和图片等

A Script for Installing the Prerequisites For Running Bio-os with Minikube

Is your feature request related to a problem? Please describe.
Considering we don't have an English version of the installation guide for bio-os (#38), it might be helpful to provide a script for installing the prerequisites needed to install bioos with helm.

Describe the solution you'd like
The installation commands for prerequisites (minikube, helm, nfs-server, etc...) in the user guide into a single bash file.

Add Scripts for Installing the Dependencies For Running Bio-os

Is your feature request related to a problem? Please describe.
Considering we don't have an English version of the installation guide for bio-os (#38) and installing bio-os requires a non-trivial work of configuration and verification. It might be helpful to provide a script for installing the dependencies needed to install Bio-os with helm.

Describe the solution you'd like
A script for installing and a script for verifying storageClass for nfs-csi.
A script for installing and a script for verifying MySQL as a dependency for Bio-os.
A script for installing and a script for verifying Jupyterhub as a dependency for Bio-os.
A script for installing and a script for verifying Cromwell as a dependency for Bio-os.

Add Pull Request template

Describe the bug
We should have a PR template under .github folder created to format the message that a contributor wants to convey to the maintainer/reviewer. It's better left to the core maintainers of bio-os to draft the template, otherwise, a template could be borrowed from https://github.com/kubernetes/community considering bio-os heavily relies on Kubernetes to run. The template used in this issue is a cutoff version of the bug report.

To Reproduce
None.

Expected behavior
One or more issue templates filed.

Screenshots
None.

failed to download image

I worked in GitPod resource
"minikube was unable to download registry.cn-hangzhou.aliyuncs.com/google_containers/kicbase:v0.0.40, but successfully downloaded docker.io/kicbase/stable:v0.0.40 as a fallback image"

#2024年第三届计算机学会GLCC 题目:Bio-OS移动端应用程序开发

#2024年第三届计算机学会GLCC 题目详细介绍

##课题背景
随着生物医学研究的不断深入,大量的生物数据需要进行高效的处理和分析。生物医学大数据操作系统提供了强大的计算能力和资源,通过生物医学操作系统(Bio-OS)可以管理和处理这些生物数据及其分析任务。为了使得科研用户能够更加便捷的对自身的工作任务状态进行监控及管理,本课题拟针对该系统目前能力在移动端的适配进行开发。
本次大赛邀请参赛团队开发一个移动端应用监控程序,利用公有云生物医学操作系统(Bio-OS)提供的API接口,实时监控、查询和统计生物数据和生物分析工作流任务的运行状态。参赛者需要设计并开发一款移动应用程序,能够通过调用Bio-OS的API,获取生物数据处理和分析任务的状态和性能指标,并提供相关的告警和通知功能。

##功能要求:
可以根据自身实际情况实现下述功能或其他功能。
实时监控:能够实时显示生物数据处理和分析工作流任务的运行状态和关键性能指标(如任务进度、CPU使用率、内存使用率、数据处理速度等)。
任务管理:提供启动、暂停、重启和终止分析任务的功能。
数据查询:支持查询特定时间段内的任务运行历史数据和性能统计信息。
告警功能:当检测到异常情况(如任务失败、性能指标超出预设阈值)时,能够及时发送告警通知。
多账户支持:支持用户添加和管理多个云账户,并能够分别监控每个账户下的生物数据处理和分析任务。
用户界面:设计一个直观、易用的用户界面,方便用户查看和管理监控信息。

##技术要求:
为确保应用程序的广泛适用性和灵活性,本次开发不限制应用的实现形态。参赛者可以选择使用小程序等跨平台技术,实现兼容多种操作系统和设备的移动端解决方案。
充分利用所选公有云平台提供的Bio-OS API文档进行开发。
应用程序应具备良好的性能和安全性,确保用户数据的安全传输和存储。

##文档与展示:
提交详细的项目文档,包括功能描述、技术架构、使用说明等。
提供一个简短的视频演示,展示应用的主要功能和使用方法。

Added developer guide in the document

If the Bio-OS community can put how to build, how to setup the development environment, and how to do the contribution document into the developer guide section, that would be great!

bug: Unable to import workflow in local deployment

Describe the bug

I tried to import a demo workflow as the userguide provided, but ended with unknown error.

To Reproduce

Click “导入工作流” and fill the input fields following the userguide.

Expected behavior

Successful workflow import.

Screenshots

It claims import failure with correct inputs.

workflow-import-failure-0

I checked the inputs and found nothing to correct.

workflow-import-failure-1

Unable to re-submit the form and shows internal error.

workflow-import-failure-2

When running the notebook for the first time using the default configuration, it prompts "Internal system error"

Describe the bug
使用缺省配置首次运行notebook时,提示“系统内部错误”

To Reproduce
Steps to reproduce the behavior:

  1. Go to 'workspace'
  2. Click on 'test workspace' (setup beforehand)
  3. Go to 'Notebooks'
  4. Click on 'test notebook' (setup beforehand )
  5. Click on '编辑'
  6. Click on '运行资源配置'
  7. Click on '更新环境'

Expected behavior
A clear and concise description of what you expected to happen.
如果缺省环境存在,应当提示用户环境已被更新。
如果缺省环境不存在,应当提示用户采用正确配置而不是系统内部错误。

Screenshots
If applicable, add screenshots to help explain your problem.
截屏2023-11-28 16 54 12

Desktop (please complete the following information):

  • OS: [MacOS 13.2.1]
  • Browser [chrome]

docs: Add developer guide

This will be used in README.md file, which instructs developers how to build the components of this project

docs: Still typos and issues remain

Describe the bug
The maintainers are actively revising the docs at https://bio-os.gitbook.io/userguide/ which is good, however some issues remain due to unchanged part.
Issue related: #32, #33
PR related:
Bio-OS/helm-charts#11
Bio-OS/helm-charts#12
Bio-OS/helm-charts#13

Expected behavior
Fix the issues without causing new problems.

Screenshots
The maintainers said follow the new guide for there were problems in installing jupyterhub, but I don't see any change in this part.

jupyterhub-issues-remain

Yet typos exist.

bioos-issues-remain-0

Should't this command trigger unwanted outputs? At least on my machine it did.

bioos-issues-remain-1

And these are probably fixed in my PRs, so please take a look into them.

Add lint for checking golang coding style

Is your feature request related to a problem? Please describe.
For continuous integration, a Golang lint script is proposed which checks the coding style for all the Go language files.

Describe the solution you'd like
The script file should be configured to decide which set of checks should be enabled depending on a set of command line flags. A default set of checks should guarantee that all existing code passes without issues.

Describe alternatives you've considered
None.

Additional context
This lint script looks good to me.

任务挑战赛-【赛题解读】T2:实现工作流运行任务监控

背景介绍

生物信息学是一个快速发展的领域,其中涉及大量的数据处理和分析。在生物信息学中,流程任务监控是非常重要的,因为它可以帮助研究人员了解数据处理的状态和进度,及时发现和解决问题。Prometheus是一个开源的监控系统,它可以帮助我们实现生物信息流程任务的监控。本次生物信息开发大赛的主题是借助Prometheus开源监控实现生物信息流程任务监控的开发。参赛者需要熟悉Prometheus的基本原理和使用方法,了解生物信息学的流程任务,并能够将两者结合起来,实现对生物信息流程任务的监控。
Bio-OS开源创新大赛分为两种类型,分别为开发类和工具类,其中开发类赛题主要面向具备IT开发和工程化能力的参赛选手,对于本赛题来说,熟悉Go语言开发、Prometheus开源监控实现、Cromwell运行机制等,可以尝试本赛题。

任务介绍

本次生物信息开发大赛的主题是借助 Prometheus 开源监控实现生物信息流程任务监控的开发。该赛题要求参赛团队基于Bio-OS开源项目https://github.com/Bio-OS/bioos
实现Bio-OS平台中工作流任务运行监控能力。参赛者需要完成以下任务:

  1. 熟悉 Prometheus 的基本原理和使用方法,掌握其数据采集、存储和查询等功能。
  2. 了解生物信息学的流程任务,包括数据处理、分析和可视化等环节,明确监控的指标,如任务资源利用率(CPU/内存/磁盘)、任务开始结束时间等。
  3. 设计一个生物信息学流程任务的监控方案,包括监控的指标、采集数据的方式、数据存储的方式等。
  4. 实现监控方案,包括编写 Prometheus 的配置文件、编写 exporter 程序、编写 Prometheus 的查询语句等。
  5. 展示监控的效果,包括生成各种图表和报表,展示监控的数据和状态等。

提交内容

请将以下提交内容发送到邮箱: [email protected]
提交内容

  1. 一个名称为"生信挑战-T4-队伍名称"的md或word文件
    (示例文档https://github.com/Bio-OS/bioos/blob/main/docs/static/%E6%96%B9%E6%A1%88%E8%AE%BE%E8%AE%A1%E6%A8%A1%E7%89%88.doc仅作为参考,非必须参照)
  • 对整体流程规范扩展的方案设计介绍
  • 扩展的工作流功能使用说明
  • 包含一个视频链接(如上传到B站或其他网站,附上链接,如视频不大也可以直接发送邮箱)
  1. 将源代码提交PR到https://github.com/Bio-OS/bioos中

评分标准

本次比赛的评分标准包括以下几个方面:

  1. 难度基础分(权重10%)
  • T1:Notebook插件可视化工具集成 --难度基础分6分
  • T2:实现工作流运行任务监控 -- 难度基础分8分
  • T3:命令行工具实现Bio-OS Notebook使用 -- 难度基础分8分
  • T4:Bio-OS流程规范扩展 -- 难度基础分 10分
  1. 工具数量:工具的数量,数量越多,评分越高,超过4个工具后此项可得满分 (权重20%)
  2. Demo效果:DEMO效果展示,Demo视频中可尽量展示完整的功能以及监控效果。(权重20%)
  3. 代码实现:通过提交的源代码进行code review,将从代码质量、可维护性、可读性多个维度进行评分。(权重30%)
  • 代码质量:是否正确的实现了预期功能和需求,是否处理了预期的错误情况并提供了清晰地错误消息和处理方式,是否考虑可配置参数
  • 可维护性:是否易于修改和维护,包括代码的模块化程度、复用性,是否易于扩展
  • 可读性:代码是否易于阅读和理解,包括变量名、函数名、注释等命名是否清晰,代码结构是否合理
  1. 文档编写(20%)
  • 内容准确性:是否准确、完整、清晰的表达了所需信息
  • 文档可读性:文档的结构是否合理,逻辑清晰,是否易于阅读和理解,是否使用了恰当的图表和图片等

Unable to access services through web during installation/deployment

Describe the bug

Following the deployment guide, I stumbled over two steps that require/check the communication with the deployed services on k8s:

2.2 安装 Jupyterhub
...

# 将 jupyterhub 服务暴露出来,使浏览器可以直接访问.拿到 token 之后,就可以关掉这个端口转发。
kubectl -n bioos port-forward --address 0.0.0.0 service/hub 8081:8081

打开浏览器访问:http://服务器IP/jupyterhub, 申请新 token.

2.4 安装 Bioos 服务
...

# 端口转发:
kubectl -n ingress-nginx port-forward --address 0.0.0.0 service/ingress-nginx-controller 8888:80

...
打开浏览器访问 http://服务器IP/workspace,至此完成 bioos 安装.

To Reproduce

Follow the deployment guide at https://bio-os.gitbook.io/userguide/bu-shu/getting-set-up/bu-shu-bioos or https://github.com/Bio-OS/helm-charts/blob/main/README.md.

Expected behavior

Successful installation and interaction with k8s backend services.

Screenshots

cant-connect-jupyterhub

Error status of bioos access is the same with jupyterhub access.

Add user management service

Is your feature request related to a problem? Please describe.
I found that the open source version of bioos doesn't support multi-user! If it is targeted at a platform that serves a bunch of users, then user subdomain should be add to the system, with functions of user lifecycle, RBAC management etc.

Describe the solution you'd like
Support of multi-user and user management.

Describe alternatives you've considered
None yet.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.