Giter VIP home page Giter VIP logo

open-hand / devops-service Goto Github PK

View Code? Open in Web Editor NEW
62.0 10.0 77.0 67.77 MB

DevOps Service is the core service of Choerodon. It integrated several open source tools to automate the DevOps process of planning, coding, building, testing, and deployment, operation, monitoring.

Home Page: http://choerodon.io

Java 94.81% Groovy 4.62% Shell 0.52% Dockerfile 0.01% Mustache 0.03%
java spring-boot spring-cloud spring devops gitflow choerodon

devops-service's Introduction

简体中文 | English

DevOps Service

DevOps Service DevOps Service是Choerodon平台实现持续交付的基础. 当前版本为: 1.1.0

DevOps Service通过自主整合的DevOps工具链,集成相关的开源工具,以此形成了计划、编码、测试、部署、运维以及监控的DevOps闭环。 并且只需通过简单的配置,您便能获得最佳的开发体验。

注意:原本的前端代码已经移动到这里

特性

DevOps Service 含有以下功能:

  • 应用服务管理 :对应用服务进行管理
  • 应用服务版本管理:对持续集成(Continuous Integration)过程中产生的可以直接在Kubernetes集群中进行部署的服务版本进行管理
  • 代码管理及版本控制:对服务的代码进行版本控制和管理
  • 分支管理:能够对服务的Git分支进行相应的操作
  • 代码质量监测:在CI过程中进行代码质量数据收集,集成sonarqube对代码质量进行监测
  • 持续集成概览:查看服务的持续集成过程
  • 部署管理:对持续集成所产生的服务版本通过GitOps进行部署
  • 持续部署流水线管理:使用工作流实现持续部署
  • 资源管理:对部署的资源(如:网络,域名,密文等)进行管理
  • 集群管理:管理Kubernetes集群

前置要求

  • JAVADevOps Service基于Java8进行开发
  • GitLabDevOps Service使用GitLab进行代码的托管。同时,通过基于GitLab Runner实现持续集成以完成代码编译,单元测试执行,代码质量分析,docker镜像生成,helm chart打包,服务版本发布等自动化过程
  • Harbor:企业级Docker registry 服务,用于存放服务版本所对应的docker镜像
  • Kubernetes:容器编排管理工具,用于部署服务版本所对应的helm chart包
  • ChartMuseum:Helm Chart仓库,用于存放服务版本所对应的helm chart包
  • Sonarqube:管理代码质量的开放平台,用于管理服务的代码质量
  • MySQL:主流数据库之一,用于DevOps Service的数据持久化
  • Redis:内存数据库,用于数据缓存和部分非持久化数据存储

服务依赖

  • choerodon-register: 注册中心,在线上环境代替本地的eureka-server
  • choerodon-iam:用户服务,与用户有关的操作依赖与此服务
  • choerodon-gateway: 网关服务
  • choerodon-oauth: 授权服务
  • choerodon-asgard : 事务一致性服务
  • choerodon-message : 通知服务
  • gitlab-service:gitlab服务
  • workflow-service:工作流服务
  • agile-service:敏捷服务,查询与分支有关的敏捷Issue需要依赖此服务

服务配置

  • bootstrap.yml:

    server:
      port: 8060
    spring:
      application:
        name: devops-service
      cloud:
        config:
          failFast: true
          retry:
            maxAttempts: 6
            multiplier: 1.5
            maxInterval: 2000
          uri: localhost:8010
          enabled: false
      mvc:
        static-path-pattern: /**
      resources:
        static-locations: classpath:/static,classpath:/public,classpath:/resources,classpath:/META-INF/resources,file:/dist
    management:
      server:
        port: 8061
      endpoints:
        web:
          exposure:
            include: '*'
  • application.yml:

    spring:
      datasource:
        url: jdbc:mysql://localhost/devops_service?useUnicode=true&characterEncoding=utf-8&useSSL=false
        username: choerodon
        password: choerodon
        hikari:
          maximum-pool-size: 15 # 数据库连接池连接数
      redis:
        host: localhost
        database: ${SPRING_REDIS_DATABASE:1}
      http:
        encoding:
          charset: UTF-8
          force: true
          enabled: true
    services:
      harbor:
        baseUrl: "harbor.example.com" # harbor地址
        username: "123456" # harbor用户名
        password: "123456" # 对应harbor用户名的密码
        insecureSkipTlsVerify: false
      gitlab:
        url: "gitlab.example.com" # gitlab地址
        sshUrl: "gitlab.example.com" # 用于ssh操作的gitlab地址
        projectLimit: 100 # gitlab用户可以创建的项目的数量限制
      helm:
        url: "helm.example.com" # 存放helm chart包的仓库地址
      gateway:
        url: "http://api.example.com" # 网关地址
    hzero:
      service:
        platform:
          name: choerodon-platform
        oauth:
          name: choerodon-oauth
        iam:
          name: choerodon-iam
        file:
          name: choerodon-file
        message:
          name: choerodon-message
        admin:
          name: choerodon-admin
        swagger:
          name: choerodon-swagger
        gateway:
          name: choerodon-gateway
        monitor:
          name: choerodon-monitor
      websocket:
        # 用于连接websocket的路径
        websocket: /websocket
        # 与当前服务的redis数据库一致
        redisDb: ${SPRING_REDIS_DATABASE:1}
        # 后端长连通信密钥
        secretKey: devops_ws
    choerodon:
      saga:
        consumer:
          core-thread-num: 20
          max-thread-num:  20 # 消费线程数
          poll-interval: 3 # 拉取消息的间隔(秒),默认1秒
          enabled: true # 是否启用消费端
      schedule:
        consumer:
          enabled: true # 启用任务调度消费端
          thread-num: 1 # 任务调度消费线程数
          poll-interval-ms: 1000 # 拉取间隔,默认1000毫秒
      resource:
        jwt:
          ignore: /workflow/**, /sonar/**, /ci, /sonar/info, /v2/api-docs, /agent/**, /ws/**, /gitlab/email, /webhook/**, /v2/choerodon/**, /choerodon/**, /actuator/**, /prometheus, /devops/**, /pre_stop, /websocket
    agent:
      version: "0.5.0" # devops-service此版本所预期的 choerodon-agent 的版本
      serviceUrl: "agent.example.com" # 用于 choerodon-agent 连接 devops-service 的地址
      certManagerUrl: "agent.example.com" # 存放CertManager的地址,用于安装
      repoUrl: "helm.example.com" # 存放agent的地址,用于安装
    mybatis:
      mapperLocations: classpath*:/mapper/*.xml
      configuration:
        mapUnderscoreToCamelCase: true
    feign:
      hystrix:
        shareSecurityContext: true
        command:
          default:
            execution:
              isolation:
                thread:
                  timeoutInMilliseconds: 30000
    ribbon:
      ConnectTimeout: 50000
      ReadTimeout: 50000
    
    asgard-servie:
      ribbon:
        ConnectTimeout: 50000
        ReadTimeout: 50000
    
    devops:
      ansible:
        image: registry.cn-shanghai.aliyuncs.com/c7n/kubeadm-ha:0.1.0
      # helm 下载地址
      helm:
        download-url: https://file.choerodon.com.cn/kubernetes-helm/v3.2.4/helm-v3.2.4-linux-amd64.tar.gz
      # 流水线生成Gitlab Ci文件中默认的runner 镜像地址
      ci:
        default:
          image: registry.cn-shanghai.aliyuncs.com/c7n/cibase:0.9.1
        pipeline:
          sync:
            executor:
              # 核心线程池大小
              corePoolSize: 5
              # 最大线程池大小
              maxPoolSize: 8
            unterminated:
              # ci流水线对未终结的流水线进行数据补偿的时间阈值, 单位: 毫秒 (默认600秒)
              thresholdMilliSeconds: 600000
            pending:
              # ci流水线对pending的流水线进行数据补偿的时间阈值, 单位: 毫秒 (默认600秒)
              thresholdMilliSeconds: 600000
            jobEmpty:
              # ci流水线对非跳过状态的且没有job信息流水线进行数据补偿的时间阈值, 单位: 毫秒 (默认600秒)
              thresholdMilliSeconds: 600000
            refresh:
              # redisKey的过期时间, 用于控制同一条流水线的刷新间隔, 减少对gitlab的访问次数
              periodSeconds: 60
      # 批量部署的请求条数限制
      batch:
        deployment:
          maxSize: 20
    
    # websocket的最大缓冲区大小,单位字节byte
    websocket:
      buffer:
        maxTextMessageSize: 4194304
        maxBinaryMessageSize: 4194304

安装和启动步骤

  1. 创建数据库devops_service,创建用户choerodon,并为用户分配权限:

    CREATE USER 'choerodon'@'%' IDENTIFIED BY "choerodon";
    CREATE DATABASE devops_service DEFAULT CHARACTER SET utf8;
    GRANT ALL PRIVILEGES ON devops_service.* TO choerodon@'%';
    FLUSH PRIVILEGES;
  2. 拉取DevOps Service代码到本地:

    git clone https://github.com/choerodon/devops-service.git
  3. 在项目根目录执行命令: mvn clean package spring-boot:repackage -Dmaven.test.skip=true && bash init-database.sh

  4. 使用下列命令运行或直接在集成环境中运行 DevopsServiceApplication

    mvn clean spring-boot:run

链接

更新日志

反馈途径

如果您发现任何缺陷或bug,请及时 issue告知我们 。

如何参与

欢迎参与我们的项目,了解更多有关如何参与贡献的信息。

devops-service's People

Contributors

achineseboyjimli avatar aichovy avatar aimzhangjian avatar alforstudy avatar berubou avatar cczhung avatar chenwei0214 avatar cissiezheng avatar crockitwood avatar devil-scp avatar ethan-mu avatar fangyoga avatar hand-shanyu avatar iotale avatar isaacy-125 avatar lockshe avatar lyanfang avatar mikane200 avatar msfee avatar nollieleo avatar phyear avatar thefuckingcode avatar timebye avatar vinkdong avatar wang-ha0 avatar wangalaindelon avatar yidaqiang avatar yslernst avatar zhuzhiyang avatar zmfcn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

devops-service's Issues

devops前端新的代码在哪里,分支里版本还是0.23

Make sure these boxes are checked before submitting your issue - thank you!

  • I have checked the Choerodon logs and included it here as text if any
  • I have reproduced the issue with at least the latest released version of Choerodon
  • I have checked the issue tracker for the same issue and I haven't found one similar

Choerodon version

Expected results

Actual results

Steps to reproduce

升级v0.20后启动失败

Make sure these boxes are checked before submitting your issue - thank you!

  • [ *] I have checked the Choerodon logs and included it here as text if any
  • [ *] I have reproduced the issue with at least the latest released version of Choerodon
  • [ *] I have checked the issue tracker for the same issue and I haven't found one similar

Choerodon version

v0.20

Expected results

Actual results

2019-12-30 18:41:08.330 WARN 7 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.boot.context.properties.ConfigurationPropertiesBindException: Error creating bean with name 'gitlabConfigurationProperties': Could not bind properties to 'GitlabConfigurationProperties' : prefix=services.gitlab, ignoreInvalidFields=false, ignoreUnknownFields=true; nested exception is org.springframework.boot.context.properties.bind.BindException: Failed to bind properties under 'services.gitlab' to io.choerodon.devops.infra.config.GitlabConfigurationProperties
2019-12-30 18:41:08.367 INFO 7 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
2019-12-30 18:41:08.368 DEBUG 7 --- [ main] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Before shutdown stats (total=15, active=0, idle=15, waiting=0)
2019-12-30 18:41:08.368 DEBUG 7 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl@226a6610: (connection evicted)
2019-12-30 18:41:08.375 DEBUG 7 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl@7e1a11ff: (connection evicted)
2019-12-30 18:41:08.375 DEBUG 7 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl@270baab4: (connection evicted)
2019-12-30 18:41:08.375 DEBUG 7 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl@eacbb2d3: (connection evicted)
2019-12-30 18:41:08.376 DEBUG 7 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl@3162d57c: (connection evicted)
2019-12-30 18:41:08.376 DEBUG 7 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl@ab92d6b7: (connection evicted)
2019-12-30 18:41:08.376 DEBUG 7 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl@de35acc9: (connection evicted)
2019-12-30 18:41:08.376 DEBUG 7 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl@aa8682ff: (connection evicted)
2019-12-30 18:41:08.376 DEBUG 7 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl@bd54991e: (connection evicted)
2019-12-30 18:41:08.376 DEBUG 7 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl@5fef9f75: (connection evicted)
2019-12-30 18:41:08.377 DEBUG 7 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl@31564145: (connection evicted)
2019-12-30 18:41:08.377 DEBUG 7 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl@9361cea9: (connection evicted)
2019-12-30 18:41:08.377 DEBUG 7 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl@37bba36d: (connection evicted)
2019-12-30 18:41:08.377 DEBUG 7 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl@5877776d: (connection evicted)
2019-12-30 18:41:08.377 DEBUG 7 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl@1e83ba00: (connection evicted)
2019-12-30 18:41:08.377 DEBUG 7 --- [ main] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - After shutdown stats (total=0, active=0, idle=0, waiting=0)
2019-12-30 18:41:08.377 INFO 7 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
2019-12-30 18:41:08.378 INFO 7 --- [ main] o.apache.catalina.core.StandardService : Stopping service [Tomcat]
2019-12-30 18:41:08.394 INFO 7 --- [ main] ConditionEvaluationReportLoggingListener :

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2019-12-30 18:41:08.397 ERROR 7 --- [ main] o.s.b.d.LoggingFailureAnalysisReporter :


APPLICATION FAILED TO START


Description:

Failed to bind properties under 'services.gitlab' to io.choerodon.devops.infra.config.GitlabConfigurationProperties:

Property: services.gitlab.project-limit
Value: 100
Origin: System Environment Property "SERVICES_GITLAB_PROJECTLIMIT"
Reason: Invalid property 'services' of bean class [io.choerodon.devops.infra.config.GitlabConfigurationProperties]: Bean property 'services' is not readable or has an invalid getter method: Does the return type of the getter match the parameter type of the setter?

Action:

Update your application's configuration

Steps to reproduce

从v0.19升级到v0.20后启动失败

ci执行时获取auto_devops.sh脚本404

Make sure these boxes are checked before submitting your issue - thank you!

  • I have checked the Choerodon logs and included it here as text if any
  • I have reproduced the issue with at least the latest released version of Choerodon
  • I have checked the issue tracker for the same issue and I haven't found one similar

Choerodon version

0.22.0

1、不传入token时,接口返回200
image
2、传入gitlab-runner对应的token时,接口返回404
image

image

3、通过api接口方位,传入token和type参数时,接口返回404
image

can`t fetch shared appservice when project is more than 20

Make sure these boxes are checked before submitting your issue - thank you!

  • [* ] I have checked the Choerodon logs and included it here as text if any
  • [*] I have reproduced the issue with at least the latest released version of Choerodon
  • [*] I have checked the issue tracker for the same issue and I haven't found one similar

Choerodon version

v0.19.7 bug latest version code is same

Expected results

get shared appservice

Actual results

Steps to reproduce

  1. Project more than 20
  2. add shared rules in first project
  3. created from shared app service in last project

无法拉依赖包

Make sure these boxes are checked before submitting your issue - thank you!

  • I have checked the Choerodon logs and included it here as text if any
  • I have reproduced the issue with at least the latest released version of Choerodon
  • I have checked the issue tracker for the same issue and I haven't found one similar

Choerodon version

Expected results

INFO] Scanning for projects...
[INFO]
[INFO] --------------------< io.choerodon:devops-service >---------------------
[INFO] Building devops-service 0.23.16.RELEASE
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.870 s
[INFO] Finished at: 2021-05-17T18:17:05+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project devops-service: Could not resolve dependencies for project io.choerodon:devops-service:jar:0.23.16.RELEASE: Failed to collect dependencies at io.choerodon:choerodon-liquibase:jar:0.16.1.RELEASE: Failed to read artifact descriptor for io.choerodon:choerodon-liquibase:jar:0.16.1.RELEASE: Failure to find io.choerodon:choerodon-starter-parent:pom:0.16.1.RELEASE in http://nexus.saas.hand-china.com/content/repositories/public/ was cached in the local repository, resolution will not be reattempted until the update interval of HandPublic has elapsed or updates are forced -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

Actual results

Steps to reproduce

取 Sonarqube 指标失败

Choerodon version

1.0

Expected results

按官方文档部署 Sonarqube 使用的 Sonarqube 8,web api 中的 componentKey 应为 component

Actual results

取 Sonarqube 指标失败

Steps to reproduce

按官方文档部署,打开 Sonarqube 相关页面

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.