Giter VIP home page Giter VIP logo

immorwave / dynamic-tp-spring-cloud-starter Goto Github PK

View Code? Open in Web Editor NEW

This project forked from dromara/dynamic-tp

0.0 0.0 0.0 111 KB

🔥🔥🔥轻量级动态线程池,内置监控告警功能,基于主流配置中心(默认支持Nacos、Apollo,可通过SPI自定义实现)。Lightweight dynamic threadpool, with monitoring and alarming functions, base on popular config centers (default support Nacos、Apollo, can be customized through SPI).

License: Apache License 2.0

Java 100.00%

dynamic-tp-spring-cloud-starter's Introduction

基于配置中心的轻量级动态线程池 - DynamicTp

|  __ \                            (_) |__   __|   
| |  | |_   _ _ __   __ _ _ __ ___  _  ___| |_ __  
| |  | | | | | '_ \ / _` | '_ ` _ \| |/ __| | '_ \ 
| |__| | |_| | | | | (_| | | | | | | | (__| | |_) |
|_____/ \__, |_| |_|\__,_|_| |_| |_|_|\___|_| .__/ 
         __/ |                              | |    
        |___/                               |_|    
 :: Dynamic Thread Pool :: 

背景

  • 广泛性,在Java开发中,想要提高系统性能,线程池已经是一个90%以上的人都会选择使用的基础工具

  • 不确定性,项目中可能会创建很多线程池,既有IO密集型的,也有CPU密集型的,但线程池的参数并不好确定;需要有套机制在运行过程中动态去调整参数

  1. 最大线程数和任务队列设置过小,会导致服务大量抛出RejectedExecutionException异常
  2. 最大线程数设置过小,任务队列大小设置过大,会导致任务积压,响应时长变长
  3. 最大线程数设置过大,会导致线程切换频繁,处理速度反而下降
  4. 核心线程数设置过大,会导致空闲线程太多,浪费系统资源
  • 无感知性,线程池运行过程中的各项指标一般感知不到;需要有套监控报警机制在事前、事中就能让开发人员感知到线程池的运行状况,及时处理

  • 不便性,修改线程池参数后需要重启应用,灰常不方便

  • 高可用性,配置变更需要及时推送到客户端;需要有高可用的配置管理推送服务,分布式配置中心是现在大多数互联网系统都会使用的组件,与之结合可以大幅度减少开发量及接入难度


简介

  • 参考 美团线程池实践 ,对Java线程池参数动态化管理,增加监控、报警功能

  • 基于Spring框架,现只支持SpringBoot项目使用,轻量级,依赖配置中心,引入starter即可食用

  • 基于配置中心实现线程池参数动态调整,实时生效;集成主流配置中心,默认支持Nacos、Apollo,同时提供SPI接口可自定义扩展实现

  • 内置通知报警功能,提供多种报警维度(配置变更通知、活性报警、容量阈值报警、拒绝策略触发报警),默认支持企业微信、钉钉报警,同时提供SPI接口可自定义扩展实现

  • 内置简单线程池指标采集功能,支持通过MicroMeter、日志输出、Endpoint三种方式,可自定义扩展


架构设计

  • 功能主要分四大模块

  • 配置变更监听模块,默认实现Nacos、Apollo监听,可通过内部提供的SPI接口扩展其他实现

  • 线程池管理模块,主要实现线程池的增删查改功能

  • 监控日志模块,实现监控指标采集以及输出,默认实现Json log输出、MicroMeter采集、Endpoint监控三种方式,可通过内部提供的SPI接口扩展其他实现

  • 通知告警模块,对接办公平台,实现告警信息通知,默认实现钉钉、企微,可通过内部提供的SPI接口扩展其他实现

7FoNZV.png


使用

  • maven依赖

      <dependency>
          <groupId>io.github.lyh200</groupId>
          <artifactId>dynamic-tp-spring-cloud-starter</artifactId>
          <version>1.0.1-RELEASE</version>
      </dependency>
  • 配置文件

    spring:
      dynamic:
        tp:
          enabled: true
          enabledBanner: true        # 是否开启banner打印,默认true
          enabledCollect: false      # 是否开启监控指标采集,默认false
          collectType: logging       # 监控数据采集器类型(JsonLog | MicroMeter)
          logPath: /home             # 监控日志数据路径,默认${user.home}/logs
          monitorInterval: 5         # 监控时间间隔(报警判断、指标采集)
          nacos:                     # nacos配置,不配置有默认值(规则name-dev.yml这样)
            dataId: dynamic-tp-demo-dev.yml
            group: DEFAULT_GROUP
          apollo:                    # apollo配置,不配置默认拿apollo配置第一个namespace
            namespace: dynamic-tp-demo-dev.yml
          configType: yml            # 配置文件类型
          platforms:                 # 通知报警平台配置
            - platform: wechat
              urlKey: 3a7500-1287-4bd-a798-c5c3d8b69c  # 替换
              receivers: test1,test2  # 接受人企微名称
            - platform: ding
              urlKey: f80dad441fcd655438f4a08dcd6a     # 替换
              secret: SECb5441fa6f375d5b9d21           # 替换,非sign模式可以没有此值
              receivers: 15810119805                   # 钉钉账号手机号          
          executors:                  # 动态线程池配置
            - threadPoolName: dynamic-tp-test-1
              corePoolSize: 6
              maximumPoolSize: 8
              queueCapacity: 200
              queueType: VariableLinkedBlockingQueue
              rejectedHandlerType: CallerRunsPolicy
              keepAliveTime: 50
              allowCoreThreadTimeOut: false
              threadNamePrefix: test       # 线程名前缀
              notifyItems:                 # 报警项,不配置自动会配置(变更通知、容量报警、活性报警、拒绝报警)
                - type: capacity
                  enabled: true
                  threshold: 80
                  platforms: [ding,wechat]    # 可选配置,不配置默认拿上层platforms配置的所以平台
                  interval: 120               # 报警间隔(单位:s)
                - type: change
                  enabled: true
                - type: liveness
                  enabled: true
                  threshold: 80
                - type: reject
                  enabled: true
                  threshold: 1
  • 代码生成

    @Configuration
    public class DtpConfig {
    
        @Bean
        public DtpExecutor demo1Executor() {
            return DtpCreator.createDynamicFast("demo1-executor");
        }
    
        @Bean
        public ThreadPoolExecutor demo2Executor() {
            return ThreadPoolBuilder.newBuilder()
                    .threadPoolName("demo2-executor")
                    .corePoolSize(8)
                    .maximumPoolSize(16)
                    .keepAliveTime(50)
                    .allowCoreThreadTimeOut(true)
                    .workQueue(QueueTypeEnum.SYNCHRONOUS_QUEUE.getName(), null, false)
                    .rejectedExecutionHandler(RejectedTypeEnum.CALLER_RUNS_POLICY.getName())
                    .buildDynamic();
        }
    }
  • 代码调用

     public static void main(String[] args) {
            DtpExecutor dtpExecutor = DtpRegistry.getExecutor("dynamic-tp-test-1");
            dtpExecutor.execute(() -> System.out.println("test"));
     }

注意事项

  1. 配置项都有默认值,使用时可以查看源码,使用默认值的可以不配置相应字段

  2. 配置文件配置的参数会覆盖通过代码生成方式配置的参数

  3. 阻塞队列只有VariableLinkedBlockingQueue类型可以修改capacity

  4. 启动看到如下日志输出证明接入成功
    
    |  __ \                            (_) |__   __|   
    | |  | |_   _ _ __   __ _ _ __ ___  _  ___| |_ __  
    | |  | | | | | '_ \ / _` | '_ ` _ \| |/ __| | '_ \ 
    | |__| | |_| | | | | (_| | | | | | | | (__| | |_) |
    |_____/ \__, |_| |_|\__,_|_| |_| |_|_|\___|_| .__/ 
             __/ |                              | |    
            |___/                               |_|    
     :: Dynamic Thread Pool :: 
    
    DynamicTp register, executor: DtpMainPropWrapper(dtpName=dynamic-tp-test-1, corePoolSize=6, maxPoolSize=8, keepAliveTime=50, queueType=VariableLinkedBlockingQueue, queueCapacity=200, rejectType=RejectedCountableCallerRunsPolicy, allowCoreThreadTimeOut=false)
  5. 配置变更会输出相应日志,以及会推送通知
    
    DynamicTp [dynamic-tp-test-1] refresh end, changed keys: [corePoolSize, queueCapacity], corePoolSize: [6 => 4], maxPoolSize: [8 => 8], queueType: [VariableLinkedBlockingQueue => VariableLinkedBlockingQueue], queueCapacity: [200 => 2000], keepAliveTime: [50s => 50s], rejectedType: [CallerRunsPolicy => CallerRunsPolicy], allowsCoreThreadTimeOut: [false => false]

通知报警

  • 触发报警阈值会推送相应报警消息(活性、容量、拒绝触发),且会高亮显示相应字段

7FlTDe.png

  • 配置变更会推送通知消息,且会高亮变更了的字段

7FlouD.png


监控日志

  • 通过collectType属性配置监控指标输出类型,默认 logging

  • MicroMeter:通过引入相关依赖采集到支持的平台

  • Logging:定时采集指标以Json日志格式输出磁盘,地址:${logPath}/dynamictp/${appName}.monitor.log

  • 也有提供EndPoint端点(dynamic-tp),可以通过http方式请求

2022-01-11 00:25:20.599 INFO [dtp-monitor-thread-1:d.m.log] {"activeCount":0,"queueSize":0,"largestPoolSize":0,"poolSize":0,"rejectHandlerName":"RejectedCountableCallerRunsPolicy","queueCapacity":1024,"fair":false,"rejectCount":0,"waitTaskCount":0,"taskCount":0,"queueRemainingCapacity":1024,"corePoolSize":6,"queueType":"VariableLinkedBlockingQueue","completedTaskCount":0,"dtpName":"remoting-call","maximumPoolSize":8}
2022-01-11 00:25:25.603 INFO [dtp-monitor-thread-1:d.m.log] {"activeCount":0,"queueSize":0,"largestPoolSize":0,"poolSize":0,"rejectHandlerName":"RejectedCountableCallerRunsPolicy","queueCapacity":1024,"fair":false,"rejectCount":0,"waitTaskCount":0,"taskCount":0,"queueRemainingCapacity":1024,"corePoolSize":6,"queueType":"VariableLinkedBlockingQueue","completedTaskCount":0,"dtpName":"remoting-call","maximumPoolSize":8}
2022-01-11 00:25:30.609 INFO [dtp-monitor-thread-1:d.m.log] {"activeCount":0,"queueSize":0,"largestPoolSize":0,"poolSize":0,"rejectHandlerName":"RejectedCountableCallerRunsPolicy","queueCapacity":1024,"fair":false,"rejectCount":0,"waitTaskCount":0,"taskCount":0,"queueRemainingCapacity":1024,"corePoolSize":6,"queueType":"VariableLinkedBlockingQueue","completedTaskCount":0,"dtpName":"remoting-call","maximumPoolSize":8}
2022-01-11 00:25:35.613 INFO [dtp-monitor-thread-1:d.m.log] {"activeCount":0,"queueSize":0,"largestPoolSize":0,"poolSize":0,"rejectHandlerName":"RejectedCountableCallerRunsPolicy","queueCapacity":1024,"fair":false,"rejectCount":0,"waitTaskCount":0,"taskCount":0,"queueRemainingCapacity":1024,"corePoolSize":6,"queueType":"VariableLinkedBlockingQueue","completedTaskCount":0,"dtpName":"remoting-call","maximumPoolSize":8}
2022-01-11 00:25:40.616 INFO [dtp-monitor-thread-1:d.m.log] {"activeCount":0,"queueSize":0,"largestPoolSize":0,"poolSize":0,"rejectHandlerName":"RejectedCountableCallerRunsPolicy","queueCapacity":1024,"fair":false,"rejectCount":0,"waitTaskCount":0,"taskCount":0,"queueRemainingCapacity":1024,"corePoolSize":6,"queueType":"VariableLinkedBlockingQueue","completedTaskCount":0,"dtpName":"remoting-call","maximumPoolSize":8}

联系我

  • 对项目有什么想法或者建议,可以加我vx交流,或者创建issues,一起完善项目

7Fy2an.jpg

dynamic-tp-spring-cloud-starter's People

Contributors

yanhom1314 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.