- Golang
- Vue
- Flutter
labulakalia / crocodile Goto Github PK
View Code? Open in Web Editor NEWDistributed Task Scheduling System|分布式定时任务调度平台
License: MIT License
Distributed Task Scheduling System|分布式定时任务调度平台
License: MIT License
# server 运行命令及日志
./crocodile-linux-amd64 server -c core.toml
start run http server {"addr": "[::]:8080"}
# client 运行命令及报错
./crocodile-linux-amd64 client -c core.toml
info router/router.go:196 start run grpc server {"addr": "[::]:8088"}
error schedule/grpc.go:234 getgRPCConn failed {"error": "context deadline exceeded"}
error schedule/grpc.go:234 getgRPCConn failed {"error": "context deadline exceeded"}
我的 server 日志中没有收到任何成功注册 client 的日志, 这个是我之前注册成功时候的日志
info middleware/grpcmidd.go:36 [rpc req]
{"method": "/crocodile.task.Heartbeat/RegistryHost", "req": "ip:\"127.0.0.1\" port:8088 weight:100
hostname:\"DESKTOP-XXX\" version:\"v1.1.7\" remark:\"test remark\" ", "resp": "", "reqaddr": "127.0.0.1:59847",
"latency(ms)": 3.9941}
我将我的服务复制到另一台服务器上之后, 出现以上问题
请问这是为什么?
是需要创建个 casbin 库吗?
多个sql文件 执行时报错
任务运行后,返回结果含有中文的话,会乱码。
我们有很多定时任务,基本都是操作数据库的,所以 main.go 里会 import 包含一些私有仓库的 module,比如
package main
import "xxx.com/go-common/db"
func main() {
db.CallFunction()
}
用户sql也提供下默认用管理员用户呢?
我加的 get http://baidu.com 没有返回内容
修改core/tasktype/api.go 代码如下:
bs, err := ioutil.ReadAll(doresp.Body)
if err != nil {
log.Error("Read failed", zap.Error(err))
return
}
pw.Write(bs)
//var out = make([]byte, 1024)
//for {
// n, err := doresp.Body.Read(out)
// if err != nil {
// if err == io.EOF {
// break
// }
// log.Error("Read failed", zap.Error(err))
// return
// }
// if n > 0 {
// pw.Write(out[:n])
// }
//}
只显示了主任务的日志
报错如下,重新启动server端后恢复正常
{"level":"info","time":"2020-07-28T01:12:18.964+0800","line":"middleware/grpcmidd.go:36","msg":"[rpc req]","method":"/crocodile.task.Heartbeat/SendHb","req":"port:19588 ","resp":"","reqaddr":"127.0.0.1:39455","latency(ms)":1.551277}
{"level":"info","time":"2020-07-28T01:12:19.263+0800","line":"schedule/schedule2.go:571","msg":"start run task","taskname":"check_website_api.dstcar.com"}
{"level":"debug","time":"2020-07-28T01:12:19.263+0800","line":"schedule/schedule2.go:513","msg":"start get lock","taskid":"290165913800413184"}
{"level":"error","time":"2020-07-28T01:12:19.264+0800","line":"schedule/schedule2.go:643","msg":"model.GetTaskByID failed","taskid":"290165913800413184","error":"conn.PrepareContext failed: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)"}
{"level":"debug","time":"2020-07-28T01:12:19.265+0800","line":"schedule/schedule2.go:528","msg":"start release lock","taskid":"290165913800413184"}
{"level":"debug","time":"2020-07-28T01:12:19.265+0800","line":"schedule/schedule2.go:599","msg":"stop expire lock","lockid":"task:runlock:290165913800413184"}
{"level":"debug","time":"2020-07-28T01:12:21.973+0800","line":"schedule/grpc_impl.go:207","msg":"recv hearbeat","addr":"127.0.0.1:19588"}
{"level":"error","time":"2020-07-28T01:12:21.974+0800","line":"middleware/grpcmidd.go:28","msg":"resp failed","error":"conn.PrepareContext failed: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)"}
{"level":"info","time":"2020-07-28T01:12:21.974+0800","line":"middleware/grpcmidd.go:36","msg":"[rpc req]","method":"/crocodile.task.Heartbeat/SendHb","req":"port:19588 ","resp":"","reqaddr":"127.0.0.1:39455","latency(ms)":1.039417}
{"level":"error","time":"2020-07-28T01:12:22.088+0800","line":"middleware/auth.go:108","msg":"checkAuth failed","error":"conn.PrepareContext failed: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)"}
{"level":"warn","time":"2020-07-28T01:12:22.088+0800","line":"middleware/zaplogger.go:41","msg":"Gin","uid":"289850216985661440","method":"get","statuscode":10401,"reqip":"192.168.200.28","latency(ms)":1.281325,"url":"/api/v1/notify","respsize":36}
{"level":"debug","time":"2020-07-28T01:12:24.978+0800","line":"schedule/grpc_impl.go:207","msg":"recv hearbeat","addr":"127.0.0.1:19588"}
{"level":"error","time":"2020-07-28T01:12:24.980+0800","line":"middleware/grpcmidd.go:28","msg":"resp failed","error":"conn.PrepareContext failed: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)"}
{"level":"info","time":"2020-07-28T01:12:24.980+0800","line":"middleware/grpcmidd.go:36","msg":"[rpc req]","method":"/crocodile.task.Heartbeat/SendHb","req":"port:19588 ","resp":"","reqaddr":"127.0.0.1:39455","latency(ms)":1.379629}
{"level":"error","time":"2020-07-28T01:12:27.087+0800","line":"middleware/auth.go:108","msg":"checkAuth failed","error":"conn.PrepareContext failed: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)"}
{"level":"warn","time":"2020-07-28T01:12:27.087+0800","line":"middleware/zaplogger.go:41","msg":"Gin","uid":"289850216985661440","method":"get","statuscode":10401,"reqip":"192.168.200.28","latency(ms)":1.247689,"url":"/api/v1/notify","respsize":36}
{"level":"debug","time":"2020-07-28T01:12:27.982+0800","line":"schedule/grpc_impl.go:207","msg":"recv hearbeat","addr":"127.0.0.1:19588"}
{"level":"error","time":"2020-07-28T01:12:27.983+0800","line":"middleware/grpcmidd.go:28","msg":"resp failed","error":"conn.PrepareContext failed: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)"}
{"level":"info","time":"2020-07-28T01:12:27.983+0800","line":"middleware/grpcmidd.go:36","msg":"[rpc req]","method":"/crocodile.task.Heartbeat/SendHb","req":"port:19588 ","resp":"","reqaddr":"127.0.0.1:39455","latency(ms)":1.304077}
{"level":"debug","time":"2020-07-28T01:12:30.985+0800","line":"schedule/grpc_impl.go:207","msg":"recv hearbeat","addr":"127.0.0.1:19588"}
{"level":"error","time":"2020-07-28T01:12:30.986+0800","line":"middleware/grpcmidd.go:28","msg":"resp failed","error":"conn.PrepareContext failed: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)"}
{"level":"info","time":"2020-07-28T01:12:30.986+0800","line":"middleware/grpcmidd.go:36","msg":"[rpc req]","method":"/crocodile.task.Heartbeat/SendHb","req":"port:19588 ","resp":"","reqaddr":"127.0.0.1:39455","latency(ms)":1.220173}
{"level":"warn","time":"2020-07-28T01:12:31.297+0800","line":"schedule/schedule2.go:1081","msg":"task is stop run by auto schedule","taskname":"run_shell","taskid":"290088289640124416"}
{"level":"error","time":"2020-07-28T01:12:32.087+0800","line":"middleware/auth.go:108","msg":"checkAuth failed","error":"conn.PrepareContext failed: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)"}
{"level":"warn","time":"2020-07-28T01:12:32.087+0800","line":"middleware/zaplogger.go:41","msg":"Gin","uid":"289850216985661440","method":"get","statuscode":10401,"reqip":"192.168.200.28","latency(ms)":1.41084,"url":"/api/v1/notify","respsize":36}
{"level":"debug","time":"2020-07-28T01:12:33.988+0800","line":"schedule/grpc_impl.go:207","msg":"recv hearbeat","addr":"127.0.0.1:19588"}
{"level":"error","time":"2020-07-28T01:12:33.989+0800","line":"middleware/grpcmidd.go:28","msg":"resp failed","error":"conn.PrepareContext failed: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)"}
{"level":"info","time":"2020-07-28T01:12:33.989+0800","line":"middleware/grpcmidd.go:36","msg":"[rpc req]","method":"/crocodile.task.Heartbeat/SendHb","req":"port:19588 ","resp":"","reqaddr":"127.0.0.1:39455","latency(ms)":1.355797}
{"level":"info","time":"2020-07-28T01:12:34.263+0800","line":"schedule/schedule2.go:571","msg":"start run task","taskname":"check_website_api.dstcar.com"}
{"level":"debug","time":"2020-07-28T01:12:34.263+0800","line":"schedule/schedule2.go:513","msg":"start get lock","taskid":"290165913800413184"}
{"level":"error","time":"2020-07-28T01:12:34.266+0800","line":"schedule/schedule2.go:643","msg":"model.GetTaskByID failed","taskid":"290165913800413184","error":"conn.PrepareContext failed: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)"}
1、为了因应小场景使用,建议redis作为可选使用,存储数据库建议支持sqlite3,这样整个系统会比较轻灵,更加方便易用。
2、建议worker增加在其执行任务过程中向中心提交任务执行进度的报告。
3、建议worker与中心之间以http交互,这样方便可以对接其它语言开发的worker。
最后,感谢作者的无私奉献。
版本 v1.1.7
创建用户是手动在数据库里面插入,登陆验证通过,成功获取jwttoken,但是在如下图的代码中失败
enforcer := model.GetEnforcer()
function 的名称是checkAuth
Please Desc Your Question
StartRun()中有加锁、执行任务、释放锁的操作。
如果运行多个调度节点,虽然有锁作为互斥,但也可能执行多次。
例如,调度节点a,获取锁后执行任务1(假如执行10ms),接着释放锁。
这时调度节点b也来获取任务1的锁(调度节点对任务1的群已经释放了),能获取到锁,所以又执行了任务1…
因此,任务短期内可能在多个调度节点上重复执行多次。
不知道我理解对吗?
What Version Of crocodile you using
1.1.4
Describe your issue
从docker-compose安装的,运行一段时间worker会挂掉。
日志:
error schedule/grpc.go:281 client.SendHb failed {"error": "rpc error: code = Unknown desc = conn.PrepareContext failed: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)"}
没研究源码,目测像是句柄没释放导致
Please Desc Your Question
前端web打包有问题
error hostgroup/hostgroup.go:278 model.GetHostsByHGID {"error": "can not find hostid", "errorVerbose": "can not find hostid\ngithub.com/labulaka521/crocodile/core/model.GetHostByIDS\n\t/home/runner/work/crocodile/crocodile/core/model/host.go:228\ngithub.com/labulaka521/crocodile/core/model.GetHostsByHGID\n\t/home/runner/work/crocodile/crocodile/core/model/hostgroup.go:191\ngithub.com/labulaka521/crocodile/core/router/api/v1/hostgroup.GetHostsByIHGID\n\t/home/runner/work/crocodile/crocodile/core/router/api/v1/hostgroup/hostgroup.go:276\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:147\ngithub.com/labulaka521/crocodile/core/middleware.Oprtation.func1\n\t/home/runner/work/crocodile/crocodile/core/middleware/operation.go:61\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:147\ngithub.com/labulaka521/crocodile/core/middleware.PermissionControl.func1\n\t/home/runner/work/crocodile/crocodile/core/middleware/auth.go:118\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:147\ngithub.com/labulaka521/crocodile/core/middleware.ZapLogger.func1\n\t/home/runner/work/crocodile/crocodile/core/middleware/zaplogger.go:17\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:147\ngithub.com/gin-gonic/gin.RecoveryWithWriter.func1\n\t/home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:83\ngithub.com/gin-gonic/gin.(*Context).Next\n\t/home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:147\ngithub.com/gin-gonic/gin.(*Engine).handleHTTPRequest\n\t/home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:403\ngithub.com/gin-gonic/gin.(*Engine).ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:364\nnet/http.serverHandler.ServeHTTP\n\t/opt/hostedtoolcache/go/1.13.8/x64/src/net/http/server.go:2802\nnet/http.(*conn).serve\n\t/opt/hostedtoolcache/go/1.13.8/x64/src/net/http/server.go:1890\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.13.8/x64/src/runtime/asm_amd64.s:1357"}
Please Desc Your Question
清理日志功能:
任务名称、时间等 都是必填,否则会报错;界面显示的不是必填
如果没有任务名称,报错 “请求参数错误”; 这个操作没成功,出错了,审计日志 也会记录
没有时间,报错 error,错误不清晰
error middleware/operation.go:146 json.Unmarshal failed {"error": "json: cannot unmarshal string into Go struct field Cleanlog.preday of type int64"}
没有搜索按钮
如果没有输入任务名称,则不显示日志列表。如果显示 所有的日志,可能会更方便些。
报错日志如下
[root@croco crocodile]# ./crocodile-linux-amd64 version
Version : v1.1.5
Commit : 0f33f88
BuildDate : 2020-07-11
[root@croco crocodile]# cat crocodile.log | grep dingtalk
2020-07-21T09:01:19.514+0800 error alarm/alarm.go:271 send dingding notify failed {"error": "errmsg: invalid timestamp, more: [https://ding-doc.dingtalk.com/doc#/serverapi2/qf2nxq] errcode: 310000"}
如题
优先增加个用户删除
实在不能更名,就删除了重建吧。
What Version Of crocodile you using
1.1.7
Describe your issue
在个人中心模块,我添加通知的钉钉号,会报参数错误,调用的具体接口是
/api/v1/user/info
,请求的具体参数:
{"id":"1112222","email":"","wechat":"","dingphone":"188xxxxxxxx","slack":"","telegram":"","password":"","remark":""}
报错日志:
2020-11-23T11:38:55.493+0800 error user/user.go:170 ShouldBindJSON failed {"error": "Key: 'ChangeUserSelf.Name' Error:Field validation for 'Name' failed on the 'required' tag"}
很想用,可惜我们的服务器主要是windows。。。。
版本信息:v1.1.6 22850e
报错信息:
{"level":"error","time":"2020-07-29T14:00:30.217+0800","line":"alarm/alarm.go:271","msg":"send dingding notify failed","error":"errmsg: invalid timestamp, more: [https://ding-doc.dingtalk.com/doc#/serverapi2/qf2nxq] errcode: 310000"}
Please Desc Your Question
不提示安装,也没有管理员帐号密码
作为调度中心运行时,提示我首次需要运行安装,点击安装,显示报错请求参数错误,然后安装不成功
如题
你好,我按github 上的操作,主机列表还是空的,要怎样才能添加主机
1、支持四种调度算法随机、轮训、Worker权重、Worker最少任务数来调用Worker运行任务
假如我新增一个任务(不希望多个进程同时执行,如果同时跑需要加锁,所以脚本想做到同一时间只有一个worker在跑),定时跑,同一时间是不是只会有一个worker处理
2、看页面可以直接写脚本,但是不好进行代码维护,所以是可以直接通过页面的脚本去调用具体路径下的脚本吧
例如,shell 中 调用具体路径下的python 脚本。
这种方式要怎么写才能捕获到python 脚本的异常返回,上报异常。有没有demo 参考
shell 脚本异常 应该是通过exit xx 来返回错误码 吧?
What Version Of crocodile you using
please run
crocodile version
v1.1.7
Describe your issue
if you see some useful log ,you can copy it here
当用户为非管理员的时候,在函数runTask 的调用model.check 时似乎传参中table应该是task表,如下
exist, err := model.Check(ctx, model.TBTask, model.IDCreateByUID, runtask.ID, uid),原先传递的是hostgroup,但是看参数确实task表里面的id和创建者,请关注是否是bug
目前路由策略分为:随机、轮训、权重、最少任务。
是否可以考虑增加,主机组内所有主机都运行任务。
Docker version 19.03.8, build afacb8b
MacOS 10.15.4
docker-compose -f docker-compose.yml up master
...
Attaching to crocodile_master
master_1 | standard_init_linux.go:211: exec user process caused "no such file or directory"
1.sql的脚本里没有初始化用户,这边用户登录如何完成
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.