devchat-ai / gopool Goto Github PK
View Code? Open in Web Editor NEWGoPool is a high-performance, feature-rich, and easy-to-use worker pool library for Golang.
License: MIT License
gopool's People
Contributors
Stargazers
Watchers
Forkers
qingfengzxr oliver258 curtis18 carlziess kenny-qiao dearsun jksusu fishersyu fgadvancer bighunter513 53912323 wyy-class zzjha-cn getsilver supuna97 hu2017 91xcode ksaturn gfcom made-by-lovegopool's Issues
worker.go中executeTaskWithTimeout死锁问题
worker.go中，executeTaskWithTimeout执行有超时限制的任务会死锁。
执行代码：
func main() {
var errCount int32
pool := gopool.NewGoPool(100, gopool.WithTimeout(1*time.Second), gopool.WithErrorCallback(func(err error) {
fmt.Println("get a error")
atomic.StoreInt32(&errCount, 1)
}))
defer pool.Release()
// 超时任务
// pool.AddTask(func() (interface{}, error) {
// time.Sleep(2 * time.Second)
// return nil, nil
// })
// 不超时任务
pool.AddTask(func() (interface{}, error) {
fmt.Println("run normal task")
return nil, nil
})
pool.AddTask(func() (interface{}, error) {
fmt.Println("run normal task")
return nil, nil
})
pool.AddTask(func() (interface{}, error) {
fmt.Println("run normal task")
return nil, nil
})
pool.Wait()
fmt.Println("err count:", errCount)
}因为result的chan接收到值后，会在下面等待err的chan 消息，但是此时已经没有发送方了。
task无法传参
无法将参数传给task func，建议增加传参功能
以map为例
`pool := gopool.NewGoPool(10)
defer pool.Release()
for k, v := range myMap {
pool.AddTask(func() (interface{}, error) {
log.Println(k, v)
return nil, nil
})
}
pool.Wait()`
task输出的始终都是最后一条数据
需要优先级队列
存在任务还没有运行就提前结束的异常情况
按照预期，pool.Wait()应该会等待所有任务运行结束
实际情况在少量任务时会出现任务还没有开始执行，pool.Wait()就结束了
代码部分
以下是从日志上检查到的异常情况的示例
异常情况1，任务已添加，但是任务没有运行，pool.Wait()直接就结束了
异常情况2，任务已添加，pool.Wait()提前结束，任务才开始调度运行
when shld u use spinlock vs the mutex version?
i already read this but i prefer having your feedback in regards to usage with gopool:
https://mmomtchev.medium.com/spinlock-vs-mutex-performance-3f73de53a460
can you also list out the limitations / caveats compared with ants,
https://github.com/bytedance/gopkg/tree/develop/util/gopool
https://github.com/alphadose/itogami
just curious what ur thoughts on these are. thx
The goPool is a private type, which cannot be referenced externally for setting types
如下: 我有个服务类,希望加个pool属性
type Server struct {
Setting Setting
running sync.Map
pool gopool.GoPool
}
发现gopool内的goPool是私有类型,无法外部引用,能否改为公有类型呢?
TestGoPoolWithMutex 测试用例添加 WithTimeout 无法通过 race 测试
此用例添加 WithTimeout(10*time.Microsecond)
func TestGoPoolWithMutex(t *testing.T) {
pool := NewGoPool(100, WithLock(new(sync.Mutex)), WithTimeout(10*time.Microsecond))
defer pool.Release()
for i := 0; i < 1000; i++ {
pool.AddTask(func() (interface{}, error) {
time.Sleep(10 * time.Millisecond)
return nil, nil
})
}
pool.Wait()
}报错信息：
$ go test -v -race ./...
=== RUN TestGoPoolWithMutex
==================
WARNING: DATA RACE
Write at 0x00c000032670 by goroutine 115:
github.com/devchat-ai/gopool.(*worker).executeTaskWithTimeout.func1()
/workspaces/gopool/worker.go:61 +0x6d
Previous write at 0x00c000032670 by goroutine 13:
github.com/devchat-ai/gopool.(*worker).executeTaskWithTimeout()
/workspaces/gopool/worker.go:72 +0x344
github.com/devchat-ai/gopool.(*worker).executeTask()
/workspaces/gopool/worker.go:39 +0xc4
github.com/devchat-ai/gopool.(*worker).start.func1()
/workspaces/gopool/worker.go:26 +0xaa
Goroutine 115 (running) created at:
github.com/devchat-ai/gopool.(*worker).executeTaskWithTimeout()
/workspaces/gopool/worker.go:60 +0x275
github.com/devchat-ai/gopool.(*worker).executeTask()
/workspaces/gopool/worker.go:39 +0xc4
github.com/devchat-ai/gopool.(*worker).start.func1()
/workspaces/gopool/worker.go:26 +0xaa
Goroutine 13 (running) created at:
github.com/devchat-ai/gopool.(*worker).start()
/workspaces/gopool/worker.go:23 +0xf7
github.com/devchat-ai/gopool.NewGoPool()
/workspaces/gopool/gopool.go:75 +0x54f
github.com/devchat-ai/gopool.TestGoPoolWithMutex()
/workspaces/gopool/gopool_test.go:14 +0xd4
testing.tRunner()
/usr/local/go/src/testing/testing.go:1576 +0x216
testing.(*T).Run.func1()
/usr/local/go/src/testing/testing.go:1629 +0x47
==================
==================
WARNING: DATA RACE
Write at 0x00c000032680 by goroutine 115:
github.com/devchat-ai/gopool.(*worker).executeTaskWithTimeout.func1()
/workspaces/gopool/worker.go:61 +0xa6
Previous write at 0x00c000032680 by goroutine 13:
github.com/devchat-ai/gopool.(*worker).executeTaskWithTimeout()
/workspaces/gopool/worker.go:72 +0x384
github.com/devchat-ai/gopool.(*worker).executeTask()
/workspaces/gopool/worker.go:39 +0xc4
github.com/devchat-ai/gopool.(*worker).start.func1()
/workspaces/gopool/worker.go:26 +0xaa
Goroutine 115 (running) created at:
github.com/devchat-ai/gopool.(*worker).executeTaskWithTimeout()
/workspaces/gopool/worker.go:60 +0x275
github.com/devchat-ai/gopool.(*worker).executeTask()
/workspaces/gopool/worker.go:39 +0xc4
github.com/devchat-ai/gopool.(*worker).start.func1()
/workspaces/gopool/worker.go:26 +0xaa
Goroutine 13 (running) created at:
github.com/devchat-ai/gopool.(*worker).start()
/workspaces/gopool/worker.go:23 +0xf7
github.com/devchat-ai/gopool.NewGoPool()
/workspaces/gopool/gopool.go:75 +0x54f
github.com/devchat-ai/gopool.TestGoPoolWithMutex()
/workspaces/gopool/gopool_test.go:14 +0xd4
testing.tRunner()
/usr/local/go/src/testing/testing.go:1576 +0x216
testing.(*T).Run.func1()
/usr/local/go/src/testing/testing.go:1629 +0x47
==================
testing.go:1446: race detected during execution of test
--- FAIL: TestGoPoolWithMutex (0.10s)
=== RUN TestGoPoolWithSpinLock
--- PASS: TestGoPoolWithSpinLock (0.11s)
=== RUN TestGoPoolWithError
--- PASS: TestGoPoolWithError (0.11s)
=== RUN TestGoPoolWithResult
--- PASS: TestGoPoolWithResult (0.10s)
=== RUN TestGoPoolWithRetry
--- PASS: TestGoPoolWithRetry (0.10s)
=== NAME
testing.go:1446: race detected during execution of test
FAIL
FAIL github.com/devchat-ai/gopool 0.542s
FAILWithMinWorkers() doesn't work correctly
func main() {
pool := gopool.NewGoPool(
20,
gopool.WithMinWorkers(3),
)
defer pool.Release()
}
当设置了minWorkers就会出现死锁
fatal error: all goroutines are asleep - deadlock!
原因是Release中的判断:
for len(p.workerStack) != p.minWorkers {
p.cond.Wait()
}
runtime error: slice bounds out of range
以下这平平无奇的代码,怎么就会发生崩溃呢
package main
import (
"fmt"
"math/rand"
"sync"
"time"
"github.com/devchat-ai/gopool"
)
func main() {
var m sync.Map
pool := gopool.NewGoPool(
20,
gopool.WithMinWorkers(3),
)
defer pool.Release()
var i = 0
for {
i += 1
if _, ok := m.Load(i); !ok {
pool.AddTask(func() (interface{}, error) {
m.Store(i, 1)
task(i)
m.Delete(i)
return nil, nil
})
}
if i > 20 {
break
}
}
pool.Wait()
}
func task(t int) error {
i := rand.Intn(20)
time.Sleep(time.Second * time.Duration(i))
fmt.Printf("> %d -- %d \n", t, i)
return nil
}
adjustWorkers is not notified on exit, thus potential resource leak
execute `go test -race` failed
go test -v -race ./...
$ go test -v -race ./...
=== RUN TestGoPoolWithMutex
--- PASS: TestGoPoolWithMutex (0.11s)
=== RUN TestGoPoolWithSpinLock
--- PASS: TestGoPoolWithSpinLock (0.11s)
=== RUN TestGoPoolWithError
--- PASS: TestGoPoolWithError (0.11s)
=== RUN TestGoPoolWithResult
--- PASS: TestGoPoolWithResult (0.11s)
=== RUN TestGoPoolWithRetry
==================
WARNING: DATA RACE
Read at 0x00c00001c258 by goroutine 423:
github.com/devchat-ai/gopool.TestGoPoolWithRetry()
/workspaces/gopool/gopool_test.go:147 +0x284
testing.tRunner()
/usr/local/go/src/testing/testing.go:1576 +0x216
testing.(*T).Run.func1()
/usr/local/go/src/testing/testing.go:1629 +0x47
Previous write at 0x00c00001c258 by goroutine 523:
github.com/devchat-ai/gopool.TestGoPoolWithRetry.func1()
/workspaces/gopool/gopool_test.go:138 +0x64
github.com/devchat-ai/gopool.(*worker).executeTaskWithoutTimeout()
/workspaces/gopool/worker.go:78 +0xd1
github.com/devchat-ai/gopool.(*worker).executeTask()
/workspaces/gopool/worker.go:41 +0xc7
github.com/devchat-ai/gopool.(*worker).start.func1()
/workspaces/gopool/worker.go:26 +0xaa
Goroutine 423 (running) created at:
testing.(*T).Run()
/usr/local/go/src/testing/testing.go:1629 +0x805
testing.runTests.func1()
/usr/local/go/src/testing/testing.go:2036 +0x8d
testing.tRunner()
/usr/local/go/src/testing/testing.go:1576 +0x216
testing.runTests()
/usr/local/go/src/testing/testing.go:2034 +0x87c
testing.(*M).Run()
/usr/local/go/src/testing/testing.go:1906 +0xb44
main.main()
_testmain.go:61 +0x2e9
Goroutine 523 (running) created at:
github.com/devchat-ai/gopool.(*worker).start()
/workspaces/gopool/worker.go:23 +0xf7
github.com/devchat-ai/gopool.NewGoPool()
/workspaces/gopool/gopool.go:75 +0x54f
github.com/devchat-ai/gopool.TestGoPoolWithRetry()
/workspaces/gopool/gopool_test.go:134 +0xfb
testing.tRunner()
/usr/local/go/src/testing/testing.go:1576 +0x216
testing.(*T).Run.func1()
/usr/local/go/src/testing/testing.go:1629 +0x47
==================
testing.go:1446: race detected during execution of test
--- FAIL: TestGoPoolWithRetry (0.10s)
=== NAME
testing.go:1446: race detected during execution of test
FAIL
FAIL github.com/devchat-ai/gopool 0.558s
FAILRecommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.