bytedance / gopkg Goto Github PK
View Code? Open in Web Editor NEWUniversal Utilities for Go
License: Apache License 2.0
Universal Utilities for Go
License: Apache License 2.0
context: I want to package this library for debian official archive but found the some tests failed to run in some environments,
refer to this link https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=995566 for more infomation
Given that we are working on an opensource project, I'd like to propose to opensource the releasing process.
Maybe following work can be documented on Wiki:
Below simple PR of release does not behave elegantly like other opensource projects:
#26
asynccache只能基于时间进行过期,如果过期时间配置过长或者load数据过于庞大,会导致OOM的问题。
我想基于asynccache实现一个文件缓存功能,但是在使用过程中发现,一旦文件数据过大,程序会因为OOM问题而直接被系统kill掉。
是否可以像ristretto那样实现一个内存配额限制呢?
建议
No response
For now it just runs benchmarking and does not check results, which looks like nop effectively.
For each Pull Request, need a mechanism to determine the benchmarking result is positive or negative?
请问有没有gopool和通过channel实现goroutine pool的性能对比,个人认为这个锁太大了:
1、 P的数量越多,channel的优势会优于锁
2、 如果goroutine创建速度快,锁等待会更高
感觉参照runtime,通过runtime.LockOSThread单独占几个系统线程,去处理这件事情(每次从全局的tasks中拿1/n的task),效率会更高,因为不会被runtime抢占。
GoPool seems to perform poorly from the benchmarks I ran
name time/op
UnlimitedGoroutines-8 301ms ± 4%
ErrGroup-8 515ms ± 9%
AntsPool-8 582ms ± 9%
GammaZeroPool-8 740ms ±13%
BytedanceGoPool-8 572ms ±18%
ItogamiPool-8 331ms ± 7%
name alloc/op
UnlimitedGoroutines-8 96.3MB ± 0%
ErrGroup-8 120MB ± 0%
AntsPool-8 22.4MB ± 6%
GammaZeroPool-8 18.8MB ± 1%
BytedanceGoPool-8 82.2MB ± 2%
ItogamiPool-8 25.6MB ± 2%
name allocs/op
UnlimitedGoroutines-8 2.00M ± 0%
ErrGroup-8 3.00M ± 0%
AntsPool-8 1.10M ± 2%
GammaZeroPool-8 1.05M ± 0%
BytedanceGoPool-8 2.59M ± 1%
ItogamiPool-8 1.05M ± 0%
Reference -> https://github.com/alphadose/itogami
Benchmarking code -> https://github.com/alphadose/go-threadpool-benchmarks
我看代码里的配置是在初始化熔断器时传入的,怎么才能在运行时动态的修改相关配置选项
Linux
go version go1.16.15 darwin/amd64
develop
gctunner
use GODEBUG=gctrace=1 go run main.go
to start
want see gc not frequent
gc frequent
gc 1 @1.007s 0%: 0.015+0.21+0.018 ms clock, 0.18+0.11/0.13/0.072+0.21 ms cpu, 4->4->0 MB, 5 MB goal, 12 P
gc 2 @1.017s 0%: 0.011+0.19+0.016 ms clock, 0.14+0.076/0.099/0.009+0.19 ms cpu, 20->20->0 MB, 21 MB goal, 12 P
gc 3 @1.021s 0%: 0.010+0.28+0.047 ms clock, 0.12+0.061/0.076/0.012+0.56 ms cpu, 20->20->0 MB, 21 MB goal, 12 P
gc 4 @1.025s 0%: 0.009+0.19+0.027 ms clock, 0.11+0.065/0.071/0.029+0.33 ms cpu, 20->20->0 MB, 21 MB goal, 12 P
gc 5 @1.028s 0%: 0.008+0.21+0.014 ms clock, 0.10+0.059/0.16/0.048+0.17 ms cpu, 20->20->0 MB, 21 MB goal, 12 P
gc 6 @1.032s 0%: 0.010+0.17+0.027 ms clock, 0.12+0.075/0.065/0.007+0.32 ms cpu, 20->20->0 MB, 21 MB goal, 12 P
gc 7 @1.035s 0%: 0.009+0.15+0.019 ms clock, 0.11+0.056/0.061/0.006+0.23 ms cpu, 20->20->0 MB, 21 MB goal, 12 P
gc 8 @1.039s 0%: 0.009+0.17+0.018 ms clock, 0.11+0.055/0.084/0.005+0.21 ms cpu, 20->20->0 MB, 21 MB goal, 12 P
gc 9 @1.042s 0%: 0.008+0.17+0.059 ms clock, 0.10+0.054/0.059/0.002+0.71 ms cpu, 20->20->0 MB, 21 MB goal, 12 P
package main
import (
"global/gctuner"
)
func main() {
const gb = 1024 * 1024 * 1024
threshold := gb * 4
gctuner.Tuning(uint64(threshold))
println(gctuner.GetGCPercent())
// 执行具体的业务逻辑
for {
data1 := make([]int, 10000, 10000)
data2 := make([]int, 10000, 10000)
for i, _ := range data1 {
data1[i] = i
}
for i, _ := range data2 {
data2[i] = i
}
result := 0
for i, _ := range data1 {
result += data1[i] * data2[i]
}
}
}
GODEBUG=gctrace=1 go run main.go
No response
possible to show an example of passing value / reference?
nice work by the way, i've tested against
https://github.com/panjf2000/ants
and found this to be faster but not sure if there are any significant difference between the two?
Windows 10
1.21
latest
collection/zset
使用zset(skiplist)时,发现其remove,incr, updateScore比想象中的慢非常多。
初步看了下cpu开销,发现lessthan的比较函数执行次数远高于 incr, zrem的复杂度M*log(N)的次数量级。
比如添加1w个元素耗时lessthan执行次数:221038,更新这1w个元素要lessthan执行次数: 98797380
把测试数据增加到10w个,lessthan次数到:2748187, 等了很久都没等到10w个元素更新完成(手动停止了)
创建一个zset,
No response
在计算bsr的时候,某些操作系统上使用一种for循环的方式计算,时间复杂度是O(n),n是这个数第从小到大最大的为1的位。
经过golang sdk库math/bits/bits.go/Len函数的启发,我想出了一种算法,具有O(1)的复杂度, 并且计算时间稳定。
并且针对64位操作系统和32位操作系统有不同的优化。
几乎减少了百分之九十的计算时间。
macos
go 1.22.1
all version
protobuf/grpc_compress.go
returned normally
panic index out of range [-1]
1、Define the empty reponse
message XXXRes { }
2、Write an arbitrary compressor and register it (encoding.RegisterCompressor(compressor))
3、Triggering the request
4、When the request returns,have a panic,'index out of range [-1]'
mcache.Free
func Free(buf []byte) {
size := cap(buf)
if !isPowerOfTwo(size) {
return
}
buf = buf[:0]
caches[bsr(size)].Put(buf)
}
when size = 0,bsr(size) return -1
caches[-1] = nil
QuoteSingle("a") -> 'a'
QuoteSingle("abc") -> 'abc'
QuoteSingle("ab'c") -> 'ab'c'
wu
No response
is the latest 1.18 go routine faster than this?
just curious coz it seems faster etc now
https://github.com/bytedance/gopkg/blob/develop/lang/mcache/mcache.go
这里使用bsr有啥特殊的背景或者考虑吗?我在mac跟linux端的bench中都是bits.Len64()更快一些。
for long running processes how big is this gopool size?
is there a setting to set the size of the pool as i have many goroutines?
what's the limit / limitations?
err = gopool.RegisterPool(gopool.NewPool("test", 5, gopool.NewConfig())) 这种命令是不是放到 init 中,假设有10个用户同时访问,是针对1个用户同时最多起5个协程,还是,针对10个用户同时最多起5个协程
I do not see that step from directory .github/workflow, please correct me if I'm wrong.
We should run all the functional tests automatically in each sub-repo to ensure no regression, for every pull request.
it seems that collection LSCQ use function cacheRemap16Byte
to remap index used by SCQ, to do best effort to avoid false-sharing, maybe we could refer to folly UnboundedQueue's index
, to remap index with less instructions.
however, i see no noticeable change in performance via BenchmarkDefault, perhaps because the function is not on the critical path.
improve cacheRemap16Byte
with less instructions
for scqsize = 2^n
, and k = 2*m + 1
, define remap(i)=k*i % scqsize
we can prove that: for any abs(i_1 - i_2) < scqsize
, if i_1 != i_2
, than remap(i_1) != remap(i_2)
.
this property makes remap() one-to-one map any i
in [x, x + scqsize)
to [0, scqsize)
.
package main
const (
scqsize = 1 << 16
cacheLineSize = 64
)
func cacheRemap16ByteA(index uint64) uint64 {
const cacheLineSize = cacheLineSize / 2
rawIndex := index & uint64(scqsize-1)
cacheLineNum := (rawIndex) % (scqsize / uint64(cacheLineSize))
cacheLineIdx := rawIndex / (scqsize / uint64(cacheLineSize))
return cacheLineNum*uint64(cacheLineSize) + cacheLineIdx
}
func cacheRemap16ByteB(index uint64) uint64 {
return (index * 5) & (scqsize - 1)
}
func main() {}
TEXT main.cacheRemap16ByteA(SB), NOSPLIT|ABIInternal, $0-8
FUNCDATA $0, gclocals·g2BeySu+wFnoycgXfElmcg==(SB)
FUNCDATA $1, gclocals·g2BeySu+wFnoycgXfElmcg==(SB)
FUNCDATA $5, main.cacheRemap16ByteA.arginfo1(SB)
FUNCDATA $6, main.cacheRemap16ByteA.argliveinfo(SB)
PCDATA $3, $1
MOVWLZX AX, CX
ANDL $2047, AX
SHRQ $11, CX
SHLQ $5, AX
ADDQ CX, AX
RET
TEXT main.cacheRemap16ByteB(SB), NOSPLIT|ABIInternal, $0-8
FUNCDATA $0, gclocals·g2BeySu+wFnoycgXfElmcg==(SB)
FUNCDATA $1, gclocals·g2BeySu+wFnoycgXfElmcg==(SB)
FUNCDATA $5, main.cacheRemap16ByteB.arginfo1(SB)
FUNCDATA $6, main.cacheRemap16ByteB.argliveinfo(SB)
PCDATA $3, $1
LEAQ (AX)(AX*4), CX
MOVWLZX CX, AX
RET
Current go.mod use go1.16 with golang.org/x/sys v0.0.0-20221010170243-090e33056c14 package which introduced go1.17's unsafe.Slice in 20220910 (see history https://cs.opensource.google/go/x/sys/+/master:unix/syscall.go;bpv=1). This will cause build failed in go1.16 with following error messges.
# golang.org/x/sys/unix
../compile_path/pkg/mod/golang.org/x/[email protected]/unix/syscall.go:83:16: undefined: unsafe.Slice
../compile_path/pkg/mod/golang.org/x/[email protected]/unix/syscall_linux.go:2255:9: undefined: unsafe.Slice
../compile_path/pkg/mod/golang.org/x/[email protected]/unix/syscall_unix.go:118:7: undefined: unsafe.Slice
../compile_path/pkg/mod/golang.org/x/[email protected]/unix/sysvshm_unix.go:33:7: undefined: unsafe.Slice
note: module requires Go 1.17
......
// Similar: group, ctxx := errgroup.WithCancelPool(ctx, pool)
group, ctxx := errgroup.WithCancel(ctx)
group.Go(func(ctxx)error)
if err := group.Wait(); err != nil {
return;
}
......
gopool
features (Auto-recovering Panics, Limit Goroutine Numbers, Reuse Goroutine Stack...) can make up for the lack of errgroup
very well.
No response
circuitbreaker.Panel 接口可以添加一个 State(key string) State 的方法吗
在某些场景下,我希望获取某个key关联的breaker的状态,来做不同的动作,比如进行重新进行调度
No response
ATT. At least the range of random floats should be given.
Bump Go version in the go.mod
to 1.17.
go.mod
is still 1.16.golang.org/x/sys
, the minimum version of this repo is 1.17, related issue: #163No response
related functions: IsAlpha/IsNumeric/IsAlphanumeric.
fix: check empty string first .
like Java ThreadPool.shutdown() or ThreadPool.shutdownNow()
main coroutine exit, but there are still tasks running in the gopool.
No response
Once the go routine from gopool panics, gopool would print stack info, and try to invoke panicHandler.
Even if we already set a panicHandler , there's no way to prevent the log printing.
Can we just let the panicHandler handle the log printing if it exists.
from
if r := recover(); r != nil {
msg := fmt.Sprintf("GOPOOL: panic in pool: %s: %v: %s", w.pool.name, r, debug.Stack())
logger.CtxErrorf(t.ctx, msg)
if w.pool.panicHandler != nil {
w.pool.panicHandler(t.ctx, r)
}
}
to
if r := recover(); r != nil {
if w.pool.panicHandler != nil {
w.pool.panicHandler(t.ctx, r)
} else {
msg := fmt.Sprintf("GOPOOL: panic in pool: %s: %v: %s", w.pool.name, r, debug.Stack())
logger.CtxErrorf(t.ctx, msg)
}
}
Thanka a lot.
two many logs in log files and test logs
No response
Hello!
I hope you are doing well!
We are a security research team. Our tool automatically detected a vulnerability in this repository. We want to disclose it responsibly. GitHub has a feature called Private vulnerability reporting, which enables security research to privately disclose a vulnerability. Unfortunately, it is not enabled for this repository.
Can you enable it, so that we can report it?
Thanks in advance!
PS: you can read about how to enable private vulnerability reporting here: https://docs.github.com/en/code-security/security-advisories/repository-security-advisories/configuring-private-vulnerability-reporting-for-a-repository
linux
go1.18.6 linux/amd64
AND
go1.19.1 linux/amd64
AND
go latest
latest
util/gctuner
ALL TEST PASSED
--- FAIL: TestTuner (0.20s)
tuner_test.go:37:
Error Trace: tuner_test.go:37
Error: Not equal:
expected: 0x1f4
actual : 0x32
Test: TestTuner
FAIL
exit status 1
FAIL github.com/bytedance/gopkg/util/gctuner 0.217s
cd util/gctuner && go test -count=1
OR
cd util/gctuner && go test -count=10
@joway PTAL, thanks!
Arm71 Linux
go1.17
20230728082804
collection/skipmap
在32位系统上运行正常。
在32位系统上运行时候,会报错:panic: unaligned 64-bit atomic operation。原因是使用了atomic.Load64(),在32位下操作64位数据不是原子的。
我认为这是个普遍存在于32位系统的问题,不一定非要arm71系统。应该是32位系统就能复现。
go something(str string) {
}(str)
how to do the equivalent in gopool?
Line 41 in 0b97f08
大佬您好,我使用了一下gopool.RegisterPool 好像仅仅只是将 这个 Pool 存入了一个sync.Map中,在后续的 gopool.Go 中并没有使用到自己 RegisterPool。是否考虑给 gopool.go文件中init 函数加一个check,如果调用者没有 RegisterPool,在使用 defaultPool。还有就是整个capacity容量是固定的,是否可以加一些动态机制,在大流量的时候taskList中的足够多了,可以进行capacity扩容,当task少的时候capacity可以进行缩容。还望大佬指正观点
1
1
1
1
1
1
1
No response
mac
go version go1.21.0 darwin/arm64
latest
gopkg/util/logger
预期输出实际的logger.CtxInfo 代码所在行数,参考截图中的差异
参考截图
gopkg/util/logger/default.go
55行:ll.logger.Output(3, msg)
issue 描述:
第一个参数calldepth 写死了3,输出log永远都是 "logger.go 55 ***"
期望Fix:
支持customized or default value = 4
No response
gopool是否可以开放WorkerCount()接口,用于业务监控gopool内goroutines数量,为gopool SetCap阈值提供依据
服务由于某些原因偶发Goroutines毛刺,导致CPU&Mem瞬间上涨,影响服务稳定性。因此想基于gopool.SetCap限制Goroutines数量,但Cap的阈值不太好确定(服务可以提供整个实例的Goroutines数量,但无法确定gopool协程池部分的Goroutines数量)。
No response
gopool中的f func()能否新增可入参的 func()类型?
形参的使用场景,还是比较多的
No response
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.