https://github.com/bytedance/gopkg/blob/develop/lang/mcache/mcache.go
这里使用bsr有啥特殊的背景或者考虑吗？我在mac跟linux端的bench中都是bits.Len64()更快一些。

Operating System

Windows 10

Go Version

1.21

Package Version

latest

Affected Packages

collection/zset

Expected Behavior

使用zset(skiplist)时，发现其remove,incr, updateScore比想象中的慢非常多。
初步看了下cpu开销，发现lessthan的比较函数执行次数远高于 incr, zrem的复杂度M*log(N)的次数量级。
比如添加1w个元素耗时lessthan执行次数：221038，更新这1w个元素要lessthan执行次数： 98797380
把测试数据增加到10w个，lessthan次数到：2748187， 等了很久都没等到10w个元素更新完成（手动停止了）

Actual Behavior

Reproduction Steps

创建一个zset，

1. 添加10000个元素
2. incr 10000个元素
3. 删除这10000个元素

Other Information

No response

Question

is the latest 1.18 go routine faster than this?

just curious coz it seems faster etc now

Operating System

Arm71 Linux

Go Version

go1.17

Package Version

20230728082804

Affected Packages

collection/skipmap

Expected Behavior

在32位系统上运行正常。

Actual Behavior

在32位系统上运行时候，会报错：panic: unaligned 64-bit atomic operation。原因是使用了atomic.Load64()，在32位下操作64位数据不是原子的。

Reproduction Steps

1. 使用skipmap的LoadOrStoreLazy。
2. 编译32位Arm下运行的程序。
3. 在Arm71上运行编译好的程序。

Other Information

我认为这是个普遍存在于32位系统的问题，不一定非要arm71系统。应该是32位系统就能复现。

Summary

like Java ThreadPool.shutdown() or ThreadPool.shutdownNow()

Motivation

main coroutine exit, but there are still tasks running in the gopool.

Explanation

No response

I do not see that step from directory .github/workflow, please correct me if I'm wrong.

We should run all the functional tests automatically in each sub-repo to ensure no regression, for every pull request.

For now it just runs benchmarking and does not check results, which looks like nop effectively.

For each Pull Request, need a mechanism to determine the benchmarking result is positive or negative?

Question

go something(str string) {
}(str)

how to do the equivalent in gopool?

related functions: IsAlpha/IsNumeric/IsAlphanumeric.
fix: check empty string first .

Operating System

Linux

Go Version

go version go1.16.15 darwin/amd64

Package Version

develop

Affected Packages

gctunner

Expected Behavior

use GODEBUG=gctrace=1 go run main.go to start
want see gc not frequent

Actual Behavior

gc frequent
gc 1 @1.007s 0%: 0.015+0.21+0.018 ms clock, 0.18+0.11/0.13/0.072+0.21 ms cpu, 4->4->0 MB, 5 MB goal, 12 P
gc 2 @1.017s 0%: 0.011+0.19+0.016 ms clock, 0.14+0.076/0.099/0.009+0.19 ms cpu, 20->20->0 MB, 21 MB goal, 12 P
gc 3 @1.021s 0%: 0.010+0.28+0.047 ms clock, 0.12+0.061/0.076/0.012+0.56 ms cpu, 20->20->0 MB, 21 MB goal, 12 P
gc 4 @1.025s 0%: 0.009+0.19+0.027 ms clock, 0.11+0.065/0.071/0.029+0.33 ms cpu, 20->20->0 MB, 21 MB goal, 12 P
gc 5 @1.028s 0%: 0.008+0.21+0.014 ms clock, 0.10+0.059/0.16/0.048+0.17 ms cpu, 20->20->0 MB, 21 MB goal, 12 P
gc 6 @1.032s 0%: 0.010+0.17+0.027 ms clock, 0.12+0.075/0.065/0.007+0.32 ms cpu, 20->20->0 MB, 21 MB goal, 12 P
gc 7 @1.035s 0%: 0.009+0.15+0.019 ms clock, 0.11+0.056/0.061/0.006+0.23 ms cpu, 20->20->0 MB, 21 MB goal, 12 P
gc 8 @1.039s 0%: 0.009+0.17+0.018 ms clock, 0.11+0.055/0.084/0.005+0.21 ms cpu, 20->20->0 MB, 21 MB goal, 12 P
gc 9 @1.042s 0%: 0.008+0.17+0.059 ms clock, 0.10+0.054/0.059/0.002+0.71 ms cpu, 20->20->0 MB, 21 MB goal, 12 P

Reproduction Steps

1. run following code

package main

import (
	"global/gctuner"
)

func main() {

	const gb = 1024 * 1024 * 1024
	threshold := gb * 4

	gctuner.Tuning(uint64(threshold))
	println(gctuner.GetGCPercent())

	// 执行具体的业务逻辑
	for {
		data1 := make([]int, 10000, 10000)
		data2 := make([]int, 10000, 10000)
		for i, _ := range data1 {
			data1[i] = i
		}
		for i, _ := range data2 {
			data2[i] = i
		}
		result := 0
		for i, _ := range data1 {
			result += data1[i] * data2[i]
		}
	}

}

2. Then run it with command, You will see the gc info

 GODEBUG=gctrace=1 go run main.go

Other Information

No response

Summary

gopool中的f func()能否新增可入参的 func()类型？

Motivation

形参的使用场景，还是比较多的

Explanation

No response

Summary

gopool是否可以开放WorkerCount()接口，用于业务监控gopool内goroutines数量，为gopool SetCap阈值提供依据

Motivation

服务由于某些原因偶发Goroutines毛刺，导致CPU&Mem瞬间上涨，影响服务稳定性。因此想基于gopool.SetCap限制Goroutines数量，但Cap的阈值不太好确定(服务可以提供整个实例的Goroutines数量，但无法确定gopool协程池部分的Goroutines数量)。

Explanation

No response

Summary

Bump Go version in the go.mod to 1.17.

Motivation

• Since Go 1.19 is released, we officially support 1.17 1.18 1.19, but the Go version in the go.mod is still 1.16.
• gopkg requires golang.org/x/sys, the minimum version of this repo is 1.17, related issue: #163

Explanation

No response

Question

for long running processes how big is this gopool size?
is there a setting to set the size of the pool as i have many goroutines?

what's the limit / limitations?

Question

GoPool seems to perform poorly from the benchmarks I ran

name                   time/op
UnlimitedGoroutines-8   301ms ± 4%
ErrGroup-8              515ms ± 9%
AntsPool-8              582ms ± 9%
GammaZeroPool-8         740ms ±13%
BytedanceGoPool-8       572ms ±18%
ItogamiPool-8           331ms ± 7%

name                   alloc/op
UnlimitedGoroutines-8  96.3MB ± 0%
ErrGroup-8              120MB ± 0%
AntsPool-8             22.4MB ± 6%
GammaZeroPool-8        18.8MB ± 1%
BytedanceGoPool-8      82.2MB ± 2%
ItogamiPool-8          25.6MB ± 2%

name                   allocs/op
UnlimitedGoroutines-8   2.00M ± 0%
ErrGroup-8              3.00M ± 0%
AntsPool-8              1.10M ± 2%
GammaZeroPool-8         1.05M ± 0%
BytedanceGoPool-8       2.59M ± 1%
ItogamiPool-8           1.05M ± 0%

Reference -> https://github.com/alphadose/itogami

Benchmarking code -> https://github.com/alphadose/go-threadpool-benchmarks

Summary

Once the go routine from gopool panics, gopool would print stack info, and try to invoke panicHandler.
Even if we already set a panicHandler , there's no way to prevent the log printing.
Can we just let the panicHandler handle the log printing if it exists.
from

if r := recover(); r != nil {
  msg := fmt.Sprintf("GOPOOL: panic in pool: %s: %v: %s", w.pool.name, r, debug.Stack())
  logger.CtxErrorf(t.ctx, msg)
  if w.pool.panicHandler != nil {
	  w.pool.panicHandler(t.ctx, r)
  }
}

to

if r := recover(); r != nil {
  if w.pool.panicHandler != nil {
	  w.pool.panicHandler(t.ctx, r)
  } else {
    msg := fmt.Sprintf("GOPOOL: panic in pool: %s: %v: %s", w.pool.name, r, debug.Stack())
    logger.CtxErrorf(t.ctx, msg)
  }
}

Thanka a lot.

Motivation

two many logs in log files and test logs

Explanation

No response

Question

大佬您好，我使用了一下gopool.RegisterPool 好像仅仅只是将 这个 Pool 存入了一个sync.Map中，在后续的 gopool.Go 中并没有使用到自己 RegisterPool。是否考虑给 gopool.go文件中init 函数加一个check，如果调用者没有 RegisterPool，在使用 defaultPool。还有就是整个capacity容量是固定的，是否可以加一些动态机制，在大流量的时候taskList中的足够多了，可以进行capacity扩容，当task少的时候capacity可以进行缩容。还望大佬指正观点

Hello!

I hope you are doing well!

We are a security research team. Our tool automatically detected a vulnerability in this repository. We want to disclose it responsibly. GitHub has a feature called Private vulnerability reporting, which enables security research to privately disclose a vulnerability. Unfortunately, it is not enabled for this repository.

Can you enable it, so that we can report it?

Thanks in advance!

PS: you can read about how to enable private vulnerability reporting here: https://docs.github.com/en/code-security/security-advisories/repository-security-advisories/configuring-private-vulnerability-reporting-for-a-repository

Question

ATT. At least the range of random floats should be given.

Operating System

mac

Go Version

go version go1.21.0 darwin/arm64

Package Version

latest

Affected Packages

gopkg/util/logger

Expected Behavior

预期输出实际的logger.CtxInfo 代码所在行数，参考截图中的差异

Actual Behavior

参考截图

Reproduction Steps

gopkg/util/logger/default.go
55行：ll.logger.Output(3, msg)

issue 描述：
第一个参数calldepth 写死了3，输出log永远都是 "logger.go 55 ***"

期望Fix：
支持customized or default value = 4

Other Information

No response

Summary

https://github.com/bytedance/gopkg/blob/614d0af6619bf612a797e07e6a1ac1533f529491/cloud/circuitbreaker/interface.go#L20C8-L20C8

circuitbreaker.Panel 接口可以添加一个 State(key string) State 的方法吗

Motivation

在某些场景下，我希望获取某个key关联的breaker的状态，来做不同的动作，比如进行重新进行调度

Explanation

No response

Question

context: I want to package this library for debian official archive but found the some tests failed to run in some environments,

refer to this link https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=995566 for more infomation

在计算bsr的时候，某些操作系统上使用一种for循环的方式计算，时间复杂度是O(n)，n是这个数第从小到大最大的为1的位。
经过golang sdk库math/bits/bits.go/Len函数的启发，我想出了一种算法，具有O（1）的复杂度, 并且计算时间稳定。
并且针对64位操作系统和32位操作系统有不同的优化。
几乎减少了百分之九十的计算时间。

Given that we are working on an opensource project, I'd like to propose to opensource the releasing process.

Maybe following work can be documented on Wiki:

• Release model
  • cadence: like milestones, maybe we can use the 'Projects' page of this repo
  • code quality of each release, how to prevent regression
• Steps
  • code, artifacts to update
  • documents to update
  • where to announce
• Roles
  • who is responsible of making a release?

Below simple PR of release does not behave elegantly like other opensource projects:
#26

Summary

......
// Similar:  group, ctxx := errgroup.WithCancelPool(ctx, pool) 
group, ctxx := errgroup.WithCancel(ctx) 
group.Go(func(ctxx)error)
if err := group.Wait(); err != nil { 
       return;
}
......

Motivation

gopool features (Auto-recovering Panics, Limit Goroutine Numbers, Reuse Goroutine Stack...) can make up for the lack of errgroup very well.

Explanation

No response

https://github.com/bytedance/gopkg/runs/2526939630

Question

请问有没有gopool和通过channel实现goroutine pool的性能对比，个人认为这个锁太大了：
1、 P的数量越多，channel的优势会优于锁
2、 如果goroutine创建速度快，锁等待会更高

感觉参照runtime，通过runtime.LockOSThread单独占几个系统线程，去处理这件事情（每次从全局的tasks中拿1/n的task），效率会更高，因为不会被runtime抢占。

Operating System

1

Go Version

1

Package Version

1

Affected Packages

1

Expected Behavior

1

Actual Behavior

1

Reproduction Steps

1

Other Information

No response

Question

Current go.mod use go1.16 with golang.org/x/sys v0.0.0-20221010170243-090e33056c14 package which introduced go1.17's unsafe.Slice in 20220910 (see history https://cs.opensource.google/go/x/sys/+/master:unix/syscall.go;bpv=1). This will cause build failed in go1.16 with following error messges.

# golang.org/x/sys/unix
../compile_path/pkg/mod/golang.org/x/[email protected]/unix/syscall.go:83:16: undefined: unsafe.Slice
../compile_path/pkg/mod/golang.org/x/[email protected]/unix/syscall_linux.go:2255:9: undefined: unsafe.Slice
../compile_path/pkg/mod/golang.org/x/[email protected]/unix/syscall_unix.go:118:7: undefined: unsafe.Slice
../compile_path/pkg/mod/golang.org/x/[email protected]/unix/sysvshm_unix.go:33:7: undefined: unsafe.Slice
note: module requires Go 1.17

Summary

asynccache只能基于时间进行过期，如果过期时间配置过长或者load数据过于庞大，会导致OOM的问题。
我想基于asynccache实现一个文件缓存功能，但是在使用过程中发现，一旦文件数据过大，程序会因为OOM问题而直接被系统kill掉。
是否可以像ristretto那样实现一个内存配额限制呢？

Motivation

建议

Explanation

No response

Summary

it seems that collection LSCQ use function cacheRemap16Byte to remap index used by SCQ, to do best effort to avoid false-sharing, maybe we could refer to folly UnboundedQueue's index, to remap index with less instructions.

however, i see no noticeable change in performance via BenchmarkDefault, perhaps because the function is not on the critical path.

Motivation

improve cacheRemap16Byte with less instructions

Explanation

Why it works?

for scqsize = 2^n, and k = 2*m + 1, define remap(i)=k*i % scqsize we can prove that: for any abs(i_1 - i_2) < scqsize, if i_1 != i_2, than remap(i_1) != remap(i_2).

this property makes remap() one-to-one map any i in [x, x + scqsize) to [0, scqsize).

Compare Assembly Code

package main

const (
	scqsize       = 1 << 16
	cacheLineSize = 64
)

func cacheRemap16ByteA(index uint64) uint64 {
	const cacheLineSize = cacheLineSize / 2
	rawIndex := index & uint64(scqsize-1)
	cacheLineNum := (rawIndex) % (scqsize / uint64(cacheLineSize))
	cacheLineIdx := rawIndex / (scqsize / uint64(cacheLineSize))
	return cacheLineNum*uint64(cacheLineSize) + cacheLineIdx
}

func cacheRemap16ByteB(index uint64) uint64 {
	return (index * 5) & (scqsize - 1)
}

func main() {}

        TEXT    main.cacheRemap16ByteA(SB), NOSPLIT|ABIInternal, $0-8
        FUNCDATA        $0, gclocals·g2BeySu+wFnoycgXfElmcg==(SB)
        FUNCDATA        $1, gclocals·g2BeySu+wFnoycgXfElmcg==(SB)
        FUNCDATA        $5, main.cacheRemap16ByteA.arginfo1(SB)
        FUNCDATA        $6, main.cacheRemap16ByteA.argliveinfo(SB)
        PCDATA  $3, $1
        MOVWLZX AX, CX
        ANDL    $2047, AX
        SHRQ    $11, CX
        SHLQ    $5, AX
        ADDQ    CX, AX
        RET
        TEXT    main.cacheRemap16ByteB(SB), NOSPLIT|ABIInternal, $0-8
        FUNCDATA        $0, gclocals·g2BeySu+wFnoycgXfElmcg==(SB)
        FUNCDATA        $1, gclocals·g2BeySu+wFnoycgXfElmcg==(SB)
        FUNCDATA        $5, main.cacheRemap16ByteB.arginfo1(SB)
        FUNCDATA        $6, main.cacheRemap16ByteB.argliveinfo(SB)
        PCDATA  $3, $1
        LEAQ    (AX)(AX*4), CX
        MOVWLZX CX, AX
        RET

Operating System

linux

Go Version

go1.18.6 linux/amd64
AND
go1.19.1 linux/amd64
AND
go latest

Package Version

latest

Affected Packages

util/gctuner

Expected Behavior

ALL TEST PASSED

Actual Behavior

--- FAIL: TestTuner (0.20s)
    tuner_test.go:37: 
                Error Trace:    tuner_test.go:37
                Error:          Not equal: 
                                expected: 0x1f4
                                actual  : 0x32
                Test:           TestTuner
FAIL
exit status 1
FAIL    github.com/bytedance/gopkg/util/gctuner 0.217s

Reproduction Steps

cd util/gctuner && go test -count=1
OR
cd util/gctuner && go test -count=10

Other Information

@joway PTAL, thanks!

Question

我看代码里的配置是在初始化熔断器时传入的，怎么才能在运行时动态的修改相关配置选项

Summary

QuoteSingle("a") -> 'a'
QuoteSingle("abc") -> 'abc'
QuoteSingle("ab'c") -> 'ab'c'

Motivation

wu

Explanation

No response

Question

possible to show an example of passing value / reference?

nice work by the way, i've tested against
https://github.com/panjf2000/ants

and found this to be faster but not sure if there are any significant difference between the two?