mosn / holmes Goto Github PK

RT

now, we can only write the profile on the local filesystem.
it's better to upload them to a central platform so that we can get real-time alerts based on the center platform, and we can analyze them on the central platform.

design:
we can add a new profile callback option, call the callback on each write profile file if it has been set.

callback API maybe:

callback(type, filename, reason)

自己项目中有用到consul版本为v1.4.3，holmes中的consul/api的版本为v1.3.0

去看holmes代码中其实并没用用到consul，去追依赖发现依赖层级如下

mosn.io/pkg-> github.com/dubbogo/gost -> github.com/prometheus/client_golang  -> github.com/prometheus/common ->  github.com/go-kit/kit -> consul/api v1.3.0

本身holmes中并没有任何consul的依赖，即使是使用的日志库holmes.io/pkg/log的功能中，也没有用到consul/api的功能。

不知是否可以把moson.io/pkg/log中有关consul的依赖去除，要不可惜了这么优秀的开源不能使用。

 type ReporterImpl struct{}
    func (r *ReporterImple) Report(pType string, buf []byte, reason string, eventID string) error{
        // do something	
    }

ReporterImple && ReporterImpl

investigate for deadlock detection,

if plenty of goroutines block on lock acquire, is it a deadlock?

if goroutines stack matches some rules, is it a deadlock?

if runtime_SemacquireMutex numbers keep increasing, is it a deadlock

support viewcore and make it better to use.
https://github.com/golang/debug/tree/master/cmd/viewcore

Hi team,

In my opinion, there is should be a dangerous_limit parameter means holmes will not dump profile when current CPU usage reached this limit, cuz CPU pprof usually waste some resource, commonly 5% or less, if holmes executes the CPU pprof causes CPU usage up 5% and result of the service crash, I won't hope that.

func WithCPUDump(min int, diff int, abs int)

change to

func WithCPUDump(min int, diff int, abs int, dangerous int)

or add a new withOption func

func WithDangerousLimit(d int)

I prefer the latter.

for auto rotated log deletion, the file name should support customization

eg .

text.2020-01-02, text.20201102

最近遇到流量比较大的模块，可能会出现too many files句柄打满错误。其中一个原因是物理机句柄限制。另外一个，由于是偶现，抓不到现场，现在还不知道原因。。所以这个工具是否可以帮助我们抓到这个现场，定位到具体的问题。

we only create the finCh in holmes.New, and will also it in finalizerCallback when holmes is stoppen.

@songzhibin97 @Jun10ng Thoughts?

var buf bytes.Buffer
_ = pprof.Lookup("xxx").WriteTo(&buf, int(h.opts.DumpProfileType))
几秒就要运行一次
此处的 buffer是否可以搞成复用呢，不然这样不停的分配，又不停的回收, 这会不会成为性能点，
可能理解的不对

What version of Go are you using (go version)?

$  1.17.8

Does this issue reproduce with the latest release?

Haven't tried.

What operating system and processor architecture are you using (go env)?

$ MacOS and amd64

What did you do?

I wrote a piece of code:

var (
    default = 1073741824
    a []byte
)

func MakeNGbSlice(ctx *gin.Context) {
    sizeStr ,ok := ctx.GetQuery("size")
    if ok {
        size, _ := strconv.Atoi(sizeStr)
        if size > 0 {
            defaultSize = size * defaultSize
        }
    }

    a = make([]byte, 0, defaultSize)
    for i := 0; i < defaultSize; i ++ {
        a = append(a, byte('a'))
    }
    time.Sleep(time.Second * 10)
    a = nil // for gc
}

Then I curled the api to trigger the code to run.

What did you expect to see?

1. The RSS my app used went up to 1GB.
2. The heap profile data was right and could help me find out the reason why RSS went up after I dumped the heap samples.

What did you see instead?

1. The RSS my app used went up to 1GB.
2. The heap profile seemed to be weird:

Type: inuse_space
Time: Aug 3, 2022 at 2:32pm (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 12052.56kB, 64.38% of 18721.44kB total
Showing top 10 nodes out of 153
      flat  flat%   sum%        cum   cum%
 2562.81kB 13.69% 13.69%  3075.02kB 16.43%  runtime.allocm
 2048.81kB 10.94% 24.63%  2048.81kB 10.94%  runtime.malg
 2048.19kB 10.94% 35.57%  2048.19kB 10.94%  github.com/Shopify/sarama.(*TopicMetadata).decode
 1097.69kB  5.86% 41.44%  1097.69kB  5.86%  github.com/Shopify/sarama.(*client).updateMetadata
 1089.33kB  5.82% 47.26%  1089.33kB  5.82%  google.golang.org/grpc/internal/transport.newBufWriter
// ...

3. After about 2 minutes, I profiled again and the result was what I expected:

Type: inuse_space
Time: Aug 3, 2022 at 2:33pm (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 1024.50MB, 98.25% of 1042.78MB total
Dropped 154 nodes (cum <= 5.21MB)
Showing top 10 nodes out of 18
      flat  flat%   sum%        cum   cum%
    1024MB 98.20% 98.20%     1024MB 98.20%  .../examples.Make1GbSlice
    0.50MB 0.048% 98.25%     6.63MB  0.64%  runtime.main
         0     0% 98.25%  1024.51MB 98.25%  github.com/gin-gonic/gin.(*Context).Next (inline)
         0     0% 98.25%  1024.51MB 98.25%  github.com/gin-gonic/gin.(*Engine).ServeHTTP
         0     0% 98.25%  1024.51MB 98.25%  github.com/gin-gonic/gin.(*Engine).handleHTTPRequest

So, it seems like that the samples pprof dumps in time are wrong.

For more discussion, Please see golang issue #54233

We found the test case for the WithCollectInterval function failure of some times. so I dived into it, the root of the failure of the test case is:

1. select-case has no order, it will pick a random case to execute.
2. the logic after ticker.C takes more than 1 second.

so we would set priority for ticker.C and intervalResetting , making sure that the message inside intervalResetting channel would be consumed before ticker.C

In the memory leak scenario, usually, find the leaking place through two profiles from different times.

The feature of dumping profile from abnormal time holmes has support already.
So, I thought we need a feature of dumping from normal times, and that can be set like a cron job.

https://github.com/avelino/awesome-go

Hi team,

I saw there was an issue about "dynamic configuration", and I want to discuss some questions about it.

1. The definition of "dynamic" is holmes pull/receive configurations from a remote server? or holmes can adjust its configuration automatically through pre-prepared plans？
2. If its definition is the former, holmes receive configuration from Apollo/Redis/API or some service built by users themselves, we better draft an abstract API, and implement dynamic config feature based on the API.
3. Does we support hot-load configuration? I mean weather holmes supports modifying its config when it is running? Or just on a simple way-- only load configuration from remote when holmes is initiating. The change to support the former seems great.

如图，dump文件名字第一个字段后有两个点，请问这里是一个问题还是说这里应该填写用户指定的字符串。如果是用户指定字符串，如何指定没看到相关API

1、我看完代码感觉ring的作用只是计算平均值。那么为啥没有用队列，添加元素的时候从队头删除一个，队尾添加一个。

2、如果没有用队列的原因是队列没有像ring那样严格的数量限制容易出现问题，那么为啥没有封装一下官方库提供的ring，而是自己写了一个。自己写的ring里面idx的最大值等于len(ring.data)，又有ring.data[ring.idx]的操作，感觉这个有点危险。

我看到cgroup场景下的getUsage方法复杂了很多，请问是有什么考量吗。

Currently, the goroutines are dumped into specified file path, sometimes it might be hard for us to check.
Is it possible to support to dump goroutines to logger? If so we can see check it in our existing log system(ELK). Thanks.

we create logger based on the file in holmes, it makes holmes easier to use standalone.
but sometimes, we already have logger, we'd better support just use the existing logger instead create a new one.
it will make holmes easier to integrate.

user can easily find where the file is, or they have to read the code to know.

Now, the previous data is weird, not in human-readable order.
79 and 70 should be the last value.

2022-09-26 17:00:02,934 [ERROR] [holmes.cpu] [Holmes] pprof dump cpu, config_min : 30, config_diff : 25, config_abs : 100, config_max : 0, previous : [13 13 13 13 79 13 13 14 14 13], current: 79
2022-09-26 17:11:02,938 [ERROR] [holmes.cpu] [Holmes] pprof dump cpu, config_min : 30, config_diff : 25, config_abs : 100, config_max : 0, previous : [16 12 13 12 13 13 70 13 13 13], current: 70

This is due to, we print the data from a ringbuffer directly, we should print them in nature order:
https://github.com/mosn/holmes/blob/master/ring.go#L21

Maybe we can introduce a data method for ring, which returns the data in the natural order, so that we can use data() instead of ring.data.

I run holmes inside my application and set the follow options.

       // other initial


	h, _ := holmes.New(
		holmes.WithCollectInterval("5s"),
		holmes.WithCoolDown("1m"),
		holmes.WithDumpPath("/tmp"),
		holmes.WithCPUDump(1, 25, 80),
		holmes.WithCPUMax(90),
	)
	h.EnableCPUDump()

	// start the metrics collect and dump loop
	h.Start()

      // server start

then I run a eating cpu shell script, and the following screenshot is top command output, CPU usage rate is almost 100%.

but I check holmes.log it shows CPU usage rate is 0%

https://github.com/grantseltzer/weaver

In (*Holmes).EnableDump, the current cpu percent shouldn't be assigned with cpu but curCPU. Like the following code:

func (h *Holmes) EnableDump(curCPU int) (err error) {
	if h.opts.CPUMaxPercent != 0 && curCPU >= h.opts.CPUMaxPercent {
		return fmt.Errorf("current cpu percent [%v] is greater than the CPUMaxPercent [%v]", curCPU, h.opts.CPUMaxPercent)
	}
	return nil
}

看 readme 提及的意思是，为了解决的问题是被 OOM-Killer kill 掉而无法保留现场的问题
那么是否只监控物理机或容器的内存使用率即可，而无需监控 cpu 利用率和 goroutine 数量？

When using dump path that does not exist, for example

holmes.WithDumpPath("./tmp"),

Expected behavior:
The goroutine should be dumped successfully.

Current behavior:
Will encounter error like

2022-03-29 17:07:06,609 [ERROR] failed to write profile to file(tmp/goroutine..20220329170706.609.log), err: pprof goroutine open file failed : open tmp/goroutine..20220329170706.609.log: no such file or directory

Source Code

package main

import (
	"fmt"
	"runtime"
	"time"

	"mosn.io/holmes"
	mlog "mosn.io/pkg/log"
)

func main() {
	logger := holmes.NewStdLogger()
	logger.SetLogLevel(mlog.INFO)

	h, _ := holmes.New(
		holmes.WithCollectInterval("1s"),

		holmes.WithTextDump(),
		holmes.WithDumpPath("./tmp"),
		// dump will happen when current_goroutine_num > 500 && current_goroutine_num < 1500
		holmes.WithGoroutineDump(500, 0, 500, 1500, 40*time.Second),
		holmes.WithLogger(logger),
	)
	h.EnableGoroutineDump()
	h.Start()

	spawnGoroutine(490)

	for {
		fmt.Println(time.Now(), "Number of goroutines:", runtime.NumGoroutine())
		spawnGoroutine(10)
		time.Sleep(10 * time.Second)
	}
}

func spawnGoroutine(n int64) {
	for i := int64(0); i < n; i++ {
		go func() {
			time.Sleep(500 * time.Minute)
		}()
	}
}

go.mod

module test_holmes

go 1.16

require (
	mosn.io/holmes v1.0.0
	mosn.io/pkg v0.0.0-20211217101631-d914102d1baf
)

During my use, I found that there was a code issue in the v1.0.1 and v1.1.0 package, but there was no problem with the code here in the master branch of GitHub. Should type a new version of the tag？
holmes.go：
Error code:
func (h *Holmes) DisableMemDump() *Holmes { h.opts.gCHeapOpts.Enable = false return h }
Correct code:
func (h *Holmes) DisableMemDump() *Holmes { h.opts.memOpts.Enable = false return h }

鉴于线上告警场景的需求，如果能够将指标当前值、前n-1次采样的平均值以及指标相关的配置(min、max、abs、diff等)作为Report的参数传进来就好了。
目前参数中就只有reason能用，并且内容可读性不强。

I found there are a lot of different meaning variables but they have the same name & type.
And one of them is the global variable.
I think that is unsafe, for example, if I change the func signature at line 270 to cpuCheckAndDump(cpuUsage int)
, then everything is ok, the cpu variable at line 280 will be recognized as the global variable quietly,
IDE/Editor/Compiler ever have no warning. (also same about mem)

In my opinion, It's better to change the name of variables in consts.go
from cpu to cpuType

代码：
h, err := holmes.New(
holmes.WithProfileReporter(r),
holmes.WithCollectInterval("30s"),
holmes.WithDumpPath(pprofDumpPath),
holmes.WithLogger(holmes.NewFileLog(holmesLogFilePath, mlog.INFO)),
holmes.WithCPUDump(30, 50,60, time.Minute),
holmes.WithCPUMax(90),
holmes.WithCGroup(true),
)
h.EnableCPUDump()

// start the metrics collect and dump loop
h.Start()

holmes log

pprof_file

看起来感觉cpu突然飙高了一下，然后又正常了，是采集周期或者参数设置的问题吗

filename is more flexible, people can read bytes from the file, or just report the filename.

it's a breaking change, but I think it's worthing.

Some users want to use holmes on their production env and hesitate for the license and beta ver, currently most features of holmes have already been stable enough

v1 should be soon released,

another question is what license should this lib be? @nejisama @doujiang24 @taoyuanyuan
apache v2?

输出的log文件中会打印出设置的配置和是否进行了dump,但是没看懂后面这两个代表什么意思，是内存吗？

• we can get allgs by go:linkname demo

• can print all goroutines with goroutine id by debug/pprof/goroutine?debug=2

So we can output every goroutine stack size

Split the stats collect function, if only memory dump is enabled, don’t collect stats other than rss

since thread check dump already contains a goroutine profile.

Feel free to assign yourself(or request to assign to you.) if you are interested.

// DisableMemDump disables the mem dump.
func (h *BHolmes) DisableMemDump() *BHolmes {
h.opts.gCHeapOpts.Enable = false
return h
}
是不是应该是
// DisableMemDump disables the mem dump.
func (h *BHolmes) DisableMemDump() *BHolmes {
h.opts.memOpts.Enable = false
return h
}

win环境下，enable holmes，dump之后查看文本，出现乱码，不确定是不是bug

类似这个 golang/go#40974

goland里面执行gotest没有问题。

项目里go install 命令的时候出现以下问题：
/usr/local/Cellar/go/1.15.2/libexec/pkg/tool/darwin_amd64/link: running clang failed: exit status 1
ld: sectionForAddress(0x13654DE) address not in any section file '/var/folders/rr/2rdmy50d1jb0tjnjfhcbnbqh0000gn/T/go-link-273425566/go.o' for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

升级go版本至1.15.5后问题消失