Giter VIP home page Giter VIP logo

hcache's Introduction

hcache - a tool fork from pcstat, with a feature that showing top X biggest cache files globally

The pcstat tool can get page cache statistics for one file by the file's name, or all cached files for a process by the process's pid.

However, I often meet with a question: I know os cached memories by cat /proc/meminfo or free, but I can't find out those BIG files which is being cached. Of course I can use the ps aux command to get those processes who used a lot of memory and get the details by the pcstat --pid [pid], but pcstat does not sort the result, and it is not convenient.

So I add a feature to pcstat: you can use the option --top [X] to show the top X biggest cached files globally. After that, you can use lsof to find out the bad guy.

Download

I also build a bin file. You can download it from here, have a try! I have tested it on centos7.2 and ubuntu 16.04.

Usage

hcache has the same options that is used by pcstat, and a new option --top [X]. Attention, you CANNOT use both -pid and -top.

hcache <-json <-pps>|-terse|-default> <-nohdr> <-bname> file file file
 -json output will be JSON
   -pps include the per-page information in the output (can be huge!)
 -terse print terse machine-parseable output
 -default print ascii tables
 -histo print a histogram using unicode block characters
 -nohdr don't print the column header in terse or default format
 -bname use basename(file) in the output (use for long paths)
 -plain return data with no box characters
 -unicode return data with unicode box characters
 -pid int show all open maps for the given pid
 -top int show top x cached files in descending order

Examples

$ sudo hcache --top 3
[sudo] password for silenceshell: 
+-------------------------------------------------+----------------+-------------+----------------+-------------+---------+
| Name                                            | Size           │ Pages       │ Cached Size    │ Cached Pages│ Percent │
|-------------------------------------------------+----------------+-------------+----------------+-------------+---------|
| /opt/apps/cn.google.chrome/files/chrome         | 170.811M       | 43728       | 120.652M       | 30887       | 70.634  |
| /opt/apps/com.visualstudio.code/files/code/code | 125.409M       | 32105       | 92.593M        | 23704       | 73.833  |
| /usr/lib/i386-linux-gnu/libLLVM-11.so.1         | 74.384M        | 19043       | 48.057M        | 12303       | 64.606  |
|-------------------------------------------------+----------------+-------------+----------------+-------------+---------|
│ Sum                                             │ 370.604M       │ 94876       │ 261.301M       │ 66894       │ 70.507  │
+-------------------------------------------------+----------------+-------------+----------------+-------------+---------+
$ 
$ sudo ./hcache --top 3  --bname  
+-----------------+----------------+-------------+----------------+-------------+---------+
| Name            | Size           │ Pages       │ Cached Size    │ Cached Pages│ Percent │
|-----------------+----------------+-------------+----------------+-------------+---------|
| chrome          | 170.811M       | 43728       | 122.030M       | 31240       | 71.442  |
| code            | 125.409M       | 32105       | 92.913M        | 23786       | 74.088  |
| libLLVM-11.so.1 | 74.384M        | 19043       | 44.486M        | 11389       | 59.807  |
|-----------------+----------------+-------------+----------------+-------------+---------|
│ Sum             │ 370.604M       │ 94876       │ 259.430M       │ 66415       │ 70.002  │
+-----------------+----------------+-------------+----------------+-------------+---------+
$ 
$ lsof /usr/lib/x86_64-linux-gnu/libQtWebKit.so.4.10.2 
COMMAND    PID   USER  FD   TYPE DEVICE SIZE/OFF    NODE NAME
quiterss 20630 silenceshell mem    REG    8,5 36462184 3936610 /usr/lib/x86_64-linux-gnu/libQtWebKit.so.4.10.2

Building

hcache needs go version > 1.12 for go mod

git clone https://github.com/silenceshell/hcache.git
cd hcache
make build
sudo cp hcache /usr/local/bin/ 

Requirements

Go 1.4 or higher and golang.org/x/sys/unix.

From the mincore(2) man page:

  • Available since Linux 2.3.99pre1 and glibc 2.2.
  • mincore() is not specified in POSIX.1-2001, and it is not available on all UNIX implementations.
  • Before kernel 2.6.21, mincore() did not return correct information some mappings.

Author

silenceshell

License

Apache 2.0

Thanks to

@tobert for pcstat and @mitchellh for go-ps

hcache's People

Contributors

movie0125 avatar qinzhiben11 avatar ruishawn avatar silenceshell avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

hcache's Issues

top命令无法正常运行

你好,在使用hcache时,发现工具里pid,help等功能都可以正常使用,但top无法使用

image

不知道问题出在哪里?谢谢

建议增加输出统计功能

我在用你开发 Linux Cache 查看工具 hcache,很好用的一个工具。
希望能增加统计总量的功能,类似下面的样子。
image

go build failed

root@qhm-PC:~/test/hcache# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS"

root@qhm-PC:~/test/hcache# go version
go version go1.12.4 linux/amd64
root@qhm-PC:~/test/hcache# export GOPATH=/root/test/hcache
root@qhm-PC:~/test/hcache# go build
$GOPATH/go.mod exists but should not
root@qhm-PC:~/test/hcache# ls
formats.go  go.mod  go.sum  main.go  pkg  process.go  process_linux.go  process_unix.go  README.md  src  winsize.go

我对go的编译不是很熟悉,不清楚上面这个错误是上面意思,go.mod是我执行了
go get github.com/tobert/pcstat
之后出现的

KVM宿主机占用缓存较大,hcache可能无法定位?

1、服务器为KVM宿主机,启动了多个KVM虚拟机。

free -h
total used free shared buff/cache available
Mem: 251G 67G 770M 4.1G 183G 179G
Swap: 0B 0B 0B

2、 以下是./hcache --top 10 ,但是缓存占用在183G,其中是不是有其他比较大的文件缓存,但是通过hcache无法确认?有没有办法确认cache中的内容,谢谢。

3、/proc/meminfo信息

MemTotal: 263566160 kB
MemFree: 2093520 kB
MemAvailable: 169410064 kB
Buffers: 0 kB
Cached: 168794964 kB
SwapCached: 604572 kB
Active: 160983136 kB
Inactive: 94351416 kB
Active(anon): 78603948 kB
Inactive(anon): 8031548 kB
Active(file): 82379188 kB
Inactive(file): 86319868 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 15204348 kB
SwapFree: 13583428 kB
Dirty: 20 kB
Writeback: 0 kB
AnonPages: 85955084 kB
Mapped: 63308 kB
Shmem: 95256 kB
Slab: 4086124 kB
SReclaimable: 3335448 kB
SUnreclaim: 750676 kB
KernelStack: 9584 kB
PageTables: 181892 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 146987428 kB
Committed_AS: 90918056 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 901844 kB
VmallocChunk: 34224481256 kB
HardwareCorrupted: 0 kB
AnonHugePages: 79038464 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0

build error in arm64 ubuntu20.04

image
image

cpu 和系统信息如上图所示,报错信息如下:

image

修改了 Makefile
'''
ubuntu@166:~/xiaoping.chen/hcache-master$ cat Makefile
build:
GOPROXY=https://goproxy.io GOARCH="arm64" GOOS="linux" go build
'''

期待您的回复,谢谢。

can't find files using most cache by hcache top

hcache top bug

The result of the hcache --top does not have /root/rui/re

[root@clickhouse02 hcache]# du -sh /root/rui/re
445M	/root/rui/re

[root@clickhouse02 hcache]# cat /root/rui/re >/dev/null
[root@clickhouse02 hcache]# ./hcache /root/rui/re
+--------------+----------------+-------------+----------------+-------------+---------+
| Name         | Size           │ Pages       │ Cached Size    │ Cached Pages│ Percent │
|--------------+----------------+-------------+----------------+-------------+---------|
| /root/rui/re | 444.798M       | 113869      | 444.798M       | 113869      | 100.000 |
|--------------+----------------+-------------+----------------+-------------+---------|
│ Sum          │ 444.798M       │ 113869      │ 444.798M       │ 113869      │ 100.000 │
+--------------+----------------+-------------+----------------+-------------+---------+

[root@clickhouse02 hcache]# ./hcache --top 5
+-------------------------------------------------------------------------------------------------------------------------------------+----------------+-------------+----------------+-------------+---------+
| Name                                                                                                                                | Size           │ Pages       │ Cached Size    │ Cached Pages│ Percent │
|-------------------------------------------------------------------------------------------------------------------------------------+----------------+-------------+----------------+-------------+---------|
| /usr/bin/clickhouse                                                                                                                 | 435.936M       | 111600      | 54.644M        | 13989       | 12.535  |
| /run/log/journal/ef219b153e8049718c374985be33c24e/system@839b2a266700434097a56aba4bfcce81-0000000000000001-0005d9e858986d57.journal | 32.000M        | 8192        | 25.750M        | 6592        | 80.469  |
| /run/log/journal/ef219b153e8049718c374985be33c24e/system@839b2a266700434097a56aba4bfcce81-00000000000068d4-0005dc4caf5c4183.journal | 32.000M        | 8192        | 24.145M        | 6181        | 75.452  |
| /run/log/journal/ef219b153e8049718c374985be33c24e/system@839b2a266700434097a56aba4bfcce81-000000000000c75d-0005deb0fba8437a.journal | 32.000M        | 8192        | 22.277M        | 5703        | 69.617  |
| /run/log/journal/ef219b153e8049718c374985be33c24e/system@839b2a266700434097a56aba4bfcce81-0000000000018b6b-0005e5de04567c2f.journal | 24.000M        | 6144        | 17.414M        | 4458        | 72.559  |
|-------------------------------------------------------------------------------------------------------------------------------------+----------------+-------------+----------------+-------------+---------|
│ Sum                                                                                                                                 │ 555.936M       │ 142320      │ 144.230M       │ 36923       │ 25.944  │
+-------------------------------------------------------------------------------------------------------------------------------------+----------------+-------------+----------------+-------------+---------+

hcache -top 5的结果和 直接 hcache file的结果让人疑惑

我先是 ./hcache -top 5 了一个结果,我看到最大的文件是大概23M,
image

之后我用 dd命令生成了一个23M的文件 good
然后我看到这个good这个文件被缓存了100%
image

接着,我再次使用 ./hcache -top 5命令,没有在里面找到 good这个文件,我感觉很奇怪
image

arm类型linux机器 bin 文件获取问题

当前 readme文件中的Download 给出的bin 文件不能在arm类型的机器下使用,看起来是x86的

在Building 的 make build 这一步,出现
go: github.com/tobert/[email protected]: Get "https://gocenter.io/github.com/tobert/pcstat/@v/v0.0.0-20170528024208-91a7346e5b46.mod": dial tcp 35.230.74.213:443: i/o timeout
经过分析,应该是build的机器需要连接互联网实际上却连不上导致。

那么请问如何获取 arm类型linux机器 bin 文件, 或者如何在本地build(连不上互联网的情况下)

服务器离线状态下如何安装hcache?

尝试离线下安装,发现需要下载依赖文件,安装失败;下载提供的二进制执行文件,上传到服务器后执行相关命令报错

当前操作系统为麒麟V10,架构为linux aarch64

It is recommended to increase the cache size column

The current output cache is the number of pages, and it needs to be converted to calculate the real cache size
It is recommended to increase the cache size by one column

I will submit related PRs recently

hcache does not show the files which are open by some process and cached in pagecache

Hcache can't show the files which are opened by some process and cached in pagecache.

# pcstat /data/nvme0n1/oradata/ORAINTEL/EEE54678E49E7CCAE053086EA8C0B9D1/datafile/o1_mf_bmsqlts_krol4w3h_.dbf
|------------------------------------------------------------------------------------------------------+----------------+------------+-----------+---------|
| Name                                                                                                 | Size           | Pages      | Cached    | Percent |
|------------------------------------------------------------------------------------------------------+----------------+------------+-----------+---------|
| /data/nvme0n1/oradata/ORAINTEL/EEE54678E49E7CCAE053086EA8C0B9D1/datafile/o1_mf_bmsqlts_krol4w3h_.dbf | 53687099392    | 13107202   | 12495372  | 095.332 |
|------------------------------------------------------------------------------------------------------+----------------+------------+-----------+---------|

# hcache -top 100 |grep dbf
(none)

I guess the reason is that it gets the files from /proc/{pid}/maps ,not from /proc/{pid}/fd .

I can't get it.

what's the meaning of the outputing ?

root@sc-demo:tmp$ hcache  -top 60
+------------------------------------------------------------+----------------+------------+-----------+---------+
| Name                                                       | Size (bytes)   | Pages      | Cached    | Percent |
|------------------------------------------------------------+----------------+------------+-----------+---------|
| /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar                  | 66159478       | 16153      | 11144     | 068.990 |
| /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so  | 16995339       | 4150       | 3715      | 089.518 |
| /usr/lib/jvm/java-8-oracle/jre/lib/ext/jfxrt.jar           | 18490072       | 4515       | 864       | 019.136 |
| /usr/lib/jvm/java-8-oracle/jre/lib/charsets.jar            | 3135391        | 766        | 766       | 100.000 |
| /bin/hcache                                                | 2542252        | 621        | 621       | 100.000 |

I cant understand the meaning of the Size,Page,Cached,Percent!
SIze: Is the file size?
Pages:?
Cache:? bytes?
Percent: /?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.