Hi,

JFYI, I was not able to build the project without specifying version constraint in Gopkg.toml:

[[constraint]]
  name = "github.com/simagix/keyhole"
  branch = "master"

Maybe something related with my golang installation, but dep was constantly trying to fetch v0.3.0 keyhole version without that constraint.

this 172.217.160.113 not ping
[root@vmt30129 keyhole-master]# ./build.sh
go: github.com/simagix/[email protected]: missing go.sum entry; to add it:
go mod download github.com/simagix/gox
[root@vmt30129 keyhole-master]# go mod download github.com/simagix/gox
go: github.com/simagix/[email protected]: Get "https://proxy.golang.org/github.com/simagix/gox/@v/v0.2.1-0.20210204133338-1ac49535b8c0.mod": dial tcp 172.217.160.113:443: i/o timeout

There's a Find() call when gathering data on the collections that fails because "Must use indexes" is turned on in the config.

This is more a question than an issue. I am trying to setup keyhole on my Mac to diagnose a performance alert I got from ATLAS. I followed the instructions and got as far as Grafana is installed correctly and the data source for keyhole has been setup. However, the next step in the docs states that I need a copy of the diagnostic.data directory. Yet I could not find a way to download this data form an ATLAS cloud cluster. Is there a way to do this somehow?

TTL index was identified if expireAfterSeconds is zero.

keyhole --web --span 1 --diag /data/db/diagnostic.data
flag provided but not defined: -span
simagix/keyhole v1.0.1-20201026

It appears that "keyhole" has trouble connecting to an Atlas cluster

Given a MDB URI:

atlas_uri=$(echo mongodb+srv://${UN}:${PW}@sghtest2-pri.qurxw.mongodb.net/test)

where UN and PW are set to the username and password respectively

keyhole fails to connect

$ ./keyhole -simonly $atlas_uri
2021/02/04 22:11:08 error parsing uri: lookup <server-redacted> on 127.0.0.53:53: cannot unmarshal DNS message

Yet the mongo shell connects just fine

$ mongo --quiet $atlas_uri --eval 'db.version()' | grep -v NETWORK
4.0.22

The keyhole version is

$ ./keyhole -version
simagix/keyhole v1.0.4-20210126

@simagix

I followed this link https://www.mongodb.com/blog/post/peek-at-your-mongodb-clusters-like-a-pro-with-keyhole-part-2 to setup ftdc and ran

C:\pfg>keyhole-win-x64.exe --web --diag "C:\bsuk\mongo perf"
2021/08/02 21:42:19 reading 20 files with 1 second(s) interval
2021/08/02 21:42:23 metrics.2021-07-26T01-23-14Z-00000 blocks: 34 , time: 3.950505s Memory Alloc = 6618 MiB, TotalAlloc = 10840 MiB
2021/08/02 21:42:41 metrics.2021-07-27T05-17-41Z-00000 blocks: 135 , time: 22.0620114s Memory Alloc = 14235 MiB, TotalAlloc = 28476 MiB
2021/08/02 21:42:42 metrics.2021-07-26T04-11-43Z-00000 blocks: 141 , time: 22.1586125s Memory Alloc = 14342 MiB, TotalAlloc = 28582 MiB
2021/08/02 21:42:42 metrics.2021-07-28T06-33-31Z-00000 blocks: 138 , time: 22.6116106s Memory Alloc = 14957 MiB, TotalAlloc = 29197 MiB
2021/08/02 21:42:44 metrics.2021-07-24T05-09-42Z-00000 blocks: 156 , time: 25.0781908s Memory Alloc = 18261 MiB, TotalAlloc = 32501 MiB
2021/08/02 21:42:49 metrics.2021-07-26T15-25-14Z-00000 blocks: 166 , time: 29.3671965s Memory Alloc = 20947 MiB, TotalAlloc = 35187 MiB
2021/08/02 21:42:50 metrics.2021-07-27T16-30-43Z-00000 blocks: 168 , time: 30.2089482s Memory Alloc = 21593 MiB, TotalAlloc = 35834 MiB
2021/08/02 21:42:50 metrics.2021-07-28T18-05-19Z-00000 blocks: 152 , time: 27.0083984s Memory Alloc = 22060 MiB, TotalAlloc = 36301 MiB
2021/08/02 21:42:51 metrics.2021-07-25T10-55-18Z-00000 blocks: 173 , time: 31.4579756s Memory Alloc = 22351 MiB, TotalAlloc = 36591 MiB
2021/08/02 21:42:59 metrics.2021-07-22T19-37-10Z-00000 blocks: 198 , time: 39.4435651s Memory Alloc = 19075 MiB, TotalAlloc = 39185 MiB
2021/08/02 21:43:00 metrics.2021-07-24T18-11-57Z-00000 blocks: 200 , time: 40.6473015s Memory Alloc = 18287 MiB, TotalAlloc = 39557 MiB
2021/08/02 21:43:01 metrics.2021-07-23T12-10-50Z-00000 blocks: 203 , time: 42.0635971s Memory Alloc = 16436 MiB, TotalAlloc = 39843 MiB
2021/08/02 21:43:12 metrics.2021-07-29T06-47-24Z-00000 blocks: 138 , time: 30.1929673s Memory Alloc = 13486 MiB, TotalAlloc = 43133 MiB
2021/08/02 21:43:16 metrics.2021-07-30T06-41-35Z-00000 blocks: 138 , time: 33.814659s Memory Alloc = 14283 MiB, TotalAlloc = 43930 MiB
2021/08/02 21:43:16 metrics.2021-07-29T18-19-42Z-00000 blocks: 148 , time: 34.9524538s Memory Alloc = 14388 MiB, TotalAlloc = 44034 MiB
2021/08/02 21:43:41 metrics.2021-07-30T18-13-14Z-00000 blocks: 150 , time: 56.6814478s Memory Alloc = 7598 MiB, TotalAlloc = 48783 MiB
2021/08/02 21:43:44 metrics.2021-07-31T06-45-18Z-00000 blocks: 137 , time: 55.0562283s Memory Alloc = 7977 MiB, TotalAlloc = 49162 MiB
2021/08/02 21:43:46 metrics.2021-08-01T06-44-18Z-00000 blocks: 137 , time: 55.4096129s Memory Alloc = 8202 MiB, TotalAlloc = 49388 MiB
2021/08/02 21:43:48 metrics.2021-07-31T18-12-15Z-00000 blocks: 150 , time: 58.4966293s Memory Alloc = 8409 MiB, TotalAlloc = 49594 MiB
2021/08/02 21:43:49 metrics.2021-08-01T18-11-17Z-00000 blocks: 148 , time: 57.7626193s Memory Alloc = 8424 MiB, TotalAlloc = 49609 MiB
2021/08/02 21:43:55 20 files loaded, time spent: 1m35.2600374s

Stats from 2021-07-23T01:22:11Z to 2021-08-02T12:18:16Z
2021/08/02 21:43:55 host-0 ww7-mn1-a:27017
2021/08/02 21:43:55 host-1 ww7-mn1-p:27017
2021/08/02 21:43:55 host-2 ww7-mn1-s:27017
2021/08/02 21:44:16 http://localhost:3000/d/simagix-grafana/mongodb-mongo-ftdc?orgId=1&from=1626982631000&to=1627885996000

But when I go to the above link http://localhost:3000/d/simagix-grafana/mongodb-mongo-ftdc?orgId=1&from=1626982631000&to=1627885996000
Cannot see any data

When analyzing performance issues, it's important to see also how many Bytes were transferred, because even if a query uses an index, it can have a negative impact on database performance if large amounts of data are transferred.
This metric is logged as reslen in mongodb's logfiles. Keyhole should retrieve these values and Maobi should show the summed up value in the table 2.2 Slow Ops Pattern in a column sum reslen and maybe also avg reslen.

Hey,

Is there any reason why Disk Utilization is calculated this way?
https://github.com/simagix/keyhole/blob/master/web/time_series_data.go#L229

Seems like it produces wrong values. The proper formula would be to divide IOTimeMS diff by total delta time like this:
u := float64(100 * float64(disk.IOTimeMS - pstat.Disks[k].IOTimeMS) / (float64(stat.Start.Sub(pstat.Start)) / float64(time.Millisecond)))

Sysstat calculates it like this here: https://github.com/sysstat/sysstat/blob/master/rd_stats.c#L372

I was able to run Maobi last week, but a failed Docker update forced me to uninstall/reinstall and now I can't find the image:

Error response from daemon: pull access denied for simagix/maobi, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

Is this accidental, or have you pulled the image from Docker Hub?

MongoDB version 3.2.12
there are some slow queries in my log file

2021-07-27T00:00:32.837592+08:00 [conn11837] command monitor.$cmd command: update { update: "process_snapshot_v1", updates: 85, ordered: false, shardVersion: [ Timestamp 45458000|0, ObjectId('5836a370dd42ce3bf562ebe9') ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:2309 locks:{ Global: { acquireCount: { r: 172, w: 172 } }, Database: { acquireCount: { w: 172 } }, Collection: { acquireCount: { w: 87 } }, Metadata: { acquireCount: { w: 85 } }, oplog: { acquireCount: { w: 85 } } } protocol:op_command 190ms
2021-07-27T00:00:32.911740+08:00 [conn11815] command monitor.$cmd command: update { update: "process_snapshot_v1", updates: 122, ordered: false, shardVersion: [ Timestamp 45458000|0, ObjectId('5836a370dd42ce3bf562ebe9') ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:2753 locks:{ Global: { acquireCount: { r: 248, w: 248 } }, Database: { acquireCount: { w: 248 } }, Collection: { acquireCount: { w: 126 } }, Metadata: { acquireCount: { w: 122 } }, oplog: { acquireCount: { w: 122 } } } protocol:op_command 296ms
2021-07-27T00:00:32.941457+08:00 [conn11746] command monitor.$cmd command: update { update: "process_1min_1627257600", updates: 48, ordered: false, shardVersion: [ Timestamp 915000|25, ObjectId('60fb922daf63c4b02a245f1b') ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:2013 locks:{ Global: { acquireCount: { r: 96, w: 96 } }, Database: { acquireCount: { w: 96 } }, Collection: { acquireCount: { w: 48 } }, Metadata: { acquireCount: { w: 48 } }, oplog: { acquireCount: { w: 48 } } } protocol:op_command 113ms
2021-07-27T00:00:32.943920+08:00 [conn11787] command monitor.$cmd command: update { update: "process_snapshot_v1", updates: 84, ordered: false, shardVersion: [ Timestamp 45458000|0, ObjectId('5836a370dd42ce3bf562ebe9') ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:2642 locks:{ Global: { acquireCount: { r: 174, w: 174 } }, Database: { acquireCount: { w: 174 } }, Collection: { acquireCount: { w: 90 } }, Metadata: { acquireCount: { w: 84 } }, oplog: { acquireCount: { w: 84 } } } protocol:op_command 280ms
2021-07-27T00:00:32.949948+08:00 [conn11847] command monitor.$cmd command: update { update: "process_snapshot_v1", updates: 111, ordered: false, shardVersion: [ Timestamp 45458000|0, ObjectId('5836a370dd42ce3bf562ebe9') ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:3345 locks:{ Global: { acquireCount: { r: 225, w: 225 } }, Database: { acquireCount: { w: 225 } }, Collection: { acquireCount: { w: 114 } }, Metadata: { acquireCount: { w: 111 } }, oplog: { acquireCount: { w: 111 } } } protocol:op_command 309ms
2021-07-27T00:00:32.981136+08:00 [conn11865] update monitor.process_snapshot_v1 query: { host: "h55-battle01-12026", id: "31436_1627315201.04" } update: { $set: { status: "sleeping", last_updated: 1627315231.290128, name: "sh", created: 1627315201.04, cmd_str: "/bin/sh -c [ -e /usr/local/hwtool/script/hwcheck.py ] && python /usr/local/hwtool/script/hwcheck.py >> /var/log/hwcheck.log 2>&1", pid: 31436, ts: new Date(1627315231000), alive: true, user: "root", id: "31436_1627315201.04" }, $unset: { disappear_ts: 1, raw_disappear_ts: 1 } } keysExamined:0 docsExamined:0 nMatched:1 nModified:1 upsert:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { r: 155, w: 155 } }, Database: { acquireCount: { w: 155 } }, Collection: { acquireCount: { w: 78 } }, Metadata: { acquireCount: { w: 77 } }, oplog: { acquireCount: { w: 77 } } } 103ms

I run command like with

root@oceantest-xucltestmongos01-2140180:/home/data/smthing# keyhole -loginfo mongodb-20210727.log
2021-07-28T17:21:42+08:00 R simagix/keyhole v1.0.7-20210722 begins at 2021-07-28T17:21:42+08:00
+----------+--------+------+--------+------+---------------------------------+--------------------------------------------------------------+
| Command  |COLLSCAN|avg ms| max ms | Count| Namespace                       | Query Pattern                                                |
|----------+--------+------+--------+------+---------------------------------+--------------------------------------------------------------|
+----------+--------+------+--------+------+---------------------------------+--------------------------------------------------------------+

top 0 of 0 lines displayed; see HTML report for details.

however, there is nothing to display

On the docs it is mentioned to use Maobi docker image to generate html reports.

After running it as mentioned the html report contains:

Unlicensed! Maobi v1.5.8-20200802 - Data provided is for reference only

I used to use Maobi before and it will provide more information on the report.

Is there a way to access to a previous version of Maobi/get a license?

error code :-listen tcp :5408: listen: operation not permitted

can you please help me out with this issue with my enviroment details and how to integrate it into this for grafana server?
i try on the root

• Ubuntu18.04
• Grafana 7.3

Have you removed the image from Docker Hub?

$ docker pull simagix/keyhole Using default tag: latest Error response from daemon: manifest for simagix/keyhole:latest not found: manifest unknown: manifest unknown

Ass mentioned in issue #94 the Maobi image is also not available.

From where can I download the binary of keyhole?

I updated my keyhole repo and build the binary today

$ keyhole --version
github/keyhole v1.2.7-20220707

The info command

keyhole -v --info $uri

does not produce a JSON file with the details as specified in the blog, "Peek at your MongoDB Clusters like a Pro with Keyhole: Part 1" says it should

All I get is something like

2022-07-07T09:09:07-05:00 R github/keyhole v1.2.7-20220707 begins at 2022-07-07T09:09:07-05:00
MongoDB v4.4.15 enterprise <hostname> (CentOS Linux release 7.9.2009 (Core)) mongod replica 2 cores 7821 mem

When using syslog option for journalctl, the output is empty. Would it be possible to add support for the syslog (journalctl) format?

https://jira.mongodb.org/browse/SERVER-45389

Run db.serverStatus on the config primary

	"shardedIndexConsistency" : {
		"numShardedCollectionsWithInconsistentIndexes" : NumberLong(1)
	},

Hello,

I see that you have a link to Maobi in the readme. However, it's to the private URL.

The Moabi images are public, but the github repo that they're built from is private. Are you able to make the Moabi github repo public?

Thanks!

Hi,

Could you please help me with this issue -

root@xxxxx:~# keyhole --info --sslCAFile /data/ssl/ca.crt --sslPEMKeyFile /data/ssl/client.pem "mongodb://xxxx:xxxx@server1:27017,server2:27017,server3:27017/keyhole?authSource=admin&replicaSet=test&ssl=true" 2022/05/11 05:03:11 error parsing uri: scheme must be "mongodb" or "mongodb+srv" root@dev-mongo-rs-1:~#

root@xxxxx:~# keyhole --info "mongodb://xxx:xxxx@server1:27017,server2:27017,server3:27017/keyhole?authSource=admin&replicaSet=test&ssl=true" --sslCAFile /data/ssl/ca.crt --sslPEMKeyFile /data/ssl/client.pem flag provided but not defined: -sslCAFile

root@xxxxx:~# keyhole --info "mongodb://xxxx:xxxx@server1:27017,server2:27017,server3:27017/?authSource=admin&replicaSet=test&ssl=true&sslCAFile=/data/ssl/ca.crt&sslPEMKeyFile=/data/ssl/client.pem" 2022-05-11T05:07:19Z R simagix/keyhole v1.2.6-20220511 begins at 2022-05-11T05:07:19Z server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: dev-mongo-rs-1:27017, Type: Unknown, Last error: connection() error occured during connection handshake: x509: certificate is not authorized to sign other certificates }, { Addr: dev-mongo-rs-2:27017, Type: Unknown, Last error: connection() error occured during connection handshake: x509: certificate is not authorized to sign other certificates }, { Addr: dev-mongo-rs-3:27017, Type: Unknown, Last error: connection() error occured during connection handshake: x509: certificate is not authorized to sign other certificates }, ] } root@dev-mongo-rs-1:~#

I have tried all these ways but failing.

Could you please help me parsing the connection string with SSL optioins to Keyhole.

Thanks,
Tom

go 1.18:

• had to run go mod tidy before building
• doesn't need to be in $GOPATH any more
• need to run e.g. go build -o path/keyhole ./main

I have 2 shards (2 nodes, 1 arbiter), Config server with 3 node replica set and 1 mongos instance.
When running keyhole for 5 mins, data getting loaded but getting below error and not progressing further.

Command
./keyhole --duration 5 --conn 5 --tps 5 "mongodb://mongodba:[email protected]:20007"

Error Message
Begin a load test [y/N]: y
2021-05-12T01:05:48-04:00 I Duration in minute(s):5
2021-05-12T01:05:48-04:00 I Total TPS: 25 (5 tps/conn * 5 conns), duration: 5 (mins)
2021-05-12T01:05:48-04:00 I Sharding collection:_KEYHOLE_88800.__examples
2021-05-12T01:05:48-04:00 I splitting chunks...
2021-05-12T01:05:48-04:00 I moving chunks...
2021-05-12T01:05:48-04:00 I moving map[email:O] from rs1 to rs0
2021-05-12T01:06:19-04:00 I context deadline exceeded
2021-05-12T01:06:49-04:00 I server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: mvp-mongodb-rs1-0.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp: lookup mvp-mongodb-rs1-0.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local on 168.244.217.13:53: no such host }, { Addr: mvp-mongodb-rs1-1.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp: lookup mvp-mongodb-rs1-1.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local on 168.244.217.13:53: no such host }, ] }
2021-05-12T01:07:20-04:00 I Thread1existing withserver selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: mvp-mongodb-rs1-0.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp: lookup mvp-mongodb-rs1-0.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local on 168.244.217.13:53: no such host }, { Addr: mvp-mongodb-rs1-1.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp: lookup mvp-mongodb-rs1-1.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local on 168.244.217.13:53: no such host }, ] }
2021-05-12T01:07:20-04:00 I Thread0existing withserver selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: mvp-mongodb-rs1-0.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp: lookup mvp-mongodb-rs1-0.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local on 168.244.217.13:53: no such host }, { Addr: mvp-mongodb-rs1-1.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp: lookup mvp-mongodb-rs1-1.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local on 168.244.217.13:53: no such host }, ] }
2021-05-12T01:07:20-04:00 I Thread4existing withcontext deadline exceeded
2021-05-12T01:07:20-04:00 I Thread2existing withserver selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: mvp-mongodb-rs1-0.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp: lookup mvp-mongodb-rs1-0.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local on 168.244.217.13:53: no such host }, { Addr: mvp-mongodb-rs1-1.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp: lookup mvp-mongodb-rs1-1.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local on 168.244.217.13:53: no such host }, ] }
2021-05-12T01:07:20-04:00 I Thread3existing withserver selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: mvp-mongodb-rs1-0.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp: lookup mvp-mongodb-rs1-0.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local on 168.244.217.13:53: no such host }, { Addr: mvp-mongodb-rs1-1.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp: lookup mvp-mongodb-rs1-1.mvp-mongodb-rs1.mdb-mvp.svc.cluster.local on 168.244.217.13:53: no such host }, ] }

If a sharded cluster is connected, keyhole attempts to conenct to all shards using the mongos' credentials. (This fails if the same user does not exist everywhere, but this can be worked around)

If the password on the mongos contains a "/" character, keyhole fails connecting to the shards,

To reproduce:

• Create a user (drop immediately after, this password is horrible)

mongos> use admin
switched to db admin
mongos> db.createUser({user:"testuser",pwd:"p/a",roles:["root"]})
Successfully added user: { "user" : "testuser", "roles" : [ "root" ] }
mongos>
bye

• Run keyhole against this mongos user:

 # keyhole mongodb://testuser:p%[email protected]/?authSource=admin
MongoDB v4.0.20 community <redacted> (Ubuntu) mongos sharded 2 cores 7710 mem
Begin a load test [y/N]: y
2020/10/09 23:30:34 Duration in minute(s): 5
2020/10/09 23:30:34 Sharding collection: _KEYHOLE_88800.__examples
2020/10/09 23:30:34 splitting chunks...
new split key { email: "J" } is a boundary key of existing chunk [{ email: "J" },{ email: "[email protected]" })
2020/10/09 23:30:34 Total TPS: 40 (20 tps/conn * 2 conns), duration: 5 (mins)
2020/10/09 23:30:34 error parsing uri: unescaped slash in password
2020/10/09 23:30:34 error parsing uri: unescaped slash in password
2020/10/09 23:30:34 error parsing uri: unescaped slash in password

(It then hangs there)

Keyhole failed to output bson file when there are too much data in the bson struct. The error is

document size (2147483793) is larger than the max int32

Using keyhole for generating documents we tried to generate 1M but in the end we got 700.000. We think may be due to unique index and keyhole generating twice same fields

keyhole doesn't support readPreferenceTags, for example:

"mongodb+srv://<redacted>@tags.jgtm2.mongodb.net/?readPreference=secondary&readPreferenceTags=nodeType:ANALYTICS"

The command keyhole --index "mongodb://... annotates TTL indexes as unused (blue ?).

Please fix the output of keyhole as you did already for Maobi in #73
Thanks!

2021/01/06 11:21:39 Seed data to collection DAT_KEYS using 32 connections
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x171c686]
goroutine 124 [running]:
github.com/simagix/keyhole/sim.populateData(0xc000350300, 0xa, 0xc0002c17d0, 0x0, 0x0, 0x0)
/Users/jp18/go/src/github.com/simagix/keyhole/sim/seed.go:515 +0x256
github.com/simagix/keyhole/sim.(*Feeder).seedFromTemplate.func1(0xc0003c1820, 0xc000350300, 0xc00000e1c0, 0xc0003cfa60, 0xc0004031d0, 0xc00025e9c8, 0xc0005b2050, 0xa)
/Users/jp18/go/src/github.com/simagix/keyhole/sim/seed.go:452 +0x97
created by github.com/simagix/keyhole/sim.(*Feeder).seedFromTemplate

From using -allinfo, upon completion but having "document size (2147483679) is larger than the max int32" error.

To improve confidence in contributions, make github actions that build the code and also run golangci-lint.

Hi

KeyHole is good tool....

MongoDB engineer provides cache overflow related parameter in their graphical tool of diagnostic data

Can you please add those metric in grafana graphical representation of FTDC

If you can add it will be very usefull

.

After binaries removal, following the build instructions results in failure:

# ./build.sh
# command-line-arguments
./keyhole.go:209:23: undefined: mdb.ParseURI

Tested with Centos 7 and macOS Catalina.

# go version
go version go1.13 linux/amd64

When there is an index on { "a" : 1 } and another on { "a" : 1, "b" : "2dsphere" } the former is not redundant since the 2dsphere field is always sparse and so for queries on a alone, a COLLSCAN needs to be done in the absence of the { "a" : 1 } index. This is explained in SERVER-13830.

Keyhole did not prompt for password when using authMechanism PLAIN.

When using Keyhole to perform logfile analysis, if a query filter that has been logged using the new log format uses $in (or presumably $nin, though I haven't tested this) with $binary values, keyhole doesn't collapse this down to $in[...], instead leaving $in[$binary{base64:1,subType:1}, $binary{base64:1,subType:1}] with as many $binary subclauses as there were values in the original query. This prevents accurate fingerprinting of long-running operations on these fields.

Example log entry:

{"t":{"$date":"2020-11-05T09:30:50.680+00:00"},"s":"I",  "c":"WRITE",    "id":51803,   "ctx":"conn851","msg":"Slow query","attr":{"type":"update","ns":"testcase.samplecollection","command":{"q":{"_id":{"$in":[{"$binary":{"base64":"jtWtXe56xEGlJ+2cNiz+Yg==","subType":"3"}},{"$binary":{"base64":"tgP+I+Jl+I1wya7jxapMAg==","subType":"3"}},{"$binary":{"base64":"ElxhlRkJfT8FOCxfe+OJ9w==","subType":"3"}},{"$binary":{"base64":"nFTaxhmUzWC/yEnrqYN3NA==","subType":"3"}}]}},"u":{"$set":{"Touched":{"$date":"2020-11-05T09:30:47.707Z"}}},"multi":true,"upsert":false},"planSummary":"IXSCAN { _id: 1 }","keysExamined":4,"docsExamined":4,"nMatched":4,"nModified":4,"numYields":4,"queryHash":"EDADD1D2","planCacheKey":"E5EEC495","locks":{"ParallelBatchWriterMode":{"acquireCount":{"r":5}},"ReplicationStateTransition":{"acquireCount":{"w":5}},"Global":{"acquireCount":{"w":5}},"Database":{"acquireCount":{"w":5}},"Collection":{"acquireCount":{"w":5}},"Mutex":{"acquireCount":{"r":1}}},"flowControl":{"acquireCount":5,"timeAcquiringMicros":49},"storage":{"data":{"bytesRead":40}},"durationMillis":234}}

Keyhole currently outputs the following as the parsed query:

{"_id":{"$in":[{"$binary":{"base64":1,"subType":1}},{"$binary":{"base64":1,"subType":1}},{"$binary":{"base64":1,"subType":1}},{"$binary":{"base64":1,"subType":1}}]}}

I've worked around this locally by altering mdb/logv2_parser.go where it performs filter normalisation to prepend another regex replace, replacing the line:

    re := regexp.MustCompile(`\[(1,)*1\]`)

...with:

    re := regexp.MustCompile(`\{"\$binary":\{[^\}]+\}\}`)
    stat.filter = re.ReplaceAllString(stat.filter, `1`)
    re = regexp.MustCompile(`\[(1,)*1\]`)

With this modification, Keyhole parses the above log entry as follows:

{"_id":{"$in":[...]}}

This feels like the intended behaviour here, though there may be a more efficient way to achieve it.

Hi, Using x509 cert based authentication doesn't seem to work for keyhole. It seems to expect a username/password combination always.

Hello,
I am trying to install keyhole and I get below error. I am following instructions under build. I already installed go & dep. Could you able to help?

keyhole % ./build.sh
main.go:16:2: cannot find package "." in:
/usr/local/go/src/vendor/github.com/simagix/gox
main.go:17:2: cannot find package "." in:
/usr/local/go/src/vendor/github.com/simagix/keyhole/mdb
main.go:18:2: cannot find package "." in:
/usr/local/go/src/vendor/github.com/simagix/keyhole/sim
main.go:19:2: cannot find package "." in:
/usr/local/go/src/vendor/github.com/simagix/keyhole/sim/util
main.go:20:2: cannot find package "." in:
/usr/local/go/src/vendor/github.com/simagix/mongo-atlas/atlas
main.go:21:2: cannot find package "." in:
/usr/local/go/src/vendor/github.com/simagix/mongo-ftdc/analytics
main.go:22:2: cannot find package "." in:
/usr/local/go/src/vendor/go.mongodb.org/mongo-driver/mongo
main.go:23:2: cannot find package "." in:
/usr/local/go/src/vendor/go.mongodb.org/mongo-driver/x/mongo/driver/connstring

I'm testing out the -compare feature but running into an issue viewing the output as HTML in Maobi.
It's working great in the console. It appears to correctly output the comparison.

Steps to Reproduce:

1. Run keyhole -compare "mongoUri1" "mongoUri2"
2. After completion, the console displays: 2021-04-28T08:15:54-07:00 I bson data written to ./out/my-local-mongodb-sharded-mongos-6fddcc7cf-cqjnt-compare.bson.gz
3. Start maobi using docker: docker run -d -p 3030:3030 simagix/maobi
4. Drag-and-drop my-local-mongodb-sharded-mongos-6fddcc7cf-cqjnt-compare.bson.gz into Maobi.

Expected Result:

• An HTML report of the differences between the two mongo setups.

Actual Result:

• Error: "Type not recognized! "

Keyhole version: simagix/keyhole v1.0.6-20210421

I am using Keyhole for MongoDB and when giving 100 TPS with 100 connections and a duration of 5 mins, the database is created around 1.6GB which is too low for my testing. Is there a way to create a database with 80-90 GB automatically and use that for testing.

Looks like the commit #469f687 has introduced a bug while generating the output directory for json when running a print command.

For example, the command below is generating its output in directory outfile.json instead of out/file.json.
keyhole --print path/to/file.bson.gz

Here is the output log of this command in my local machine:

➜  out keyhole --print kolsch-log.bson.gz
2022-02-28T17:37:16-03:00 R simagix/keyhole v1.2.2-20220215 begins at 2022-02-28T17:37:16-03:00
2022-02-28T17:49:20-03:00 R simagix/keyhole v1.2.2-20220215 begins at 2022-02-28T17:49:20-03:00
+----------+--------+------+--------+------+---------------------------------+--------------------------------------------------------------+
| Command  |COLLSCAN|avg ms| max ms | Count| Namespace                       | Query Pattern                                                |
|----------+--------+------+--------+------+---------------------------------+--------------------------------------------------------------|
|aggregate  COLLSCAN   2.3s     2723      8 Invoices.SV_Invoice               {status: {$nin: [... ]}}, group: , sort: { __agg_sum: -1 }    |
|aggregate             1.8s     2202      8 Invoices.SV_Invoice               {updatedDate: {$gte:1}, status: {$in: [... ]}}, group: ,      |
|                                                                              sort: { y.year: 1, y.month: 1, y.date: 1 }                   |
|...index:  { updatedDate: -1, deleted: 1 }                                                                                                 |
|find       COLLSCAN     49       61      8 EmptiesManagement.MXEmptiesDDC    {deliveryCenterId:1, sku: {$in: [... ]}}                      |
+----------+--------+------+--------+------+---------------------------------+--------------------------------------------------------------+

top 3 of 3 lines displayed; see HTML report for details.
json data written to ./outkolsch-log.json
➜  out   

Hello. It seems that there is a new release that version is 1.0.7 on the GitHub now.
So, I try to download the new version from DockerHub but I couldn't find the 1.0.7 version yet in here.

Could you upload the 1.0.7 version of the Docker image?

➜  keyhole ./keyhole "mongodb://admin:1234@x:27017,x:27018"

2021-04-27T16:53:04-03:00 I MongoDB v4.4.5 community aed6234f3232 (Ubuntu) mongos sharded(1) 1 cores 978 mem
Begin a load test [y/N]: y
2021-04-27T16:53:22-03:00 I Duration in minute(s):5
2021-04-27T16:53:22-03:00 I Total TPS: 160 (20 tps/conn * 8 conns), duration: 5 (mins)
2021-04-27T16:53:24-03:00 I Sharding collection:_KEYHOLE_88800.__examples
2021-04-27T16:53:25-03:00 I splitting chunks...
2021-04-27T16:53:25-03:00 I moving chunks...
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/simagix/keyhole/sim.(*Runner).splitChunks(0xc00051a400, 0xf33b80, 0xc0000b6020)
	/go/src/github.com/simagix/keyhole/sim/runner.go:454 +0x1a6b
github.com/simagix/keyhole/sim.(*Runner).createIndexes(0xc00051a400, 0x0, 0x0, 0x0, 0x4, 0xc000418080)
	/go/src/github.com/simagix/keyhole/sim/runner.go:338 +0x5cb
github.com/simagix/keyhole/sim.(*Runner).Start(0xc00051a400, 0xc000200320, 0x50)
	/go/src/github.com/simagix/keyhole/sim/runner.go:190 +0x474
github.com/simagix/keyhole.StartSimulation(0xc00051a400, 0xc00019f800, 0x1)
	/go/src/github.com/simagix/keyhole/simulation.go:12 +0x2b
github.com/simagix/keyhole.Run(0xc0000c6860, 0x1f)
	/go/src/github.com/simagix/keyhole/keyhole.go:286 +0x1c0d
main.main()

When I try to run the executable with many connection in the connection string I got this error. Seems like something to do with the split used to parsing the connection string. This will be nice to be supported since MongoDB Shard Cluster with many mongos could be a scenario to test.

Hi,

The link under "Build and Download" seems to be broken:

https://www.simagix.com/2021/02/build-and-download-keyhole_7.html

https://github.com/simagix/keyhole/blob/master/README.md?plain=1#L42

It could be worth supplementing the reference to an external blog with go get or Docker commands instead

Command: keyhole -index "connectionString"

Test case from compound indexes from the same collection:
Fields 1: {a:1, b:1, d:1}
Fields 2: {a:1, b:1, c:1, d:1}

The first compound index gets flagged as redundant b/c all of the fields in it are contained by the second compound indexes fields.

However, I believe this is incorrect according to the MonogDB docs on prefixes -- instead, to be fully redundant, the fields of compound index 2 would need to be {a:1, b:1, d:1, c:1}, b/c order matters.

After running the command to get DB info ./keyhole -v --info "mongodb://localhost:27017/", when I upload my gzipped bson file to Maobi, I get unsupported version infra/keyhole v1.1.8-20211230

I guess this is not the expected behavior.

If you're not the creator of Maobi, could you please tell me so I can open an issue there?

Thank you in advance!

• wrap summary from keyhole_stats into keyhole_perf
• rename keyhole_perf*.bson.gz to -perf.bson.gz
• zip all stats and perf file into an archive

Hi All,

I am running keyhole testing for 30 minutes and the run fails at random time with the below error
Attached the complete error message

Command
keyhole --duration 30 --conn 50 --tps 50 ""
30min_50con_50tps.txt

Error Message
fatal error: concurrent map read and map write

goroutine 164 [running]:
runtime.throw(0xc60bef, 0x21)
/usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc001683888 sp=0xc001683858 pc=0x4360d2
runtime.mapaccess1_faststr(0xba32a0, 0xc0004acd20, 0xc01c016bfa, 0x6, 0xc00a29f110)
/usr/local/go/src/runtime/map_faststr.go:21 +0x465 fp=0xc0016838f8 sp=0xc001683888 pc=0x413ac5
github.com/simagix/keyhole/sim.(*Runner).Simulate(0xc0004ea400, 0x1d, 0x0, 0x0, 0x0, 0xc, 0x0, 0x0)
/usr/local/go/src/github.com/simagix/keyhole/sim/simulation.go:172 +0x10ed fp=0xc001683f00 sp=0xc0016838f8 pc=0xb0786d
github.com/simagix/keyhole/sim.(*Runner).Start.func1(0xc0004ea400, 0xc000310180, 0xc0003293b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc)
/usr/local/go/src/github.com/simagix/keyhole/sim/runner.go:203 +0x93 fp=0xc001683f90 sp=0xc001683f00 pc=0xb0a693
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc001683f98 sp=0xc001683f90 pc=0x46b341
created by github.com/simagix/keyhole/sim.(*Runner).Start
/usr/local/go/src/github.com/simagix/keyhole/sim/runner.go:194 +0x3ef

It seems build.sh is unable to install keyhole. Attached the go version, build.sh output.

~/go-workspace/src/github.com/simagix/keyhole (master)$ go version go version go1.15.5 darwin/amd64 test-user@user-MacBook-Pro:~/go-workspace/src/github.com/simagix/keyhole (master)$ ./build.sh simagix/keyhole v1.0.2-20201119 test-user@user-MacBook-Pro:~/go-workspace/src/github.com/simagix/keyhole (master)$ keyhole --version -bash: keyhole: command not found test-user@user-MacBook-Pro:~/go-workspace/src/github.com/simagix/keyhole (master)$ keyhole --help -bash: keyhole: command not found test-user@user-MacBook-Pro:~/go-workspace/src/github.com/simagix/keyhole (master)$

According to mongoDB's documention, the statistics [reported by the $indexStats aggregation] only includes index access driven by user requests. It does not include internal operations like deletion via TTL Indexes or chunk split and migration operations.
See also: https://jira.mongodb.org/browse/SERVER-21815
Therefore, in Maobi, in section 1.1.5 Unused Indexes (mongod up > 7 days), you may see "unused" indexes which in fact may be TTL indexes and as such are not unused.
Suggestion: Maobi should take into account TTL indexes and not show or count them as "unused".

The DAT_KEYS.json (which I generated using the keyhole "-v --schema" flags is as follows:

{
	"_id": {
		"$oid": "5ff5b387a17fc87994620b25"
	},
	"ALG": "AES-256-CBC",
	"DEVICE_ID": "dXJJOmdzbWE6aW1laToxMzy2NjkxMC90MTU4MTAtMA==",
	"PLATFORM": "Android",
	"LAST_UPDT_TIME": {
		"$date": "2021-01-05T21:37:59.625Z"
	},
	"LAST_READ_TIME": {
		"$date": "2021-01-05T21:37:59.625Z"
	},
	"DEVICE_PB_KEY": "jKc7I1m5WMyDKsJUKEkTdC3lfEcGxTqEavQ+EB2hQqDMuUjkKCpdOZ/oUY/hN6hLqoWOqgKtgh+LYuHedAKcKjOSuFdfXrUmZGcmSNmB8nP7urnMEk/kSnbUSgbi/0q0V3f0DweWBlcMFrxTQrocI2MYUBCoG1DkpSuMJkfjYnXs2z33DYXc1cXO9UPAVNz05GKQUrTL1vaQrPcjX7KL6S8GuAb6ZMYBAkX+33kDcwtiCtbH5Ej4sYyFWSiysTjw5H0O/vxzsUFRYTIIVwLFf/Npa1jwvFpx9Z/MPVl/UBZY932qs2celFWdH0lQaPVmOVfG9ACgT3PaskFamL7SVQ==",
	"SEC_KEY": "cSU+Buie5ZdbgwEseojcFwKu6MV5J/RpU5hqreEN2MVB4pLJDP+dc9WTg2f8bsN2zG9AIyCSfZW+rq4blu/XRgTumAPlQ49k4TJAC8/FwzTo3zUk44So1it3gkSWVfMr50/Z5RXPdLzK7HRNXKoHUwSkjo/vezfrHFtHXPR63ByPJ8WPrZqPB9cH5z7jIdyDEyCntz9PQwIdHC7xK4Fo40hANs4nWoxu7lbvqLiwpyXypIUzt/nC985VSeh0luvEaHpVa2hn8kk5A2RAHIfRMNJUtAxgeRxPoFQfjKrd2fKhqfacTovta7CcCqRFB4dpkV48Wo4b2MJzkaqpp4LEOw==",
	"CLIENT_ID": "ClientID1",
	"FCM_ID": "jpJcXwcfb4M:APA91bH4s7STx99VvXycx8Q0ycDsQNtSGG4CvGGC9E3p4iUDFWT2vle-rJhI-YYkKh9PstnqyK-yLBexfTPFrPKYWphpzDV0IUds0jGK5I-LRYfieCb_nI-XqlLuoCW6tHS1WnK1ENuE"
}

However, the generated documents are being created with the intended date fields LAST_UPDT_TIME and LAST_READ_TIME as object types instead of date types. Below is a sample of the created documents.

{
    "_id": {
        "$oid": "5ff61d72bce8ac370581b2f4"
    },
    "LAST_READ_TIME": {
        "$date": {
            "$date": "2020-10-06T08:56:35.000Z"
        }
    },
    "SEC_KEY": "eSR+Utxa4HroqxRwbcxrIwGx4YO5O/PaS4wodaNV5TUJ8qCQRU+sa1GVr7g0txE0hG0VMzWErEH+cp2woy/JNdNtfSJwC85g2FECO0/FvxTv4zOk41Uk9gk7auJRMxFc56/J0BFEeNrD9ZUYVNoFFaKgxc/fcxvuTZuIKHD60OnIO0WYiJmQL2hM8c3zHseNOyIfnh7YTzEfMS5hU3Tm31fUOz2lEdqk2zcutUrplkZzuUFjt/iO112NLtf0oyaYjWbOt8dv1vi1F0QRHOgVEHNNzFarfSbVgBRahJff4qTxovqyXbjay0BsPgHEQ9xumW96Mk7q2EVnslyyl9PESg==",
    "DEVICE_PB_KEY": "sBa4T2p7SUgEPqDYBZmBkX0bwScOlUlIiiQ+PI7oMlXCvUojFHbhOQ/gRM/sU8lNipGLxaWnpu+RNoVmtKOgGxNQpPriWmNuFOpcTLbL2zX0xwxTUk/mWjpKLdpa/0d1B5t7GyqEUdwBBasCNegzZ0PMGINtK4AnyMbMIxyxCpQj4c83CDRk3uYV9PEZJTf78VTZShZZ1aaNhQfnU8UG1X7YcXd3AIHVQjQ+98wMfhunIjqY0Te4zLhVLBsmeFgh7B4T/pysbLEWKDAIQyPVy/Atk8bngWcz2R/NONc/GOBQ235ji5ekzHKmJ5wSeMIyVEtD6MApY7VfwrBnaO8HNY==",
    "FCM_ID": "jcDxWeith4Q:DYH95jM4c3ZZf06PmAonx0E5qfSvTVoWPQ0TmNMO5Q4i7zVYSNX8plz-nLdR-AXkSc1BafgypN-kBPonfUGFaMGPVghtaML1DKed2eMM5H-VQIwhsYk_mM-JsjYwuAO3kAK8XhX2QKjC",
    "LAST_UPDT_TIME": {
        "$date": {
            "$date": "2020-12-04T00:51:03.000Z"
        }
    },
    "CLIENT_ID": "XjplhpNX4",
    "PLATFORM": "Scjulnk",
    "ALG": "MNS-411-BMX",
    "DEVICE_ID": "gYPZRoccaJY2jI4elCiiQyd8PezaHL58YSX4QYAzOW=="
}