osrg / namazu Goto Github PK

View Code? Open in Web Editor NEW

492.0 31.0 43.0 29.28 MB

:fish: 鯰: Programmable fuzzy scheduler for testing distributed systems

Home Page: http://osrg.github.io/namazu

License: Apache License 2.0

Shell 6.73% Go 67.72% Makefile 0.24% C++ 4.70% C 1.40% CMake 0.21% Java 9.45% Python 8.19% HTML 0.60% Dockerfile 0.76%

testing-tools flaky-tests

namazu's Issues

Flaky test? (endpoint)

Build 193 (successful)
https://travis-ci.org/osrg/earthquake/builds/124123660

ok      github.com/osrg/earthquake/earthquake/cli   2.055s  coverage: 0.3% of statements
ok      github.com/osrg/earthquake/earthquake/cli/inspectors    2.028s  coverage: 14.3% of statements
ok      github.com/osrg/earthquake/earthquake/cli/tools 1.014s  coverage: 2.5% of statements
ok      github.com/osrg/earthquake/earthquake/endpoint  11.878s coverage: 72.1% of statements
ok      github.com/osrg/earthquake/earthquake/endpoint/local    2.870s  coverage: 85.4% of statements
ok      github.com/osrg/earthquake/earthquake/endpoint/pb   1.076s  coverage: 76.8% of statements
....

Build 195 (fail)
https://travis-ci.org/osrg/earthquake/builds/124127161

ok      github.com/osrg/earthquake/earthquake/cli   2.184s  coverage: 0.3% of statements
ok      github.com/osrg/earthquake/earthquake/cli/inspectors    2.241s  coverage: 14.3% of statements
ok      github.com/osrg/earthquake/earthquake/cli/tools 1.016s  coverage: 2.5% of statements
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
The build has been terminated

Implement process inspector

Port the PoC written in Python into Go: https://github.com/AkihiroSuda/MicroEarthquake/tree/v20160203

ETA: this February

Coveralls reports transceiver pkg is not covered, but it's actually covered

https://coveralls.io/builds/5286609

There is no test for the transceiver pkg, but endpoint_test.go covers the transceiver pkg.
https://github.com/osrg/earthquake/blob/4bc356dcde8eefc3bdf2d37c0906d7b1cbc72413/earthquake/endpoint/endpoint_test.go

Flaky test: nmz/endpoint

4919228
https://travis-ci.org/osrg/namazu/builds/134960743

ok      github.com/osrg/namazu/nmz/cli  2.768s  coverage: 0.3% of statements
ok      github.com/osrg/namazu/nmz/cli/container/run    2.432s  coverage: 0.0% of statements
ok      github.com/osrg/namazu/nmz/cli/inspectors   2.568s  coverage: 12.6% of statements
ok      github.com/osrg/namazu/nmz/cli/tools    1.024s  coverage: 2.5% of statements
ok      github.com/osrg/namazu/nmz/container    2.534s  coverage: 0.0% of statements
ok      github.com/osrg/namazu/nmz/container/ns 1.025s  coverage: 0.0% of statements
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
The build has been terminated

Typical output:
https://travis-ci.org/osrg/namazu/builds/134956146

ok      github.com/osrg/namazu/nmz/cli  2.220s  coverage: 0.3% of statements
ok      github.com/osrg/namazu/nmz/cli/container/run    2.226s  coverage: 0.0% of statements
ok      github.com/osrg/namazu/nmz/cli/inspectors   2.230s  coverage: 12.6% of statements
ok      github.com/osrg/namazu/nmz/cli/tools    1.017s  coverage: 2.5% of statements
ok      github.com/osrg/namazu/nmz/container    2.223s  coverage: 0.0% of statements
ok      github.com/osrg/namazu/nmz/container/ns 1.015s  coverage: 0.0% of statements
ok      github.com/osrg/namazu/nmz/endpoint 2.404s  coverage: 72.1% of statements    <--- this
...

Add an option to specify the validated number of executions

json file storage should be optional

naive storage makes single gob file plus many json files.
This can exhaust inode space.

Use channel for local-to-local RPC

ETA: this February

LOW HANGING FRUIT: make goreportcard happy

goreportcard says A+ for Namazu v0.2.0, but there are still things we should do:

Component	Score
go_vet	100%
gocyclo	98%
gofmt	98%
golint	41%
ineffassign	98%
license	100%
misspell	98%

gocyclo: 98%

namazu/nmz/cli/init.go
  Line 108: warning: cyclomatic complexity 22 of function _init() is high (> 15) (gocyclo)
namazu/nmz/explorepolicy/random/randompolicy.go
  Line 156: warning: cyclomatic complexity 17 of function (*Random).LoadConfig() is high (> 15) (gocyclo)

gofmt 98%

namazu/nmz/cli/tools/visualize.go
  Line 1: warning: file is not gofmted (gofmt)
namazu/nmz/signal/action_sched_procset_test.go
  Line 1: warning: file is not gofmted (gofmt)

golint 41%

...

ineffassign 98%

namazu/nmz/inspector/transceiver/resttransceiver.go
  Line 92: warning: err assigned and not used (ineffassign)
  Line 69: warning: err assigned and not used (ineffassign)
  Line 47: warning: err assigned and not used (ineffassign)
namazu/nmz/container/ns/boot.go
  Line 42: warning: err assigned and not used (ineffassign)

misspell 98%

namazu/nmz/endpoint/pb/pbendpoint.go
  Line 30: warning: 53:found "diferrent" a misspelling of "different" (misspell)
namazu/nmz/endpoint/pb/pbendpoint_test.go
  Line 107: warning: 55:found "diferrent" a misspelling of "different" (misspell)

Docker-in-Docker Image won't work on Fedora

This is not a bug of Eathquake, but we should fix it

Improve go package structure

Improve godocability https://godoc.org/github.com/osrg/earthquake
Import specific version deps http://stackoverflow.com/questions/24855081/how-do-i-import-a-specific-version-of-a-package

a replay command based on stored history

Define JSON schema for config.json

For clarification and validation.

Note json schema files can be statically included in earthquake binary using https://github.com/laher/goxc .

Implement design introduced in doc/figs/eq-redesign.pdf

Tasks

Define config.json schema #30
s/searcher/explorer/g
List up env vars

Godoc.org not working

https://godoc.org/github.com/osrg/earthquake/earthquake

Nothing shown?

Add support for YAML (still able to keep compatibility with JSON)

YAML is much more friendly to humans

[doc] Mention and promote Namazu Swarm: CI Job Parallelizer built on Docker and Kubernetes

Namazu Swarm executes multiple CI jobs in parallel across a Docker cluster

https://github.com/osrg/namazu-swarm

We need to mention it in osrg/namazu/README.md

Reset proc sched attrs at a certain probability

Purposes

Greedy state exploration
Statistics of explored state patterns

Candidates:

JSON DBs
- MongoDB
- CouchDB
Graph DBs
- Cayley
- HypergraphDB
- Neo4j
- Tinkerpop

Go 1.6: panic: runtime error: cgo argument has Go pointer to Go pointer

This should be golang/go#14210.

$ sudo EQ_DEBUG=1 ./bin/earthquake inspectors ethernet -autopilot ~/WORK/ether.toml -nfq-number 42
..
panic: runtime error: cgo argument has Go pointer to Go pointer [recovered]
        panic: runtime error: cgo argument has Go pointer to Go pointer

goroutine 1 [running]:
panic(0xbf6fc0, 0xc8211b9d00)
        /home/suda/go/src/runtime/panic.go:464 +0x3e6
github.com/osrg/earthquake/earthquake/util/core.Recoverer()
        /home/suda/gopath/src/github.com/osrg/earthquake/earthquake/util/core/coreutil.go:50 +0x106
panic(0xbf6fc0, 0xc8211b9d00)
        /home/suda/go/src/runtime/panic.go:426 +0x4e9
github.com/AkihiroSuda/go-netfilter-queue._cgoCheckPointer0(0xa8c7e0, 0xc8211abe58, 0xc8211b9cf0, 0x1, 0x1, 0xa4d380)
        github.com/AkihiroSuda/go-netfilter-queue/_obj/_cgo_gotypes.go:59 +0x4d
github.com/AkihiroSuda/go-netfilter-queue.NewNFQueue(0x100013a002a, 0xffff, 0xdd0b60, 0x0, 0x0)
        /home/suda/gopath/src/github.com/AkihiroSuda/go-netfilter-queue/netfilter.go:101 +0x47a
github.com/osrg/earthquake/earthquake/inspector/ethernet.(*NFQInspector).Serve(0xc8200e1600, 0x0, 0x0)
        /home/suda/gopath/src/github.com/osrg/earthquake/earthquake/inspector/ethernet/ethernet_nfq.go:53 +0x2c4
github.com/osrg/earthquake/earthquake/cli/inspectors.runEtherInspector(0xc82000a180, 0x4, 0x4, 0x1)
        /home/suda/gopath/src/github.com/osrg/earthquake/earthquake/cli/inspectors/ethernet.go:125 +0x903
github.com/osrg/earthquake/earthquake/cli/inspectors.etherCmd.Run(0xc82000a180, 0x4, 0x4, 0xc8211d1b30)
        /home/suda/gopath/src/github.com/osrg/earthquake/earthquake/cli/inspectors/ethernet.go:62 +0x35
github.com/osrg/earthquake/earthquake/cli/inspectors.(*etherCmd).Run(0x18f20e8, 0xc82000a180, 0x4, 0x4, 0xeaaec0)
        <autogenerated>:2 +0xab
github.com/mitchellh/cli.(*CLI).Run(0xc8211b63c0, 0xc8211af2c0, 0x0, 0x0)
        /home/suda/gopath/src/github.com/mitchellh/cli/cli.go:153 +0x56e
github.com/osrg/earthquake/earthquake/cli.inspectorsCmd.Run(0xc82000a170, 0x5, 0x5, 0xc8211d1d28)
        /home/suda/gopath/src/github.com/osrg/earthquake/earthquake/cli/inspectors.go:51 +0x1f2
github.com/osrg/earthquake/earthquake/cli.(*inspectorsCmd).Run(0x18f20e8, 0xc82000a170, 0x5, 0x5, 0xeaaea8)
        <autogenerated>:5 +0xab
github.com/mitchellh/cli.(*CLI).Run(0xc8211b6300, 0xc8211af170, 0x0, 0x0)
        /home/suda/gopath/src/github.com/mitchellh/cli/cli.go:153 +0x56e
github.com/osrg/earthquake/earthquake/cli.CLIMain(0xc82000a150, 0x7, 0x7, 0x0)
        /home/suda/gopath/src/github.com/osrg/earthquake/earthquake/cli/main.go:36 +0x2d6
main.main()
        /home/suda/gopath/src/github.com/osrg/earthquake/earthquake/main.go:25 +0x3b

Project Rename: Earthquake --> Namazu

We decided to rename "Earthquake" to "Namazu".

Namazu (鯰) means a catfish in Japanese: https://en.wiktionary.org/wiki/%E9%AF%B0#Japanese

Plan:

github repository: osrg/namazu
executable name: nmz (namazu has been already taken by http://www.namazu.org/ (https://packages.debian.org/jessie/namazu2))
- installation: go get github.com/osrg/namazu/nmz
twitter: NamazuTest

ETA: by the end of this April

init command error occurs when specifying different device to materials dir and storage dir

-----------------
root@ubuntu-01:~/earthquake# bin/earthquake init example/zk.byteman.add_node/config_dumb.json 
example/zk.byteman.add_node/materials /eq_test
failed to link (src:
example/zk.byteman.add_node/materials/quorumStart.sh(example/zk.byteman.add_node/materials/quorumStart.sh), dst: 
/eq_test/materials/quorumStart.sh): link example/zk.byteman.add_node/materials/quorumStart.sh 
/eq_test/materials/quorumStart.sh: invalid cross-device link
link example/zk.byteman.add_node/materials/quorumStart.sh /eq_test/materials/quorumStart.sh: invalid cross-device link
-----------------

Not only link, and add copy options.

Use package manager (govendor or something else)

govendor seems friendly to go get because it utilizes the standard vendor directory

https://github.com/kardianos/govendor

Apply Namazu to logcabin

Originally suggested by @siddontang at logcabin/logcabin#214

Perhaps we can test his Jepsen scenario with Earthquake process inspector and filesystem inspector.
https://github.com/aphyr/jepsen/tree/master/logcabin/test/jepsen

We also may be able to use C inspector, but I don't think it still works.

scientific paper

hello

I'm trouble finding if you published any paper related to namazu? Please provide a link and bibtex if you can

Support external volume (`-v`) for `earthquake-container`

external volume should support Filesystem inspector

Collect the logs of all processes

in one a file.

Use three-valued logic in verification to handle experiment failure

Testee system can get weird due to invalid configuration of Earthquake itself.

For example, when interval parameter of random explorer is extremely big, a ZooKeeper instance can get weird.
(stat command returns This ZooKeeper instance is not currently serving requests)
Such a weird state should not be regarded as a bug of ZooKeeper.
We might need three-value logic verification to handle such a weird state.

ZK Ethernet Inspector: support a case where ports are dynamically assigned

In JUnit, ports are dynamically assigned

`nmz inspectors proc -cmd ...` lacks stdio

#151 introduced nmz inspectors proc -cmd ..., but it lacks stdio.

how to reproduce the issue:

$ ./namazu/bin/nmz inspectors proc -cmd yes

expected output:

y
y
y
y
y
..

actual output: (empty)

PTAL @mitake

Circle CI is failing

https://circleci.com/gh/osrg/earthquake/124

Setting up fuse (2.9.2-4ubuntu4.15.04.1) ...

Creating fuse group...

Adding group `fuse' (GID 113) ...

Done.

Creating fuse device...

mknod: 'fuse-': Operation not permitted

makedev fuse c 10 229 root root 0660: failed

chown: cannot access '/dev/fuse': No such file or directory

dpkg: error processing package fuse (--configure):

 subprocess installed post-installation script returned error exit status 1

Errors were encountered while processing:

 fuse

E: Sub-process /usr/bin/dpkg returned an error code (1)
lxc-start: The container failed to start.
lxc-start: Additional information can be obtained by setting the --logfile and --logpriority options.

docker build -t osrg/earthquake . returned exit code 1

INFO[0273] The command [/bin/sh -c apt-get install -y default-jdk maven] returned a non-zero code: 100 Action failed: docker build -t osrg/earthquake .

summary command fail when lost the result file of part

-----------------
failed to open history 00001390, open /disk2/eq_test_dumb/00001390/result: no such file or directory
-----------------

Inspectors CLI: `-autopilot config.toml` should be set by default (with an empty config)

namazu has error when using O_DIRECT

Hello:
I wrote a program to verify the NAMAZU file system fault injection.
./nmz inspectors fs -mount-point /tmp/nmzfs-mnt -original-dir /tmp/nmzfs-orig -autopilot config.toml

The program are as follows:
test_fs.txt

I open a file that uses a direct IO option.
If the normal directory below /tmp/nmzfs-orig for write operation, you can write.But in the /tmp/nmzfs-mnt directory below operation, write will be wrong.
The error is:
write ./direct_io.data failed: Invalid argument
write ./direct_io.data failed: Invalid argument
write ./direct_io.data failed: Invalid argument

I checked the source code, found finally mounted by fusermount.So I added the direct_io option to fusermount in namazu/vendor/github.com/osrg/hookfs/hookfs/server.go.

pathFs := pathfs.NewPathNodeFs(hookfs, nil)
conn := nodefs.NewFileSystemConnector(pathFs.Root(), opts)
originalAbs, _ := filepath.Abs(hookfs.Original)
var the_opt = []string{"nonempty","direct_io"}
mOpts := &fuse.MountOptions{
AllowOther: true,
Name: hookfs.FsName,
FsName: originalAbs,
Options: the_opt,
}

But fusermount does not support the direct_io option.The error is:

[NMZ-INF] 16:34:40.02: fusermount the cmd is [/bin/fusermount /tmp/nmzfs-mnt -o nonempty,direct_io,allow_other,subtype=hookfs,fsname=/tmp/nmzfs-orig]
(at mount_linux.go:50)
[NMZ-WRN] 16:34:40.02: ignoring restPort: -1 (at endpoint.go:86)
[NMZ-WRN] 16:34:40.02: ignoring pbPort: -1 (at endpoint.go:97)
/bin/fusermount: mount failed: Invalid argument
[NMZ-CRT] 16:34:40.02: fusermount exited with code 256
(at fs.go:103)
[NMZ-CRT] 16:34:40.02: PANIC: fusermount exited with code 256
(at coreutil.go:49)
[NMZ-INF] 16:34:40.02: Hint: For debug info, please set "NMZ_DEBUG" to 1. (at coreutil.go:53)

Can you help me solve the problem? Looking forward to your early reply, thank you very much.

Golang plugin

Go 1.5 (ships in August 2015) supports c-shared/shared lib feature.

buildmode=c-shared seems matured well, but buildmode=shared seems not.
http://blog.nzlov.com/article/46/go15testsharedlib.html

earthquake-container: cannot parse explorePolicyParam

Earthquake container cannot parse explorePolicyParam correctly, and hence default values are used.

In full-stack Earthquake, int64 values are converted to float64 ones by {{config.DumpToJsonFile}}.
https://github.com/osrg/earthquake/blob/89221e5405072010607526f7797509bc4d7901f1/earthquake/util/config/config.go#L58

However, in Earthquake container, as {{config.DumpToJsonFile}} is not used, the glitch happens.

I'm going to eliminate {{config.DumpToJsonFile}} and refactor configuration functions.

Include packet []byte in PacketEvent

Now earthquake has its own Ethernet inspector and does not depend on pyearthquake for Ethernet inspection.
So including packet []byte in PacketEvent makes a sense for those who want to analyze []byte in his/her own ExplorePolicy.
Perhaps @mitake is interested in this.

State hashing should not depend on Python's hash()

Current state hashing (pyearthquake.orchestrator.state.StateBase.hash) depends on unreliable Python hash() function.
https://docs.python.org/3/whatsnew/3.3.html

State hashing should be independently implemented.

Also, some special events such as InspectionEndEvent should be ignored when computing hash.

ethernet_test fails intermittently

https://travis-ci.org/osrg/earthquake

Seems related to socket shutdown.

https://github.com/osrg/earthquake/blob/8dcaf8be0e6fb206d0fd853078ee4abba9dafa25/earthquake/inspector/ethernet/ethernet_test.go

https://travis-ci.org/osrg/earthquake/builds/113319662

[EQ-DBG] 05:28:37.06: LOCAL EP handled action Signal{map[string]interface {}{"uuid":"92fd2562-6d82-4df7-89e2-62e56f6f8808", "entity":"_dummy_entity_id", "class":"EventAcceptanceAction", "option":map[string]interface {}{}, "type":"action", "event_uuid":"4831b94c-1233-4a02-b280-ce6806c8b6b3"}} (at log.go:210) 
[EQ-INF] 05:28:37.06: Shutting down.. (at log.go:266) 
PASS
coverage: 46.2% of statements
Assertion failed: pfd.revents & POLLIN (signaler.cpp:193)
SIGABRT: abort
PC=0x7f90e2404cc9 m=3
signal arrived during cgo execution
goroutine 25 [syscall, locked to thread]:
runtime.cgocall(0x9b5910, 0xc82003fd08, 0x0)
    /home/travis/.gimme/versions/go/src/runtime/cgocall.go:123 +0x97 fp=0xc82003fcc0 sp=0xc82003fc98
github.com/vaughan0/go-zmq._Cfunc_zmq_poll(0xc821976580, 0x3, 0xffffffffffffffff, 0xc800000000)
    ??:0 +0x64 fp=0xc82003fd08 sp=0xc82003fcc0
github.com/vaughan0/go-zmq.(*PollSet).Poll(0xc82003fee0, 0xffffffffffffffff, 0x0, 0x2, 0x0)
    /home/travis/gopath/src/github.com/vaughan0/go-zmq/poll.go:107 +0x22f fp=0xc82003fda0 sp=0xc82003fd08
github.com/vaughan0/go-zmq.(*Channels).processSockets(0xc82001e420)
    /home/travis/gopath/src/github.com/vaughan0/go-zmq/channels.go:114 +0x6ea fp=0xc82003ffa8 sp=0xc82003fda0
runtime.goexit()
    /home/travis/.gimme/versions/go/src/runtime/asm_amd64.s:2006 +0x1 fp=0xc82003ffb0 sp=0xc82003ffa8
created by github.com/vaughan0/go-zmq.(*Socket).ChannelsBuffer
    /home/travis/gopath/src/github.com/vaughan0/go-zmq/channels.go:38 +0x3e2
goroutine 1 [running]:
    goroutine running on other thread; stack unavailable
goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /home/travis/.gimme/versions/go/src/runtime/asm_amd64.s:2006 +0x1
goroutine 20 [semacquire]:
sync.runtime_Syncsemacquire(0xc820064e10)
    /home/travis/.gimme/versions/go/src/runtime/sema.go:241 +0x1a0
sync.(*Cond).Wait(0xc820064e00)
    /home/travis/.gimme/versions/go/src/sync/cond.go:63 +0x85
github.com/cihub/seelog.(*asyncLoopLogger).processItem(0xc8200c41b0, 0x0)
    /home/travis/gopath/src/github.com/cihub/seelog/behavior_asynclooplogger.go:50 +0x14d
github.com/cihub/seelog.(*asyncLoopLogger).processQueue(0xc8200c41b0)
    /home/travis/gopath/src/github.com/cihub/seelog/behavior_asynclooplogger.go:63 +0x48
created by github.com/cihub/seelog.NewAsyncLoopLogger
    /home/travis/gopath/src/github.com/cihub/seelog/behavior_asynclooplogger.go:40 +0xd9
goroutine 21 [semacquire]:
sync.runtime_Syncsemacquire(0xc820064f90)
    /home/travis/.gimme/versions/go/src/runtime/sema.go:241 +0x1a0
sync.(*Cond).Wait(0xc820064f80)
    /home/travis/.gimme/versions/go/src/sync/cond.go:63 +0x85
github.com/cihub/seelog.(*asyncLoopLogger).processItem(0xc8200c42d0, 0x0)
    /home/travis/gopath/src/github.com/cihub/seelog/behavior_asynclooplogger.go:50 +0x14d
github.com/cihub/seelog.(*asyncLoopLogger).processQueue(0xc8200c42d0)
    /home/travis/gopath/src/github.com/cihub/seelog/behavior_asynclooplogger.go:63 +0x48
created by github.com/cihub/seelog.NewAsyncLoopLogger
    /home/travis/gopath/src/github.com/cihub/seelog/behavior_asynclooplogger.go:40 +0xd9
goroutine 4 [select]:
github.com/osrg/earthquake/earthquake/endpoint/local.(*LocalEndpoint).eventRoutine(0x116a920)
    /home/travis/gopath/src/github.com/osrg/earthquake/earthquake/endpoint/local/localendpoint.go:44 +0x478
created by github.com/osrg/earthquake/earthquake/endpoint/local.(*LocalEndpoint).Start
    /home/travis/gopath/src/github.com/osrg/earthquake/earthquake/endpoint/local/localendpoint.go:100 +0x1da
goroutine 5 [select]:
github.com/osrg/earthquake/earthquake/endpoint/local.(*LocalEndpoint).actionRoutine(0x116a920)
    /home/travis/gopath/src/github.com/osrg/earthquake/earthquake/endpoint/local/localendpoint.go:71 +0x482
created by github.com/osrg/earthquake/earthquake/endpoint/local.(*LocalEndpoint).Start
    /home/travis/gopath/src/github.com/osrg/earthquake/earthquake/endpoint/local/localendpoint.go:101 +0x1fc
goroutine 6 [select]:
github.com/osrg/earthquake/earthquake/util/mockorchestrator.(*MockOrchestrator).routine(0xc821b09780)
    /home/travis/gopath/src/github.com/osrg/earthquake/earthquake/util/mockorchestrator/mockorchestrator.go:81 +0x225
created by github.com/osrg/earthquake/earthquake/util/mockorchestrator.(*MockOrchestrator).Start
    /home/travis/gopath/src/github.com/osrg/earthquake/earthquake/util/mockorchestrator/mockorchestrator.go:99 +0x43
goroutine 23 [chan receive]:
github.com/osrg/earthquake/earthquake/inspector/transceiver.(*LocalTransceiver).routine(0x1169b70)
    /home/travis/gopath/src/github.com/osrg/earthquake/earthquake/inspector/transceiver/localtransceiver.go:74 +0x6d
created by github.com/osrg/earthquake/earthquake/inspector/transceiver.(*LocalTransceiver).Start
    /home/travis/gopath/src/github.com/osrg/earthquake/earthquake/inspector/transceiver/localtransceiver.go:85 +0x43
goroutine 24 [select]:
github.com/vaughan0/go-zmq.(*Channels).processOutgoing(0xc82001e420)
    /home/travis/gopath/src/github.com/vaughan0/go-zmq/channels.go:72 +0x3b2
created by github.com/vaughan0/go-zmq.(*Socket).ChannelsBuffer
    /home/travis/gopath/src/github.com/vaughan0/go-zmq/channels.go:37 +0x3c0
rax    0x0
rbx    0x0
rcx    0xffffffffffffffff
rdx    0x6
rdi    0x525e
rsi    0x5260
rbp    0x1
rsp    0x7f90e0b4baa8
r8     0x7f90e0b4c700
r9     0xc1
r10    0x8
r11    0x202
r12    0x0
r13    0x0
r14    0x7f90e0b4bd08
r15    0x7f90e0b4bd04
rip    0x7f90e2404cc9
rflags 0x202
cs     0x33
fs     0x0
gs     0x0
FAIL    github.com/osrg/earthquake/earthquake/inspector/ethernet    1.239s

fault injection: killing process

[EXPERIMENTAL] AspectGo integration

AspectGo is my on-going work which introduces AOP (aspect-oriented programming) paradigm to Go language: https://github.com/AkihiroSuda/aspectgo

My idea is to implement "Go inspector" using AspectGo, as well as Java inspector.

Can not get mflag:Docker has removed pkg/mflag

Ref PR:moby/moby#25354. Docker has removed pkg/mflag with new cli framework . new user can not build namazu.you may need to fix it.

Enable fs inspector in container mode

Namazu supports proc inspector and ethernet inspector in container mode currently. If Namazu can support fs inspector in container mode either, it would be perfect.

inspector, java: fix typo of org.osrg.earthquake to net.osrg.earthquake

We don't possess osrg.org.

Apply Namazu to TiKV, TiDB

Originally mentioned by @ngaut at https://github.com/pingcap/weekly/blob/master/20160327/how_we_test_tikv.pdf

Perhaps testing TiKV+TiDB+pd+etcd environment is interesting.

There is also Jepsen scenario for old TiDB release: https://github.com/siddontang/jepsen-tidb

happened "out of memory" by earthquake tools visualize --mode gnuplot command.

It is necessary to refine the Implementation of memory use

Print a warning if `sched_setattr` returns EPERM

runtime panic of process inspector

Below panic was caused during my testing etcd e2e test:

panic: Invalid parameter n: 0

goroutine 37 [running]:
panic(0xa304a0, 0xc420fdb6c0)
        /usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/osrg/namazu/vendor/github.com/leesper/go_rng.DirichletGenerator.FlatDirichlet(0xc42048c208, 0x0, 0xc421e05b60, 0x0, 0xc421e05b60)
        /home/mitake/gopath/src/github.com/osrg/namazu/vendor/github.com/leesper/go_rng/dirichlet.go:56 +0x1ad
github.com/osrg/namazu/nmz/explorepolicy/random.(*dirichlet).dirichletSchedDeadline(0xc42048c2a8, 0x16414c0, 0x0, 0x0, 0xf4240, 0x3ff0000000000000, 0x0)
        /home/mitake/gopath/src/github.com/osrg/namazu/nmz/explorepolicy/random/dirichlet.go:56 +0x8b
github.com/osrg/namazu/nmz/explorepolicy/random.(*dirichlet).Action(0xc42048c2a8, 0xc421e5c540, 0xc4220c2050, 0xc4200d6b60, 0xc4200d6b60, 0xc4220c2050)
        /home/mitake/gopath/src/github.com/osrg/namazu/nmz/explorepolicy/random/dirichlet.go:42 +0x9c
github.com/osrg/namazu/nmz/explorepolicy/random.(*Random).makeActionForEvent(0xc42018cb40, 0x10e42e0, 0xc421e5c540, 0xc42129bf78, 0x2, 0x0, 0x0)
        /home/mitake/gopath/src/github.com/osrg/namazu/nmz/explorepolicy/random/randompolicy.go:303 +0xa7
github.com/osrg/namazu/nmz/explorepolicy/random.(*Random).dequeueEventRoutine(0xc42018cb40)
        /home/mitake/gopath/src/github.com/osrg/namazu/nmz/explorepolicy/random/randompolicy.go:323 +0x10d
created by github.com/osrg/namazu/nmz/explorepolicy/random.New
        /home/mitake/gopath/src/github.com/osrg/namazu/nmz/explorepolicy/random/randompolicy.go:118 +0x1d6

I used a script like below for iterating the test:

#! /bin/bash

for i in `seq 1 1000`; do
    GOPATH=`pwd`/gopath nmz inspectors proc -watch-interval 1ms -stdout nmz/log-$i -autopilot dirichlet.toml -cmd "./e2e.test -bin-dir bin -test.run TestCtlV3Lock"
done

dirichlet.toml is this:

explorePolicy = "random"
[explorePolicyParam]
 procPolicy = "dirichlet"

Any ideas? @AkihiroSuda

Evaluation of process inspector

We need to quantitatively evaluate the process inspector as well as the Ethernet inspector
(FOSDEM presentation slide)

Make EQ itself fault-tolerant

Orchestrator must tolerant to unexpected deaths/revivals of processes.

In current impl., an action being sent to dead process is lost.
Even if process has revived, an action is lost because revived process has a different TCP socket.
5ee85d0

Workarounds

Introduce ack for actions
Use ZooKeeper or Consul (too complicated?)

Rename [containerParam] to [container]

containerParam is too redundant.
container is enough.