move-below should be prevented upon:

• replicating from newer major version to older major version (e.g. 5.6 to 5.5)
• binlog_format incompatible (row|mixed->statement, row->mixed)
• no (log_bin && log_slave_updates) on current sibling (==new master)
• replication filters in place and are incompatible (will probably not handle at this stage)

Steps followed:

1. Configured conf file and later started daemon
2. Discovery -> searched for master (127.0.0.1 and port 3390)
3. Discovery --> searched for slave (127.0.0.1 and port 3391)

Master and Slave should be alligned properly as mentioned in your documentation, this is not happening in my case. (refer attached screenshot)

now using function on last_checked leading to dis-use of newly introduced index on last_checked.

(inject sending wrong username/password)

Use case: MySQL has full disk; or is attempting a STOP SLAVE which takes time.
At such case orchestrator's connection will hang on SHOW SLAVE STATUS; and the next discovery cycle will open yet another connection, leading to an infinite pileup of connections.

~$ cd /usr/local/orchestrator && ./orchestrator http
[martini] listening on :3000
2014-06-10 11:07:04 ERROR dial tcp 127.0.0.1:5532: connection refused
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x20 pc=0x50676e]

goroutine 12 [running]:
runtime.panic(0x70cf60, 0xb285a8)
/usr/local/go/src/pkg/runtime/panic.c:266 +0xb6
database/sql.(*Rows).Close(0x0, 0xc21005a580, 0x70)
/usr/local/go/src/pkg/database/sql/sql.go:1576 +0x1e
github.com/outbrain/sqlutils.QueryRowsMap(0xc21005a600, 0xc21005a580, 0x70, 0x7fdd57bb1e58, 0x7fdd57d29b48, ...)
/home/snoach/dev/outbrain/github/orchestrator/src/github.com/outbrain/sqlutils/sqlutils.go:143 +0x119
github.com/outbrain/orchestrator/inst.ReadOutdatedInstanceKeys(0x674460, 0xc210052700, 0xecb289080, 0x3457510, 0xb2ffa0)
/home/snoach/dev/outbrain/github/orchestrator/src/github.com/outbrain/orchestrator/inst/instance_dao.go:466 +0x1ee
github.com/outbrain/orchestrator/logic.ContinuousDiscovery()
/home/snoach/dev/outbrain/github/orchestrator/src/github.com/outbrain/orchestrator/logic/orchestrator.go:130 +0x10e
created by github.com/outbrain/orchestrator/app.Http
/home/snoach/dev/outbrain/github/orchestrator/src/github.com/outbrain/orchestrator/app/http.go:50 +0x1db

goroutine 1 [IO wait]:
net.runtime_pollWait(0x7fdd57d299a0, 0x72, 0x0)
/usr/local/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(_pollDesc).Wait(0xc2100d2a70, 0x72, 0x7fdd57d27098, 0xb)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(_pollDesc).WaitRead(0xc2100d2a70, 0xb, 0x7fdd57d27098)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(_netFD).accept(0xc2100d2a10, 0x826788, 0x0, 0x7fdd57d27098, 0xb)
/usr/local/go/src/pkg/net/fd_unix.go:382 +0x2c2
net.(_TCPListener).AcceptTCP(0xc210000408, 0x51db2b, 0x7fdd57b97ae0, 0x51db2b)
/usr/local/go/src/pkg/net/tcpsock_posix.go:233 +0x47
net.(_TCPListener).Accept(0xc210000408, 0x7fdd57d29a48, 0xc210000428, 0xc21005a500, 0x0)
/usr/local/go/src/pkg/net/tcpsock_posix.go:243 +0x27
net/http.(_Server).Serve(0xc2100c8d20, 0x7fdd57d28a00, 0xc210000408, 0x0, 0x0)
/usr/local/go/src/pkg/net/http/server.go:1622 +0x91
net/http.(_Server).ListenAndServe(0xc2100c8d20, 0xc2100c8d20, 0x7fdd57b97be0)
/usr/local/go/src/pkg/net/http/server.go:1612 +0xa0
net/http.ListenAndServe(0xc2100003e0, 0x5, 0x7fdd57d28980, 0xc2100513c0, 0x1, ...)
/usr/local/go/src/pkg/net/http/server.go:1677 +0x6d
github.com/go-martini/martini.(_Martini).Run(0xc2100513c0)
/usr/share/golang/src/github.com/go-martini/martini/martini.go:84 +0x303
github.com/outbrain/orchestrator/app.Http(0x7fffc347c401)
/home/snoach/dev/outbrain/github/orchestrator/src/github.com/outbrain/orchestrator/app/http.go:57 +0x224
main.main()
/home/snoach/dev/outbrain/github/orchestrator/src/github.com/outbrain/orchestrator/main.go:59 +0x518

goroutine 14 [chan receive]:
github.com/outbrain/orchestrator/logic.handleDiscoveryRequests(0x0, 0x0)
/home/snoach/dev/outbrain/github/orchestrator/src/github.com/outbrain/orchestrator/logic/orchestrator.go:39 +0x3d
created by github.com/outbrain/orchestrator/logic.ContinuousDiscovery
/home/snoach/dev/outbrain/github/orchestrator/src/github.com/outbrain/orchestrator/logic/orchestrator.go:126 +0x67

goroutine 13 [finalizer wait]:
runtime.park(0x40ad80, 0xb2e8f8, 0xb2ace8)
/usr/local/go/src/pkg/runtime/proc.c:1342 +0x66
runfinq()
/usr/local/go/src/pkg/runtime/mgc0.c:2279 +0x84
runtime.goexit()
/usr/local/go/src/pkg/runtime/proc.c:1394

goroutine 17 [chan receive]:
database/sql.(*DB).connectionOpener(0xc21005a600)
/usr/local/go/src/pkg/database/sql/sql.go:574 +0x3e
created by database/sql.Open
/usr/local/go/src/pkg/database/sql/sql.go:436 +0x24d
mahesh@mahesh-OptiPlex-990:/usr/local/orchestrator$ sudo cd /usr/local/orchestrator && ./orchestrator http
[sudo] password for mahesh:
sudo: cd: command not found
mahesh@mahesh-OptiPlex-990:/usr/local/orchestrator$ sudo ./orchestrator http
[martini] listening on :3000
2014-06-10 11:08:24 ERROR dial tcp 127.0.0.1:5532: connection refused
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x20 pc=0x50676e]

goroutine 12 [running]:
runtime.panic(0x70cf60, 0xb285a8)
/usr/local/go/src/pkg/runtime/panic.c:266 +0xb6
database/sql.(*Rows).Close(0x0, 0xc210058500, 0x70)
/usr/local/go/src/pkg/database/sql/sql.go:1576 +0x1e
github.com/outbrain/sqlutils.QueryRowsMap(0xc210058580, 0xc210058500, 0x70, 0x7f17c6340e58, 0x7f17c64b8a90, ...)
/home/snoach/dev/outbrain/github/orchestrator/src/github.com/outbrain/sqlutils/sqlutils.go:143 +0x119
github.com/outbrain/orchestrator/inst.ReadOutdatedInstanceKeys(0x674460, 0xc210050700, 0xecb2890d0, 0x29b4e2e0, 0xb2ffa0)
/home/snoach/dev/outbrain/github/orchestrator/src/github.com/outbrain/orchestrator/inst/instance_dao.go:466 +0x1ee
github.com/outbrain/orchestrator/logic.ContinuousDiscovery()
/home/snoach/dev/outbrain/github/orchestrator/src/github.com/outbrain/orchestrator/logic/orchestrator.go:130 +0x10e
created by github.com/outbrain/orchestrator/app.Http
/home/snoach/dev/outbrain/github/orchestrator/src/github.com/outbrain/orchestrator/app/http.go:50 +0x1db

goroutine 1 [IO wait]:
net.runtime_pollWait(0x7f17c64b8960, 0x72, 0x0)
/usr/local/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(_pollDesc).Wait(0xc2100cfa70, 0x72, 0x7f17c64b6098, 0xb)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(_pollDesc).WaitRead(0xc2100cfa70, 0xb, 0x7f17c64b6098)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(_netFD).accept(0xc2100cfa10, 0x826788, 0x0, 0x7f17c64b6098, 0xb)
/usr/local/go/src/pkg/net/fd_unix.go:382 +0x2c2
net.(_TCPListener).AcceptTCP(0xc210000408, 0x18, 0xc210039010, 0x51d903)
/usr/local/go/src/pkg/net/tcpsock_posix.go:233 +0x47
net.(_TCPListener).Accept(0xc210000408, 0x7f17c64b7968, 0xc2100d0060, 0x0, 0x0)
/usr/local/go/src/pkg/net/tcpsock_posix.go:243 +0x27
net/http.(_Server).Serve(0xc2100c5cd0, 0x7f17c64b79c0, 0xc210000408, 0x0, 0x0)
/usr/local/go/src/pkg/net/http/server.go:1622 +0x91
net/http.(_Server).ListenAndServe(0xc2100c5cd0, 0xc2100c5cd0, 0x7f17c6326be0)
/usr/local/go/src/pkg/net/http/server.go:1612 +0xa0
net/http.ListenAndServe(0xc2100003e0, 0x5, 0x7f17c64b7940, 0xc21004f3c0, 0x1, ...)
/usr/local/go/src/pkg/net/http/server.go:1677 +0x6d
github.com/go-martini/martini.(_Martini).Run(0xc21004f3c0)
/usr/share/golang/src/github.com/go-martini/martini/martini.go:84 +0x303
github.com/outbrain/orchestrator/app.Http(0x7fff795c3901)
/home/snoach/dev/outbrain/github/orchestrator/src/github.com/outbrain/orchestrator/app/http.go:57 +0x224
main.main()
/home/snoach/dev/outbrain/github/orchestrator/src/github.com/outbrain/orchestrator/main.go:59 +0x518

goroutine 14 [chan receive]:
github.com/outbrain/orchestrator/logic.handleDiscoveryRequests(0x0, 0x0)
/home/snoach/dev/outbrain/github/orchestrator/src/github.com/outbrain/orchestrator/logic/orchestrator.go:39 +0x3d
created by github.com/outbrain/orchestrator/logic.ContinuousDiscovery
/home/snoach/dev/outbrain/github/orchestrator/src/github.com/outbrain/orchestrator/logic/orchestrator.go:126 +0x67

goroutine 13 [finalizer wait]:
runtime.park(0x40ad80, 0xb2e8f8, 0xb2ace8)
/usr/local/go/src/pkg/runtime/proc.c:1342 +0x66
runfinq()
/usr/local/go/src/pkg/runtime/mgc0.c:2279 +0x84
runtime.goexit()
/usr/local/go/src/pkg/runtime/proc.c:1394

goroutine 16 [chan receive]:
database/sql.(*DB).connectionOpener(0xc210058580)
/usr/local/go/src/pkg/database/sql/sql.go:574 +0x3e
created by database/sql.Open
/usr/local/go/src/pkg/database/sql/sql.go:436 +0x24d

prevent move-up when:

• unseen
• unchecked
• not replicating
• lag > threshold

prevent move-below when either host:

• unseen
• unchecked
• not replicating
• lag > threshold

should minimize only subtree under node. This was introduced lately by new "svg" variables induced into "update" function, resources/public/js/cluster-tree.js

if move operation fails, make sure slaves are started again (they are stopped at the beginning of the operation)

hi,

orchestrator master is freezing after around 20 hours from restart.
the number of EST TCP connections at that point is around 20k.
removing SSL did not help.

thanks.
shaharf.

create an audit table; log any:

• move-(up|below),
• (begin|end)-maintenance (except for those implied by move-*)
• discover
• forget (need to consider auto-forget as well)

disabled on click/drag

otherwise when opening modal it doesn't show if an instance is in maintenance mode

web to support from within instance modal

In larger organizations, it'd be great to support a toggle in the frontend that doesn't allow changing the topology but it still a nice way to see the different cluster's topology and lag status

We kill the bas hscript which executes them, but the KILL signal does not propagate downwards.

and also on js side (label-error)

I've attempted to attach Orchestrator to a host as such:

Discover > Added host: istsbdcs01-pub (this host exists in dns and resolves)

The output returns this:

2014-06-27 19:32:15 INFO Starting discovery at {Hostname:istsbdcs01-pub.isllc.local Port:3306}
2014-06-27 19:32:15 INFO No entry for &{Hostname:istsbdcs01-pub.isllc.local Port:3306}
2014-06-27 19:32:15 ERROR lookup : no such host

Is there a reason it can't look up the host when it resolves on the server and elsewhere? Is that even relevant to the error?

It states "Last Check Invalid" and gives no further detail as to what part of the check failed. Is there a plan to extend these messages to be more specific?

The line with instances shows up to 4 dots which seem to represent: total servers in the cluster, warning critical and not available.

• It might be nice if a mouse-over describes this a little more completely to avoid us looking it up in the docs
• It might also be nice to be able to click on these dots and see only these servers (so we can find out more details about the problem etc).

I am not sure if this already exists and just isn't documented but if it doesn't exist, it'd be great if Orchestrator can support a prewritten JSON file of the clusters it should contain in the dashboard.

As an ops person automating the deployment of an Orchestrator dashboard, I can make my task a lot faster by leveraging infrastructure search tools (chef search for example) to create this instead of having to manually add clusters when a new orchestrator dashboard is added. Especially if I manage over a dozen clusters (which is the case for me)

I tested the orchestrator db user on the target DB before posting this. My lack of knowledge of martini and go may be a factor here rather than a bug but when I try to add a cluster thru the discover page, I get this error stack in the orchestrator.log

[martini] Started GET /api/discover/stdbcontact-002.sjc1.sendgrid.net/3306 for 10.0.2.2:60163
[martini] PANIC: Value not found for type auth.User
/usr/lib64/go/src/runtime/panic.go:387 (0x412bc8)
/usr/share/golang/src/github.com/go-martini/martini/router.go:352 (0x504281)
/usr/share/golang/src/github.com/go-martini/martini/router.go:229 (0x503364)
/usr/share/golang/src/github.com/go-martini/martini/router.go:112 (0x501fe8)
/usr/share/golang/src/github.com/go-martini/martini/martini.go:119 (0x504ecb)
/usr/lib64/go/src/runtime/asm_amd64.s:403 (0x43b005)
/usr/lib64/go/src/reflect/value.go:419 (0x5821b5)
/usr/lib64/go/src/reflect/value.go:296 (0x580fac)
/usr/share/golang/src/github.com/codegangsta/inject/inject.go:102 (0x5ea747)
/usr/share/golang/src/github.com/go-martini/martini/martini.go:173 (0x500171)
/usr/share/golang/src/github.com/go-martini/martini/martini.go:164 (0x500083)
/usr/share/golang/src/github.com/martini-contrib/gzip/gzip.go:37 (0x50c218)
/usr/lib64/go/src/runtime/asm_amd64.s:403 (0x43b005)
/usr/lib64/go/src/reflect/value.go:419 (0x5821b5)
/usr/lib64/go/src/reflect/value.go:296 (0x580fac)
/usr/share/golang/src/github.com/codegangsta/inject/inject.go:102 (0x5ea747)
/usr/share/golang/src/github.com/go-martini/martini/martini.go:173 (0x500171)
/usr/share/golang/src/github.com/go-martini/martini/martini.go:164 (0x500083)
/usr/share/golang/src/github.com/go-martini/martini/recovery.go:142 (0x505509)
/usr/lib64/go/src/runtime/asm_amd64.s:402 (0x43af95)
/usr/lib64/go/src/reflect/value.go:419 (0x5821b5)
/usr/lib64/go/src/reflect/value.go:296 (0x580fac)
/usr/share/golang/src/github.com/codegangsta/inject/inject.go:102 (0x5ea747)
/usr/share/golang/src/github.com/go-martini/martini/martini.go:173 (0x500171)
/usr/share/golang/src/github.com/go-martini/martini/martini.go:164 (0x500083)
/usr/share/golang/src/github.com/go-martini/martini/logger.go:25 (0x504bcf)
/usr/lib64/go/src/runtime/asm_amd64.s:403 (0x43b005)
/usr/lib64/go/src/reflect/value.go:419 (0x5821b5)
/usr/lib64/go/src/reflect/value.go:296 (0x580fac)
/usr/share/golang/src/github.com/codegangsta/inject/inject.go:102 (0x5ea747)
/usr/share/golang/src/github.com/go-martini/martini/martini.go:173 (0x500171)
/usr/share/golang/src/github.com/go-martini/martini/martini.go:69 (0x4feef4)
/usr/lib64/go/src/net/http/server.go:1703 (0x52986a)
/usr/lib64/go/src/net/http/server.go:1204 (0x527337)
/usr/lib64/go/src/runtime/asm_amd64.s:2232 (0x43d041)
2014/12/25 02:17:27 http: multiple response.WriteHeader calls
[martini] Completed 500 Internal Server Error in 1.80033ms

Config file

{
"MySQLTopologyUser": "orchestrator",
"MySQLTopologyPassword": "REDACTED",
"MySQLOrchestratorHost": "localhost",
"MySQLOrchestratorPort": 3306,
"MySQLOrchestratorDatabase": "orchestrator",
"MySQLOrchestratorUser": "orchestrator",
"MySQLOrchestratorPassword": "REDACTED",
"SlaveLagQuery": "",
"DiscoverByShowSlaveHosts": true,
"InstancePollSeconds": 60,
"UnseenInstanceForgetHours": 240,
"ReasonableReplicationLagSeconds" : 10,
"ReasonableMaintenanceReplicationLagSeconds" : 20,
"AuditPageSize": 20,
"SlaveStartPostWaitMilliseconds" : 1000,
"HTTPAuthUser": "",
"HTTPAuthPassword": ""
}

Let me know if I should post any more information.

for unseen instances (visualize by gray, black) don't show replication lag but instead time since last seen. Replication lag is irrelevant since it does not update as the instance cannot be accessed.

which apply FLUSH TABLES WITH READ LOCK + set global read_only := 1 + UNLOCK TABLES,
and set global read_only := 0, respectively

the first is relative to direct master
the second, if slavelagquery exists, is relative to cluster's master

need both because when validating move-up we only need to measure seconds-behind-master.

I ran the grants as indicated in the documentation, but I get the following error:

2014-08-14 15:05:03 ERROR Error 1227: Access denied; you need (at least one of) the REPLICATION SLAVE privilege(s) for this operation

This is the result of my "show grant":
GRANT PROCESS, SUPER ON . TO 'orchestrator_act'@'drgordbreport' IDENTIFIED BY PASSWORD 'hashedpassword'

Perhaps the documentation is out of date for the latest release?

Edit: This seems to occur even if I add the "replication slave" privilege.

And outside the condif file (may keep in config file for backward compatability)

Provide api for setting/clearing aliases

such that an instance's problem is still visualized (currently maintenance mode overrides problem visualization)

color black in the topology/search/status popovers

Orchestrator should use separate files for startup error from normal operation. A .err file is probably more appropriate when it didn't start due to an error and it wants to log what happened.

and should.

This is a common pattern in ops deployment of applications. Usually it would be a /health URL that returns, when healthy, return code 200 and a JSON showing all the things it checked. In orchestrator's case it would be a JSON that would look like this

{"orchestrator": "ok", "orchestrator_mysql": {"orchestrator_mysql_running": "ok", "orchestrator_mysql_query": "ok"}}

This would greatly simplify the task of monitoring orchestrator in any given infrastructure

and mark as "problem" when non master node is not read_only

As far as I can tell, all the information needed for a read only orchestrator can come from either REPLICATION CLIENT and REPLICATION SLAVE privileges. Yet with an instance set as ReadOnly true, I still can't even properly discover instances without the super privilege.

In many environments, granting super priv to a non localhost user is not recommended and if I am already deploying Orchestrator as a read only tool, i would think SUPER privileges shouldn't be needed.

Instead/in addition to using "MySQLTopologyUser", "MySQLTopologyPassword", support reading standard my.cnf file format, reading the "[client]" scope.

Ran 1.0.3 with --stack --debug, here's the output:

[martini] Started GET /api/discover/istsbdcs01-pub.isllc.local/3306
[martini] Completed 200 OK in 3.764292ms
2014-06-30 17:04:46 INFO Starting discovery at {Hostname:istsbdcs01-pub.isllc.local Port:3306}
2014-06-30 17:04:46 INFO No entry for &{Hostname:istsbdcs01-pub.isllc.local Port:3306}
2014-06-30 17:04:46 ERROR lookup : no such host
/home/snoach/dev/outbrain/github/orchestrator/src/github.com/outbrain/log/log.go:105 (0x42cec1)
/home/snoach/dev/outbrain/github/orchestrator/src/github.com/outbrain/log/log.go:151 (0x42d31a)
/home/snoach/dev/outbrain/github/orchestrator/src/github.com/outbrain/orchestrator/inst/instance_dao.go:154 (0x474dad)
/home/snoach/dev/outbrain/github/orchestrator/src/github.com/outbrain/orchestrator/logic/orchestrator.go:76 (0x4806ff)
/home/snoach/dev/outbrain/github/orchestrator/src/github.com/outbrain/orchestrator/logic/orchestrator.go:53 (0x480ed8)
/usr/local/go/src/pkg/runtime/proc.c:1394 (0x4177e0)

Here's the verification of DNS resolution:

[sbduser@istmysqlgw01 orchestrator]$ nslookup istsbdcs01-pub
Server: 10.1.104.30
Address: 10.1.104.30#53

Name: istsbdcs01-pub.isllc.local
Address: 10.11.1.80

[sbduser@istmysqlgw01 orchestrator]$ nslookup istsbdcs01-pub.isllc.local
Server: 10.1.104.30
Address: 10.1.104.30#53

Name: istsbdcs01-pub.isllc.local
Address: 10.11.1.80

It appears it can pull information on the mysql instance -- something it hasn't failed to do -- but something else is failing/causing it to throw the "last check invalid" error.

To reproduce, just introduce bad syntax to the JSON config file and issue a restart for Orchestrator

[vagrant@default-centos-64 ~]$ sudo /etc/init.d/orchestrator restart
Stopping orchestrator                             /etc/init.d/orchestrator: line 56: kill: (4366) - No such process
Ok
Starting orchestrator...                          Ok
[vagrant@default-centos-64 ~]$ echo $?
0

Check Orchestrator status and it clearly failed to start

[vagrant@default-centos-64 ~]$ tail /var/log/orchestrator.log
2014-12-20 22:43:45 INFO starting
2014-12-20 22:43:45 FATAL Cannot read config file: /etc/orchestrator.conf.json invalid character '"' after object key:value pair
[vagrant@default-centos-64 ~]$ pgrep orchestrator
[vagrant@default-centos-64 ~]$

I may look at the init script and see if I can make a PR for this but I want to update the chef cookbook to support the new GA release first

add status sidebar on all screens, featuring:

• maintained instances
• error servers (replication lag)
• error servers (unseen lately)
• audit?

Sometimes hosts are referred to by their IP/hostname that cannot be resolved from the orchestrator machine. This leads to duplication, having same machine appearing twice in the topology, once with IP and once with hostname.
Per topology, we can assume two servers with the same server_id are either a terrible mistake or simply the very same instance.
At the very least, orchestrator should identify this (and perhaps present as a problem).