Hello,When i use the DigestAuthentication of zookeeper to control the access of zookeeper's data,i can not get access to it from the explorer of exhibitor.Could you help me ?

I am getting the below error while runniong the below command to download the latest standalone exhibitor. I tried to get the exhibitor 1.4.3 jar using the same command it worked properly.

mvn assembly:single
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building exhibitor-war 1.0
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-assembly-plugin:2.2-beta-5:single (default-cli) @ exhibitor-war ---
[WARNING] Missing POM for com.netflix.exhibitor:exhibitor-standalone:jar:1.4.4
[WARNING] Missing POM for com.netflix.exhibitor:exhibitor-standalone:jar:1.4.4
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.867s
[INFO] Finished at: Tue Dec 18 16:50:47 CST 2012
[INFO] Final Memory: 4M/81M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.2-beta-5:single (default-cli) on project exhibitor-war: Failed to create assembly: Failed to resolve dependencies for project: exhibitor:exhibitor-war:jar:1.0: Missing:
[ERROR] ----------
[ERROR] 1) com.netflix.exhibitor:exhibitor-standalone:jar:1.4.4
[ERROR]
[ERROR] Try downloading the file manually from the project website.
[ERROR]
[ERROR] Then, install it using the command:
[ERROR] mvn install:install-file -DgroupId=com.netflix.exhibitor -DartifactId=exhibitor-standalone -Dversion=1.4.4 -Dpackaging=jar -Dfile=/path/to/file
[ERROR]
[ERROR] Alternatively, if you host your own repository you can deploy the file there:
[ERROR] mvn deploy:deploy-file -DgroupId=com.netflix.exhibitor -DartifactId=exhibitor-standalone -Dversion=1.4.4 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR]
[ERROR] Path to dependency:
[ERROR] 1) exhibitor:exhibitor-war:jar:1.0
[ERROR] 2) com.netflix.exhibitor:exhibitor-standalone:jar:1.4.4
[ERROR]
[ERROR] ----------
[ERROR] 1 required artifact is missing.
[ERROR]
[ERROR] for artifact:
[ERROR] exhibitor:exhibitor-war:jar:1.0
[ERROR]
[ERROR] from the specified remote repositories:
[ERROR] central (http://repo1.maven.org/maven2, releases=true, snapshots=false)
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

Until initial config is set, the system should consider instances LATENT

in exhibitor-cluster.js
makeServersList()
thisSpec.serverId = parseInt(parts[0]);
thisSpec.hostname = parts[1];
ERROR
it should be
thisSpec.serverId = parseInt(parts[1]);
thisSpec.hostname = parts[2];

Configuration Panel title show：
List of all servers in the ensemble (including this one) in the form: type:id:host,type:id:host,... The type should be 'S' for a standard server and 'O' for an observer.

Hello
When I use a IE browser to access exhibitor that deployed on the remote machine,an error display on the browser:
Object does not support this property or method.
exhibitor.js line:579 character:5
If i using the browser of google,it is OK.

Thank you for your attention.

Turn off auto restart for a server from Control Panel. Stop the server manually from Control Panel. The server stops successfully. Now there is no option to start the server from the UI. Here is the log snippet.

Thu Oct 04 17:01:45 GMT-07:00 2012 INFO Restart of ZooKeeper skipped due to control panel setting
Thu Oct 04 17:01:45 GMT-07:00 2012 INFO ZooKeeper related configuration has changed
Thu Oct 04 17:01:45 GMT-07:00 2012 INFO State: down
Thu Oct 04 17:01:26 GMT-07:00 2012 INFO ZooKeeper Server: Starting zookeeper ... already running as process 12387.
Thu Oct 04 17:01:26 GMT-07:00 2012 ERROR ZooKeeper Server: Using config: /zookeeper-3.4.3/bin/../conf/zoo.cfg
Thu Oct 04 17:01:26 GMT-07:00 2012 ERROR ZooKeeper Server: JMX enabled by default
Thu Oct 04 17:01:26 GMT-07:00 2012 INFO Process started via: /zookeeper-3.4.3/bin/zkServer.sh
Thu Oct 04 17:01:26 GMT-07:00 2012 INFO Kill attempted result: 0
Thu Oct 04 17:01:26 GMT-07:00 2012 INFO Attempting to start/restart ZooKeeper
Thu Oct 04 17:01:26 GMT-07:00 2012 INFO Attempting to stop instance

Jordan:

Do you mind updating your filename format for the exhibitor backups to use a slash "/"?

Right now they look like:

exhibitor-backup|log.f260296c3|1337099229000

Could you update so that the pipes are slashes? This will make the genpop bucket easier to navigate (S3 UIs treat slashes as directories and subdivide from there).

Thanks,
Jason

The filesystem-based config cannot be saved. From investigating it looks like it is not fully implemented yet as there is a TODO comment in FileSystemConfigProvider.newPseudoLock. I know Netflix is hosted out of AWS so that probably does not get used there, are there any plans to complete it in the near future or would you be open to a pull request for it?

I pulled the 1.0.0 Beta 7 tag and tried to build it with the instructions on
https://github.com/Netflix/exhibitor/wiki/Building-Exhibitor

mvn assembly:single doesn't seem to build the standalone parts (, it pulls in all of the other jars and then says '[WARNING] Cannot include project artifact: com.netflix.exhibitor:exhibitor-standalone:jar:1.0.0-beta7; it doesn't have an associated file or directory.'

The jar is created but without some code:
java -jar exhibitor-standalone-1.0.0-beta7-jar-with-dependencies.jar
Exception in thread "main" java.lang.NoClassDefFoundError: com/netflix/exhibitor/application/ExhibitorMain

All that seems to be required is to run a maven install and after that assembly:single works. I haven't looked into the way the build is setup, maybe its a dependency issue, maybe that part of the build isn't supposed to be part of the build.

It has to be done manually.

Using the exhibitor-standalone isn't ideal in my current situation.

Looks like a UI bug.

Have 3 servers for example configured. Try clicking "Automatic Instance Restarts" switch one by one to turn it off. Move to another tab and return. You will notice 2nd and 3rd one aren't really turned off.

The log has the most recent line at the top which is counter to most other things that let you see a log file. Can we have the log files ordered with the most recent thing at the bottom? (Imagine a stacktrace in reverse order... very confusing)

keyboard C&P doesn't work out of the "calculator" on the config screen. I had to right-mouse it to both copy out and to paste in.

Note, if the input used JodaTime Durations as descrived in the "milliseconds suck" issue, this wouldn't be a problem ;)

Exhibitor's kill instance mechanism relies on "jps" being available. It's not on all Linux/Unix installs.

There is no option to stop Exhibitor gracefully if run standalone. This will be useful, so the server can be started or stopped via a script.

The config properties are arranged alphabetically. It is better to leave in the same order as the user defined.

Seems like I get this on 1.1.0 whenever I try to save a configuration.

I'm not specifying a s3configprefix argument. From what I can tell, in ExhibitorMain.getS3Arguments you're creating an instance of S3ConfigAutoManageLockArguments with an underscore in the prefix (prefix + "lock"). S3ConfigProvider later tries to create a S3PseudoLock using that prefix which triggers the error.

With this I don't see any way that someone using S3 could ever save a config in 1.1.0.

Just set up exhibitor and it's pretty cool. Still haven't gotten to exercise the rolling deploy of config stuff yet, but will do that soon.

One thing I have experienced is that milliseconds suck. If you use the Joda time library, the "Duration" class is your friend. It understands ISO8601 durations, which are things like

PT60S -- 60 seconds
P1D -- 1 day
P1DT37M12S -- 1 day 37 minutes and 12 seconds

And it has a nifty "getMillis()" method on it that can convert whatever thing people put in. We use it for our time-based configs and I love it :).

As an example

new Duration("PT60S").getMillis() == 60000

returns true.

Hello
We've been trying to explore exhibitor using S3 shared configuration but we came across a very unstable and unpredictable behaviour.

We come across the following behaviours:
a. Exhibitors keep changing the configuration (server list change), they never reach stability
b. Some exhibitors never add themselves to the ensemble
c. Exhibitors deadlocked on an automatic rolling configuration change
d. Servers suddenly disappear from the ensemble and then come back
f. Servers stopped heartbeating S3 and were considered stale, even though they were alive.

We were once able to reach a stable ensemble but adding a new exhibitor or removing one did break the whole ensemble and we lost stability.

All the nodes can communicate with themselves over the appropriate ports, and we are using exhibitor 1.2.5 (from maven) with zookeeper 3.4.3.

Are we missing something here?

I have a 3 node ensemble (zk1.dev, zk2.dev and zk3.dev) running on CentOS 6.2 using ZooKeeper 3.3.5+19.1 from Cloudera. Exhibitor is running on all three nodes and it successfully launched ZooKeeper on each node. Unfortunately, the Exhibitor Control Panel on each node shows the same thing - the local server is working, but it shows java.net.ConnectException: Connection timed out for the remote nodes.

I verified that I can connect to ZooKeeper on each of the nodes:

java -cp zookeeper-3.3.5-cdh3u4.jar:lib/log4j-1.2.15.jar:lib/jline-0.9.94.jar:conf org.apache.zookeeper.ZooKeeperMain -server zk3.dev:2181
Connecting to zk3.dev:2181
2012-07-27 10:03:49,404 [myid:] - INFO  [main:Environment@97] - Client environment:zookeeper.version=3.3.5-cdh3u4--1, built on 05/07/2012 21:14 GMT
2012-07-27 10:03:49,413 [myid:] - INFO  [main:Environment@97] - Client environment:host.name=zk1.dev
2012-07-27 10:03:49,414 [myid:] - INFO  [main:Environment@97] - Client environment:java.version=1.6.0_24
2012-07-27 10:03:49,506 [myid:] - INFO  [main:Environment@97] - Client environment:java.vendor=Sun Microsystems Inc.
2012-07-27 10:03:49,507 [myid:] - INFO  [main:Environment@97] - Client environment:java.home=/usr/java/jdk1.6.0_24/jre
2012-07-27 10:03:49,507 [myid:] - INFO  [main:Environment@97] - Client environment:java.class.path=zookeeper-3.3.5-cdh3u4.jar:lib/log4j-1.2.15.jar:lib/jline-0.9.94.jar:conf
2012-07-27 10:03:49,508 [myid:] - INFO  [main:Environment@97] - Client environment:java.library.path=/usr/java/jdk1.6.0_24/jre/lib/amd64/server:/usr/java/jdk1.6.0_24/jre/lib/amd64:/usr/java/jdk1.6.0_24/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2012-07-27 10:03:49,508 [myid:] - INFO  [main:Environment@97] - Client environment:java.io.tmpdir=/tmp
2012-07-27 10:03:49,509 [myid:] - INFO  [main:Environment@97] - Client environment:java.compiler=<NA>
2012-07-27 10:03:49,509 [myid:] - INFO  [main:Environment@97] - Client environment:os.name=Linux
2012-07-27 10:03:49,510 [myid:] - INFO  [main:Environment@97] - Client environment:os.arch=amd64
2012-07-27 10:03:49,510 [myid:] - INFO  [main:Environment@97] - Client environment:os.version=2.6.32-220.2.1.el6.x86_64
2012-07-27 10:03:49,511 [myid:] - INFO  [main:Environment@97] - Client environment:user.name=root
2012-07-27 10:03:49,511 [myid:] - INFO  [main:Environment@97] - Client environment:user.home=/root
2012-07-27 10:03:49,513 [myid:] - INFO  [main:Environment@97] - Client environment:user.dir=/usr/lib/zookeeper
2012-07-27 10:03:49,515 [myid:] - INFO  [main:ZooKeeper@379] - Initiating client connection, connectString=zk3.dev:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@3343c8b3
2012-07-27 10:03:49,545 [myid:] - INFO  [main-SendThread():ClientCnxn$SendThread@1058] - Opening socket connection to server zk3.dev/10.80.10.87:2181
Welcome to ZooKeeper!
2012-07-27 10:03:49,564 [myid:] - INFO  [main-SendThread(zk3.dev:2181):ClientCnxn$SendThread@947] - Socket connection established to zk3.dev/10.80.10.87:2181, initiating session
JLine support is enabled
2012-07-27 10:03:49,642 [myid:] - INFO  [main-SendThread(zk3.dev:2181):ClientCnxn$SendThread@736] - Session establishment complete on server zk3.dev/10.80.10.87:2181, sessionid = 0x238c8e3a08d0001, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: zk3.dev:2181(CONNECTED) 0]

The ZooKeeper config generated by Exhibitor looks like the following:

#Auto-generated by Exhibitor zk1.dev
#Fri Jul 27 09:41:25 CDT 2012
com.netflix.exhibitor-rolling-hostnames=
com.netflix.exhibitor-rolling.zookeeper-data-directory=/var/lib/zookeeper
com.netflix.exhibitor-rolling.servers-spec=S\:0\:zk1.dev,S\:1\:zk2.dev,S\:2\:zk3.dev
com.netflix.exhibitor.java-environment=
com.netflix.exhibitor.zookeeper-data-directory=/var/lib/zookeeper
com.netflix.exhibitor-rolling-hostnames-index=0
com.netflix.exhibitor-rolling.java-environment=
com.netflix.exhibitor-rolling.observer-threshold=0
com.netflix.exhibitor.servers-spec=S\:0\:zk1.dev,S\:1\:zk2.dev,S\:2\:zk3.dev
com.netflix.exhibitor.cleanup-period-ms=60000
com.netflix.exhibitor.zookeeper-install-directory=/usr/lib/zookeeper
com.netflix.exhibitor.check-ms=5000
com.netflix.exhibitor-rolling.auto-manage-instances=1
com.netflix.exhibitor-rolling.cleanup-period-ms=60000
com.netflix.exhibitor-rolling.check-ms=5000
com.netflix.exhibitor.log-index-directory=/var/log/zookeeper
com.netflix.exhibitor.dead-instance-period-ms=30000
com.netflix.exhibitor-rolling.log-index-directory=/var/log/zookeeper
com.netflix.exhibitor.backup-period-ms=0
com.netflix.exhibitor-rolling.connect-port=2888
com.netflix.exhibitor-rolling.election-port=3888
com.netflix.exhibitor-rolling.backup-extra=
com.netflix.exhibitor.client-port=2181
com.netflix.exhibitor-rolling.zoo-cfg-extra=initLimit\=10&syncLimit\=5&tickTime\=2000
com.netflix.exhibitor-rolling.zookeeper-install-directory=/usr/lib/zookeeper
com.netflix.exhibitor.cleanup-max-files=5
com.netflix.exhibitor-rolling.backup-period-ms=0
com.netflix.exhibitor-rolling.client-port=2181
com.netflix.exhibitor.backup-max-store-ms=0
com.netflix.exhibitor-rolling.cleanup-max-files=5
com.netflix.exhibitor-rolling.backup-max-store-ms=0
com.netflix.exhibitor.connect-port=2888
com.netflix.exhibitor.log4j-properties=
com.netflix.exhibitor.observer-threshold=0
com.netflix.exhibitor.backup-extra=
com.netflix.exhibitor.election-port=3888
com.netflix.exhibitor.zoo-cfg-extra=initLimit\=10&syncLimit\=5&tickTime\=2000
com.netflix.exhibitor-rolling.log4j-properties=
com.netflix.exhibitor.auto-manage-instances=1
com.netflix.exhibitor-rolling.dead-instance-period-ms=30000

No errors in the logs.

What am I doing wrong?

I have a 3 node cluster where 1 of the hosts recently died. Exhibitor started up when the host came back, and ZK was properly started. I can see in the log that Exhibitor is pinging it with 'ruok'.

Via telnet I can run 'ruok' and 'srvr' to see that it is OK, and that it is a follower and serving connections. I even connected directly via zkCli.sh and created/deleted a node as a test.

For some reason, though, the node will stay forever latent according to Exhibitor until I force a rolling release (by say, changing the ZK data dir to a symlink of the same, making Exhibitor thing it needs to restart).

After the config is pushed to S3, the node is shown as up and serving (on all 3 Exhibitors).

I have only dug into the code a little so far, I'm thinking it has something to do with this in Checker.java

    if ( !isSet(config, StringConfigs.ZOOKEEPER_DATA_DIRECTORY) || !isSet(config, StringConfigs.ZOOKEEPER_INSTALL_DIRECTORY) )
    {
        return new StateAndLeader(InstanceStateTypes.LATENT, false);
    }

I only say that because the rest of the logic (ruok and srvr) I verified myself.

It is often difficult to have a shared file across the instances. It will be good to store the config in a DB - Mongo, MySQL or another ZK itself.

The log4j.properties file that is created does not contain the log4j.properties contents, but the java.env contents.

Would it be possible to have the S3ClientFactoryImpl incorporate the new IAM Instance Roles feature (implemented in the 1.3.11 AWS SDK)? This would allow the use of the IAM Role credentials without having to store/supply them to exhibitor.

public void changeCredentials(S3Credential credential) throws Exception
{
RefCountedClient newRefCountedClient = (credential != null)
? new RefCountedClient(new AmazonS3Client(new BasicAWSCredentials(key, secret)
: new RefCountedClient(new AmazonS3Client()); // Use default credential chain resolution
}

The credential chain is:

public AmazonS3Client()

Constructs a new client to invoke service methods on Amazon S3. A credentials provider chain will be used that searches for credentials in this order:

Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
Java System Properties - aws.accessKeyId and aws.secretKey
Instance Profile Credentials - delivered through the Amazon EC2 metadata service

I want to be able to deploy zookeeper & exhibitor completely automatically (e.g. via chef/puppet) without needing to touch the UI at all. For this either a way to specify the monitored-zookeeper-cluster configuration on the commandline, or alternatively a way to give exhibitor a pre-created configuration file would be very helpful.

It looks like exhibitor is using 1.0.3 Curator. I heard that there are nasty bugs fixed in 1.0.13 and wonder if Exhibitor shouldn't also be bumped up?

I'm getting an Access Denied error when trying to commit config changes in Exhibitor using S3 shared config. No additional information and nothing in the exhibitor log file. It's definitely updating the shared config because I started with an empty file and it now looks like this:

#Auto-generated by Exhibitor zk1.dev.example.ec2
#Thu Jul 26 09:31:17 CDT 2012
com.netflix.exhibitor-rolling-hostnames=
com.netflix.exhibitor-rolling.zookeeper-data-directory=
com.netflix.exhibitor-rolling.servers-spec=
com.netflix.exhibitor.java-environment=
com.netflix.exhibitor.zookeeper-data-directory=
com.netflix.exhibitor-rolling-hostnames-index=0
com.netflix.exhibitor-rolling.java-environment=
com.netflix.exhibitor-rolling.observer-threshold=0
com.netflix.exhibitor.servers-spec=
com.netflix.exhibitor.cleanup-period-ms=0
com.netflix.exhibitor.zookeeper-install-directory=
com.netflix.exhibitor.check-ms=0
com.netflix.exhibitor-rolling.auto-manage-instances=1
com.netflix.exhibitor-rolling.cleanup-period-ms=0
com.netflix.exhibitor-rolling.check-ms=0
com.netflix.exhibitor.log-index-directory=
com.netflix.exhibitor.dead-instance-period-ms=0
com.netflix.exhibitor-rolling.log-index-directory=
com.netflix.exhibitor.backup-period-ms=0
com.netflix.exhibitor-rolling.connect-port=0
com.netflix.exhibitor-rolling.election-port=0
com.netflix.exhibitor-rolling.backup-extra=
com.netflix.exhibitor.client-port=0
com.netflix.exhibitor-rolling.zoo-cfg-extra=
com.netflix.exhibitor-rolling.zookeeper-install-directory=
com.netflix.exhibitor.cleanup-max-files=0
com.netflix.exhibitor-rolling.backup-period-ms=0
com.netflix.exhibitor-rolling.client-port=0
com.netflix.exhibitor.backup-max-store-ms=0
com.netflix.exhibitor-rolling.cleanup-max-files=0
com.netflix.exhibitor-rolling.backup-max-store-ms=0
com.netflix.exhibitor.connect-port=0
com.netflix.exhibitor.log4j-properties=
com.netflix.exhibitor.observer-threshold=0
com.netflix.exhibitor.backup-extra=
com.netflix.exhibitor.election-port=0
com.netflix.exhibitor.zoo-cfg-extra=
com.netflix.exhibitor-rolling.log4j-properties=
com.netflix.exhibitor.auto-manage-instances=1
com.netflix.exhibitor-rolling.dead-instance-period-ms=0

The IAM policy for the credentials used to access S3 look like this:

{
  "Statement": [
    {
      "Sid": "Stmt1342209338501",
      "Action": [
        "s3:DeleteObject",
        "s3:DeleteObjectVersion",
        "s3:GetBucketAcl",
        "s3:GetBucketLocation",
        "s3:GetBucketNotification",
        "s3:GetBucketPolicy",
        "s3:GetBucketVersioning",
        "s3:GetObject",
        "s3:GetObjectAcl",
        "s3:GetObjectVersion",
        "s3:GetObjectVersionAcl",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:ListBucketVersions",
        "s3:ListMultipartUploadParts",
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:PutObjectVersionAcl"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::sbzk-dev/*",
        "arn:aws:s3:::sbzk-dev/"
      ]
    }
  ]
}

How can I further debug this?

• Have about 3 servers in the ensemble
• Start exhibitor to start all the instances
• Execute 'stat' on of the server from Control Panel. Works fine.
• Restart one server from the Control Panel.
• Once the server comes back, try stat on it again. It shows unknown. Looks like a UI glitch. The server is started properly, I checked the status on it.

The build instructions state that you should simply run gradle jar and you'll get a jar build/libs/exhibitor-<version>.jar. This does not work. First off, it should probably use ./gradlew which is distributed alongside the source code. Also it does not create a build top level folder nor does it create a fat jar in any of the sub directories as far as I can tell.

Change the settings and try restarting Exhibitor process. There are enabled again.

log4j:WARN No appenders could be found for logger (com.netflix.exhibitor.core.activity.ActivityLog).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

The transaction log files remain but only the snapshots are cleaned up.

1,
IndexCache.java public void releaseLogSearch(File indexDirectory) method
shoud be -- --holder.useCount; not -- --holder.lastUse

2, IndexResource.java
the mehtod
@path("get/{index-name}/{doc-id}")
@get
@produces(MediaType.APPLICATION_JSON)
public Response getResult(@PathParam("index-name") String indexName, @PathParam("doc-id") int docId) throws Exception {

not have
finally {
context.getExhibitor().getIndexCache().releaseLogSearch(logSearch.getFile());
}
It should be.

3,StandardProcessOperations.java
cleanupInstance method
not use for zookeeper 3.4.0 ,I had mailed to you !

4， IndexerUtil.java
startIndexing () method
then ZooKeeperLogParser.java
public ZooKeeperLogParser(InputStream log) throws IOException
{
logStream = BinaryInputArchive.getArchive(log);

    FileHeader fhdr = new FileHeader();
    fhdr.deserialize(logStream, "fileheader");
    validHeader = (fhdr.getMagic() == FileTxnLog.TXNLOG_MAGIC);
}

when new index use “Use Current Logs” the radio
two question ：1， The zkdata/version-2/ Directory not only hava log.xxx ,It contains snapshot 等 it can/t work
2. are you suce log is DirectoryInputStream can work?

if Directory only hava log.xxx，It can work ,but only one file can be indexed。 not the max zxid log。
so ,Now I changed the code just to indexed the max zxid log.
because I do not know your purpose 。

I did a rolling release and it did everything it was supposed to, but then filled the progress bar and sat there with "status: n/a" instead of having the modal go away. Looking at the boxen, it appears that everything went as expected, so I'm not sure what happened...

Hello,

I'm using the standalone version of exhibitor. I launch it this way :
java -Dlog4j.configuration=file:///opt/exhibitor/log4j.properties -jar /opt/exhibitor/exhibitor-war-1.0-jar-with-dependencies.jar -c file --fsconfigdir /mnt/storage/config --nodemodification true --jquerystyle black --filesystembackup /mnt/storage/backup/hostname/

When the web interface loads, the tab appears quickly and disappears.

Thanks
François

I have created cluster of 3 zookeeper servers on cent os and using zookeeper 3.4.5 and exhibitor 1.4.3.

Each machine has its own zookeeper and exhibitor server and had set the configuration in exhibitor for zookeeper cluster but exhibitor is not able to start zookeeper. Getting bellow error in the logs:

INFO com.netflix.exhibitor.core.activity.ActivityLog State: down [ActivityQueue-0]
INFO com.netflix.exhibitor.core.activity.ActivityLog Server list has changed [ActivityQueue-0]
INFO com.netflix.exhibitor.core.activity.ActivityLog Attempting to stop instance [ActivityQueue-0]
INFO com.netflix.exhibitor.core.activity.ActivityLog Attempting to start/restart ZooKeeper [ActivityQueue-0]
ERROR com.netflix.exhibitor.core.activity.ActivityLog Trying to kill running instance [ActivityQueue-0]
java.io.IOException: Cannot run program "jps": java.io.IOException: error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at com.netflix.exhibitor.core.processes.StandardProcessOperations.getPid(StandardProcessOperations.java:168)
at com.netflix.exhibitor.core.processes.StandardProcessOperations.killInstance(StandardProcessOperations.java:82)
at com.netflix.exhibitor.core.state.KillRunningInstance.call(KillRunningInstance.java:58)
at com.netflix.exhibitor.core.state.KillRunningInstance.call(KillRunningInstance.java:23)
at com.netflix.exhibitor.core.activity.ActivityQueue$1.run(ActivityQueue.java:126)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 11 more

S3 config throws Exception if key doesn't exist. But, it won't exist the first time it's used.

I need a bit more info on how to use this switch.

Is there a recommendation for how to setup the shared folder? NFS?

Can I just have a single linux box hosting this or will Exhibitor and Zookeeper fail when that box is rebooted/down?

I'm launching exhibitor using the following command line:

java -jar /opt/exhibitor/exhibitor-1.2.2-jar-with-dependencies.jar --s3credentials /opt/exhibitor/s3credentials --s3config sbzk-dev:zk-exhibitor.conf --s3backup /opt/exhibitor/s3credentials

If zk-exhibitor.conf doesn't exist in the sbzk-dev S3 bucket, Exhibitor fails to start with the following error:

Exception in thread "main" Status Code: 403, AWS Service: Amazon S3, AWS Request ID: F7EC459A9B17A0E5, AWS Error Code: AccessDenied, AWS Error Message: Access Denied, S3 Extended Request ID: GJow5e8FLktiyz1yuU76iq6HR0CBqaUMyIXtrSmPNeRmSzy3TibaPRqpyy7iO1/1
        at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:556)
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:289)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:170)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2723)
        at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:865)
        at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:771)
        at com.netflix.exhibitor.core.s3.S3ClientFactoryImpl$1.getObject(S3ClientFactoryImpl.java:98)
        at com.netflix.exhibitor.core.config.s3.S3ConfigProvider.getConfigObject(S3ConfigProvider.java:192)
        at com.netflix.exhibitor.core.config.s3.S3ConfigProvider.loadConfig(S3ConfigProvider.java:121)
        at com.netflix.exhibitor.core.config.ConfigManager.<init>(ConfigManager.java:79)
        at com.netflix.exhibitor.core.config.ConfigManager.<init>(ConfigManager.java:53)
        at com.netflix.exhibitor.core.Exhibitor.<init>(Exhibitor.java:116)
        at com.netflix.exhibitor.application.ExhibitorMain.<init>(ExhibitorMain.java:274)
        at com.netflix.exhibitor.application.ExhibitorMain.main(ExhibitorMain.java:202)

If I upload an empty config file to S3, Exhibitor starts fine. Clearly, the problem is that the absence of the config file triggers the exception.

I'm not sure what the best way would be to handle this. Given that AWS is erroneously throwing an AccessDenied error, I'm not sure how to distinguish the problem of a missing config file from other legitimate permission problems. That said, I think S3ConfigProvider.loadConfig() or S3ConfigProvider.getConfigObject() should check for the existence of the config file if the request to load it fails.

Running gradle jar caused this error:

build file '/Users/tbernard/repos/exhibitor/tmp/build.gradle': 4: unexpected token: 10 @ line 4, column 15. version = 1.0.10

https://github.com/Netflix/exhibitor/wiki/Building-Exhibitor

Currently, there is no way to specify that a ZK instance is an Observer

Right now the instances can be restarted all at once or a rolling restart. One option I like is to just apply the config and let me nuke the servers manually in any order I want.

Hello,

There seems to be some regression as this worked fine on 1.4.2

1.4.4/1.4.3 work fine on a local VM, but once I try to run it on an ec2 instance with an IAM role it seems to break.

let me know if I can provide any other info, and thanks in advance!

1.4.2 output

$ ~/dev/exhibitor/exhibitor-standalone/build/resources/main/buildscripts/standalone/gradle$ java -jar ./build/libs/gradle-1.4.2.jar -cs3 --s3config :config --port 8080
v1.4.2
INFO com.netflix.exhibitor.core.activity.ActivityLog Exhibitor started [main]
INFO org.mortbay.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog [main]
INFO org.mortbay.log jetty-1.4.2 [main]
Jan 3, 2013 10:31:31 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.9.1 09/14/2011 02:36 PM'
INFO org.mortbay.log Started [email protected]:8080 [main]
Jan 3, 2013 10:31:42 PM java.util.prefs.FileSystemPreferences$7 run
WARNING: Prefs file removed in background /home/ubuntu/.java/.userPrefs/prefs.xml

1.4.4/1.4.3 output

$:~/dev/exhibitor/exhibitor-standalone/build/resources/main/buildscripts/standalone/gradle$ java -jar ./build/libs/gradle-1.4.4.jar -cs3 --s3config :config --port 8080
v1.4.4
INFO com.netflix.exhibitor.core.activity.ActivityLog Exhibitor started [main]
INFO com.amazonaws.http.AmazonHttpClient Unable to execute HTTP request: null [main]
org.apache.http.client.ClientProtocolException
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:822)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:278)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2906)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:885)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:775)
at com.netflix.exhibitor.core.s3.S3ClientImpl.getObject(S3ClientImpl.java:102)
at com.netflix.exhibitor.core.config.s3.S3ConfigProvider.getConfigObject(S3ConfigProvider.java:170)
at com.netflix.exhibitor.core.config.s3.S3ConfigProvider.loadConfig(S3ConfigProvider.java:99)
at com.netflix.exhibitor.core.config.ConfigManager.(ConfigManager.java:83)
at com.netflix.exhibitor.core.config.ConfigManager.(ConfigManager.java:57)
at com.netflix.exhibitor.core.Exhibitor.(Exhibitor.java:128)
at com.netflix.exhibitor.application.ExhibitorMain.(ExhibitorMain.java:95)
at com.netflix.exhibitor.application.ExhibitorMain.main(ExhibitorMain.java:68)
Caused by: org.apache.http.ProtocolException: Received redirect response HTTP/1.1 301 Moved Permanently but no location header
at org.apache.http.impl.client.DefaultRedirectStrategy.getLocationURI(DefaultRedirectStrategy.java:107)
at org.apache.http.impl.client.DefaultRedirectStrategy.getRedirect(DefaultRedirectStrategy.java:193)
at org.apache.http.impl.client.DefaultRequestDirector.handleResponse(DefaultRequestDirector.java:1022)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:482)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
... 15 more
INFO com.amazonaws.http.AmazonHttpClient Unable to execute HTTP request: null [main]
org.apache.http.client.ClientProtocolException
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:822)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:278)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2906)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:885)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:775)
at com.netflix.exhibitor.core.s3.S3ClientImpl.getObject(S3ClientImpl.java:102)
at com.netflix.exhibitor.core.config.s3.S3ConfigProvider.getConfigObject(S3ConfigProvider.java:170)
at com.netflix.exhibitor.core.config.s3.S3ConfigProvider.loadConfig(S3ConfigProvider.java:99)
at com.netflix.exhibitor.core.config.ConfigManager.(ConfigManager.java:83)
at com.netflix.exhibitor.core.config.ConfigManager.(ConfigManager.java:57)
at com.netflix.exhibitor.core.Exhibitor.(Exhibitor.java:128)
at com.netflix.exhibitor.application.ExhibitorMain.(ExhibitorMain.java:95)
at com.netflix.exhibitor.application.ExhibitorMain.main(ExhibitorMain.java:68)
Caused by: org.apache.http.ProtocolException: Received redirect response HTTP/1.1 301 Moved Permanently but no location header
at org.apache.http.impl.client.DefaultRedirectStrategy.getLocationURI(DefaultRedirectStrategy.java:107)
at org.apache.http.impl.client.DefaultRedirectStrategy.getRedirect(DefaultRedirectStrategy.java:193)
at org.apache.http.impl.client.DefaultRequestDirector.handleResponse(DefaultRequestDirector.java:1022)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:482)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
... 15 more
INFO com.amazonaws.http.AmazonHttpClient Unable to execute HTTP request: null [main]
org.apache.http.client.ClientProtocolException
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:822)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:278)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2906)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:885)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:775)
at com.netflix.exhibitor.core.s3.S3ClientImpl.getObject(S3ClientImpl.java:102)
at com.netflix.exhibitor.core.config.s3.S3ConfigProvider.getConfigObject(S3ConfigProvider.java:170)
at com.netflix.exhibitor.core.config.s3.S3ConfigProvider.loadConfig(S3ConfigProvider.java:99)
at com.netflix.exhibitor.core.config.ConfigManager.(ConfigManager.java:83)
at com.netflix.exhibitor.core.config.ConfigManager.(ConfigManager.java:57)
at com.netflix.exhibitor.core.Exhibitor.(Exhibitor.java:128)
at com.netflix.exhibitor.application.ExhibitorMain.(ExhibitorMain.java:95)
at com.netflix.exhibitor.application.ExhibitorMain.main(ExhibitorMain.java:68)
Caused by: org.apache.http.ProtocolException: Received redirect response HTTP/1.1 301 Moved Permanently but no location header
at org.apache.http.impl.client.DefaultRedirectStrategy.getLocationURI(DefaultRedirectStrategy.java:107)
at org.apache.http.impl.client.DefaultRedirectStrategy.getRedirect(DefaultRedirectStrategy.java:193)
at org.apache.http.impl.client.DefaultRequestDirector.handleResponse(DefaultRequestDirector.java:1022)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:482)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
... 15 more
INFO com.amazonaws.http.AmazonHttpClient Unable to execute HTTP request: null [main]
org.apache.http.client.ClientProtocolException
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:822)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:278)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2906)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:885)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:775)
at com.netflix.exhibitor.core.s3.S3ClientImpl.getObject(S3ClientImpl.java:102)
at com.netflix.exhibitor.core.config.s3.S3ConfigProvider.getConfigObject(S3ConfigProvider.java:170)
at com.netflix.exhibitor.core.config.s3.S3ConfigProvider.loadConfig(S3ConfigProvider.java:99)
at com.netflix.exhibitor.core.config.ConfigManager.(ConfigManager.java:83)
at com.netflix.exhibitor.core.config.ConfigManager.(ConfigManager.java:57)
at com.netflix.exhibitor.core.Exhibitor.(Exhibitor.java:128)
at com.netflix.exhibitor.application.ExhibitorMain.(ExhibitorMain.java:95)
at com.netflix.exhibitor.application.ExhibitorMain.main(ExhibitorMain.java:68)
Caused by: org.apache.http.ProtocolException: Received redirect response HTTP/1.1 301 Moved Permanently but no location header
at org.apache.http.impl.client.DefaultRedirectStrategy.getLocationURI(DefaultRedirectStrategy.java:107)
at org.apache.http.impl.client.DefaultRedirectStrategy.getRedirect(DefaultRedirectStrategy.java:193)
at org.apache.http.impl.client.DefaultRequestDirector.handleResponse(DefaultRequestDirector.java:1022)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:482)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
... 15 more
Exception in thread "main" com.amazonaws.AmazonClientException: Unable to execute HTTP request: null
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:326)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2906)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:885)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:775)
at com.netflix.exhibitor.core.s3.S3ClientImpl.getObject(S3ClientImpl.java:102)
at com.netflix.exhibitor.core.config.s3.S3ConfigProvider.getConfigObject(S3ConfigProvider.java:170)
at com.netflix.exhibitor.core.config.s3.S3ConfigProvider.loadConfig(S3ConfigProvider.java:99)
at com.netflix.exhibitor.core.config.ConfigManager.(ConfigManager.java:83)
at com.netflix.exhibitor.core.config.ConfigManager.(ConfigManager.java:57)
at com.netflix.exhibitor.core.Exhibitor.(Exhibitor.java:128)
at com.netflix.exhibitor.application.ExhibitorMain.(ExhibitorMain.java:95)
at com.netflix.exhibitor.application.ExhibitorMain.main(ExhibitorMain.java:68)
Caused by: org.apache.http.client.ClientProtocolException
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:822)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:278)
... 12 more
Caused by: org.apache.http.ProtocolException: Received redirect response HTTP/1.1 301 Moved Permanently but no location header
at org.apache.http.impl.client.DefaultRedirectStrategy.getLocationURI(DefaultRedirectStrategy.java:107)
at org.apache.http.impl.client.DefaultRedirectStrategy.getRedirect(DefaultRedirectStrategy.java:193)
at org.apache.http.impl.client.DefaultRequestDirector.handleResponse(DefaultRequestDirector.java:1022)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:482)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
... 15 more

Referring to #97

1. the --prefspath which has to be a file NEEDs to exist otherwise preferences won't be stored, in other words I think file should be created automatically if NONEXISTENT.
2. When zookeeper is running and I restart exhibitor the state is not recognized properly (stalled at latent) and I assume it should as exhibitor keeps asking zookeeper server "ruok" . Only committing changes to the config resets the state and only then I am able to see in the logs:

INFO com.netflix.exhibitor.core.activity.ActivityLog State: serving [ActivityQueue-0]

3. When I am able to successfully store preferences via --prefspath and auto-restarts are enabled, when zookeeper is stopped it never gets restarted. Again state is stalled at "serving". Only when pressing commit config changes it tries to restart zookeeper instance, it does not detect dead zookeeper.

OS: Ubuntu 12.04
java version "1.6.0_33"

how to reproduce: drag the lower right corner of the "data" textbox towards right.

Hi All,

I've created a few znodes on my ensamble, and can read and write to them just fine using other tools, but the exhibitor explorer api seems to be having issues.

This is what a typical request looks like for me

$ curl -v /exhibitor/v1/explorer/node-data?key=%2F&_=1356650330731"

GET /exhibitor/v1/explorer/node-data?key=%2F&_=1356650330731 HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
Accept: /

< HTTP/1.1 200 OK
< Content-Type: application/json
< Transfer-Encoding: chunked
< Server: Jetty(1.4.2)
<
{"bytes":"","str":"Exception","stat":"KeeperErrorCode = ConnectionLoss for /"}

This is how exhibitor is being run

java -jar /opt/exhibitor/exhibitor-1.4.3.jar -cs3 --s3config --s3backup true --configcheckms 30000 --jquerystyle black --loglines 1000 --nodemodification false --port 8080 --servo true --timeout 30000

let me know if there's any more information I can provide

Hello,
I'm trying to use an external log4j.properties file with the "-Dlog4j.configuration" java option.

Command :
java -Dlog4j.configuration=/opt/exhibitor/log4j.properties -jar /opt/exhibitor/exhibitor-war-1.0-jar-with-dependencies.jar -c file --fsconfigdir /mnt/storage/config --nodemodification true --jquerystyle black --filesystembackup /mnt/storage/backup/$(hostname -a)/

Result : log4j.properties not picked up from jar
log4j:WARN No appenders could be found for logger (com.netflix.exhibitor.core.activity.ActivityLog).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

Am I doing it wrong?
Thank you.

François

There appears to be some sort of refresh bug with the UI. If I look at one node and then look at another, the other node will not show the control panel UI. I have to do a full refresh (splat+shift+r in Firefox) to get it to show the full control panel UI.

• We want the ops folks to know whether a ZK instance went down. They do both ICMP and HTTP ping. For HTTP ping the response should be standard HTTP error if the instance is down. Right now I use the 4LTR word (http://host:8080/exhibitor/v1/cluster/4ltr/ruok), but that returns "" string if the host is down.
• REST API to report overall status of the cluster. Like, whether the ensemble is available or not.