nathanmarz / storm-deploy Goto Github PK

View Code? Open in Web Editor NEW

517.0 517.0 148.0 559 KB

One click deploy for Storm clusters on AWS

License: Other

Clojure 100.00%

storm-deploy's People

Stargazers

Watchers

Forkers

aficionado germc flashus thaingo eldondev yechw immars muqeet joskid rummble dhruv stwang anpi harrigan mknoszlig rodrigods will2357 barkbarkuk travis atbrox dmnpignaud epicplayground dandroid88 sonarme ryanmmmmm mailmahee philrene gworley3 girishrao saurabhmalakar geomic vbajaria guarly palletops risdenk teatips hantec beachheadstudio jared314 imclab web5design yinhongzhan oghaffar xixuebin helxsz gigfork saurabhdhupar netconstructor scraping-xx teos0009 strategist922 lorcan shareaholic konradr sprabina misterfitz shareablee hoshposh contextworks mohitnagrath oubiwann andreaturli 64bitchris redhd w2ogroup pengkui mfelker empia ksumit nvdnkpr bridiver pallet chocolate1949 rundsp jwoolf 422053362 westover yiwang sathishpc gomesnayagam alex-learn yaneeve chinnurtb karthikkrk yingcmu sunnycmf jethrolam phosphene emidln harshach ngocthanhit toutiao wzb56 swinghu wattsinabox acinader max-dev priyankabiban91 ashokkumar-dhanavel c512319

storm-deploy's Issues

zookeeper 3.3.3 is no longer available

lein run :deploy --start will fail because apache project removes old versions of zookeeper from their server.

http://www.apache.org/dist/zookeeper/zookeeper-3.3.3/zookeeper-3.3.3.tar.gz

The version number (3.3.3) is hard-coded in the following:

https://github.com/nathanmarz/storm-deploy/blob/master/src/clj/backtype/storm/node.clj#L76
https://github.com/nathanmarz/storm-deploy/blob/master/src/clj/backtype/storm/crate/zookeeper.clj#L49

when I try to deploy my storm topology I get an error message, Could not find or load main class SampleTopology

I'm running the sample topology with no changes to it from

https://github.com/awslabs/kinesis-storm-spout/tree/master/src/main/samples

import java.io.FileInputStream;
import java.io.IOException;
import java.util.Properties;

import org.apache.zookeeper.KeeperException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import backtype.storm.Config;
import backtype.storm.LocalCluster;
import backtype.storm.StormSubmitter;
import backtype.storm.generated.AlreadyAliveException;
import backtype.storm.generated.InvalidTopologyException;
import backtype.storm.topology.TopologyBuilder;
import backtype.storm.tuple.Fields;

import com.amazonaws.ClientConfiguration;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.kinesis.stormspout.InitialPositionInStream;
import com.amazonaws.services.kinesis.stormspout.KinesisSpout;
import com.amazonaws.services.kinesis.stormspout.KinesisSpoutConfig;

public class SampleTopology {
private static final Logger LOG = LoggerFactory.getLogger(SampleTopology.class);
private static String topologyName = "SampleTopology";
private static String streamName;
private static InitialPositionInStream initialPositionInStream = InitialPositionInStream.LATEST;
private static int recordRetryLimit = 3;
private static Regions region = Regions.US_EAST_1;
private static String zookeeperEndpoint;
private static String zookeeperPrefix;

public static void main(String[] args) throws IllegalArgumentException, KeeperException, InterruptedException, AlreadyAliveException, InvalidTopologyException, IOException {
    String propertiesFile = null;
    String mode = null;

    if (args.length != 2) {
        printUsageAndExit();
    } else {
        propertiesFile = args[0];
        mode = args[1];
    }

    configure(propertiesFile);

    final KinesisSpoutConfig config =
            new KinesisSpoutConfig(streamName, zookeeperEndpoint).withZookeeperPrefix(zookeeperPrefix)
                    .withKinesisRecordScheme(new SampleKinesisRecordScheme())
                    .withInitialPositionInStream(initialPositionInStream)
                    .withRecordRetryLimit(recordRetryLimit)
                    .withRegion(region);

    final KinesisSpout spout = new KinesisSpout(config, new CustomCredentialsProviderChain(), new ClientConfiguration());
    TopologyBuilder builder = new TopologyBuilder();
    LOG.info("Using Kinesis stream: " + config.getStreamName());

    // Using number of shards as the parallelism hint for the spout.
    builder.setSpout("kinesis_spout", spout, 2);
    builder.setBolt("print_bolt", new SampleBolt(), 2).fieldsGrouping("kinesis_spout", new Fields(SampleKinesisRecordScheme.FIELD_PARTITION_KEY));

    Config topoConf = new Config();
    topoConf.setFallBackOnJavaSerialization(true);
    topoConf.setDebug(false);

    if (mode.equals("LocalMode")) {
        LOG.info("Starting sample storm topology in LocalMode ...");
        new LocalCluster().submitTopology("test_spout", topoConf, builder.createTopology());
    } else if (mode.equals("RemoteMode")) {
        topoConf.setNumWorkers(1);
        topoConf.setMaxSpoutPending(5000);
        LOG.info("Submitting sample topology " + topologyName + " to remote cluster.");
        StormSubmitter.submitTopology(topologyName, topoConf, builder.createTopology());            
    } else {
        printUsageAndExit();
    }

}

private static void configure(String propertiesFile) throws IOException {
    FileInputStream inputStream = new FileInputStream(propertiesFile);
    Properties properties = new Properties();
    try {
        properties.load(inputStream);
    } finally {
        inputStream.close();
    }

    String topologyNameOverride = properties.getProperty(ConfigKeys.TOPOLOGY_NAME_KEY);
    if (topologyNameOverride != null) {
        topologyName = topologyNameOverride;
    }
    LOG.info("Using topology name " + topologyName);

    String streamNameOverride = properties.getProperty(ConfigKeys.STREAM_NAME_KEY);
    if (streamNameOverride != null) {
        streamName = streamNameOverride;
    }
    LOG.info("Using stream name " + streamName);

    String initialPositionOverride = properties.getProperty(ConfigKeys.INITIAL_POSITION_IN_STREAM_KEY);
    if (initialPositionOverride != null) {
         initialPositionInStream = InitialPositionInStream.valueOf(initialPositionOverride);
    }
    LOG.info("Using initial position " + initialPositionInStream.toString() + " (if a checkpoint is not found).");

    String recordRetryLimitOverride = properties.getProperty(ConfigKeys.RECORD_RETRY_LIMIT);
    if (recordRetryLimitOverride != null) {
        recordRetryLimit = Integer.parseInt(recordRetryLimitOverride.trim());
    }
    LOG.info("Using recordRetryLimit " + recordRetryLimit);

    String regionOverride = properties.getProperty(ConfigKeys.REGION_KEY);
    if (regionOverride != null) {
        region = Regions.fromName(regionOverride);
    }
    LOG.info("Using region " + region.getName());

    String zookeeperEndpointOverride = properties.getProperty(ConfigKeys.ZOOKEEPER_ENDPOINT_KEY);
    if (zookeeperEndpointOverride != null) {
        zookeeperEndpoint = zookeeperEndpointOverride;
    }
    LOG.info("Using zookeeper endpoint " + zookeeperEndpoint);

    String zookeeperPrefixOverride = properties.getProperty(ConfigKeys.ZOOKEEPER_PREFIX_KEY);
    if (zookeeperPrefixOverride != null) {            
        zookeeperPrefix = zookeeperPrefixOverride;
    }
    LOG.info("Using zookeeper prefix " + zookeeperPrefix);

}

private static void printUsageAndExit() {
    System.out.println("Usage: " + SampleTopology.class.getName() + " <propertiesFile> <LocalMode or RemoteMode>");
    System.exit(-1);
}

}

Errors authorizing security groups from supervisor/zookeeper to nimbus group (InvalidPlacementGroup.Unknown)

The API Request to AWS requesting to add the supervisor/zookeeper groups to the nimbus security group fails consistently with an "InvalidPlacementGroup.Unknown" error, due to the fact that the API AuthorizeSecurityGroupIngress doesn't like telling a SourceSecurityGroupOwnerId when it's yourself:

2013-02-08 19:22:16,814 DEBUG [org.jclouds.http.internal.JavaUrlHttpCommandExecutorService] (i/o thread 7) Sending request -1262905079: POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1
2013-02-08 19:22:16,814 DEBUG [jclouds.wire] (i/o thread 7) >> "Action=AuthorizeSecurityGroupIngress&GroupName=jclouds%23nimbus-amplcluster%23us-east-1&Signature=8S5KsndBbwFYY8Asnd7x%2FLv1nmfBH%2B7YF63lQydRM4M%3D&SignatureMethod
=HmacSHA256&SignatureVersion=2&SourceSecurityGroupName=jclouds%23supervisor-amplcluster%23us-east-1&SourceSecurityGroupOwnerId=e26cd44119560f8e1e272d9bbc0928224c2414fde521bd81e4621789f9283ea6&Timestamp=2013-02-08T18%3A22%3A16.81
0Z&Version=2011-05-15&AWSAccessKeyId=AKIANNNNNNNNNNNNNNNNN"
2013-02-08 19:22:16,814 DEBUG [jclouds.headers] (i/o thread 7) >> POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1
2013-02-08 19:22:16,814 DEBUG [jclouds.headers] (i/o thread 7) >> Host: ec2.us-east-1.amazonaws.com
2013-02-08 19:22:16,814 DEBUG [jclouds.headers] (i/o thread 7) >> Content-Type: application/x-www-form-urlencoded
2013-02-08 19:22:16,815 DEBUG [jclouds.headers] (i/o thread 7) >> Content-Length: 449
2013-02-08 19:22:17,157 DEBUG [org.jclouds.http.internal.JavaUrlHttpCommandExecutorService] (i/o thread 7) Receiving response -1262905079: HTTP/1.1 400 Bad Request
2013-02-08 19:22:17,157 DEBUG [jclouds.headers] (i/o thread 7) << HTTP/1.1 400 Bad Request
2013-02-08 19:22:17,158 DEBUG [jclouds.headers] (i/o thread 7) << Transfer-Encoding: chunked
2013-02-08 19:22:17,158 DEBUG [jclouds.headers] (i/o thread 7) << Date: Fri, 08 Feb 2013 18:21:45 GMT
2013-02-08 19:22:17,158 DEBUG [jclouds.headers] (i/o thread 7) << Cneonction: close
2013-02-08 19:22:17,158 DEBUG [jclouds.headers] (i/o thread 7) << Server: AmazonEC2
2013-02-08 19:22:17,158 DEBUG [jclouds.headers] (i/o thread 7) << Content-Type: application/unknown
2013-02-08 19:22:17,158 DEBUG [jclouds.wire] (i/o thread 7) << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
2013-02-08 19:22:17,158 DEBUG [jclouds.wire] (i/o thread 7) << "<Response><Errors><Error><Code>InvalidGroup.NotFound</Code><Message>Unable to find group 'jclouds#supervisor-amplcluster#us-east-1'</Message></Error></Errors><Reque
stID>31001939-f631-402a-a040-7ce66a25f41a</RequestID></Response>"
2013-02-08 19:22:17,177 ERROR [java.logging] (main) Exception in thread "main"
2013-02-08 19:22:17,177 ERROR [java.logging] (main) org.jclouds.rest.ResourceNotFoundException: Unable to find group 'jclouds#supervisor-amplcluster#us-east-1' (NO_SOURCE_FILE:1)
2013-02-08 19:22:17,177 ERROR [java.logging] (main) at clojure.lang.Compiler.eval(Compiler.java:5440)
2013-02-08 19:22:17,177 ERROR [java.logging] (main) at clojure.lang.Compiler.eval(Compiler.java:5415)
2013-02-08 19:22:17,177 ERROR [java.logging] (main) at clojure.lang.Compiler.eval(Compiler.java:5391)
2013-02-08 19:22:17,177 ERROR [java.logging] (main) at clojure.core$eval.invoke(core.clj:2382)
2013-02-08 19:22:17,178 ERROR [java.logging] (main) at clojure.main$eval_opt.invoke(main.clj:235)
2013-02-08 19:22:17,178 ERROR [java.logging] (main) at clojure.main$initialize.invoke(main.clj:254)
2013-02-08 19:22:17,178 ERROR [java.logging] (main) at clojure.main$null_opt.invoke(main.clj:279)
2013-02-08 19:22:17,178 ERROR [java.logging] (main) at clojure.main$main.doInvoke(main.clj:354)
2013-02-08 19:22:17,178 ERROR [java.logging] (main) at clojure.lang.RestFn.invoke(RestFn.java:422)
2013-02-08 19:22:17,178 ERROR [java.logging] (main) at clojure.lang.Var.invoke(Var.java:369)
2013-02-08 19:22:17,178 ERROR [java.logging] (main) at clojure.lang.AFn.applyToHelper(AFn.java:165)
2013-02-08 19:22:17,178 ERROR [java.logging] (main) at clojure.lang.Var.applyTo(Var.java:482)
2013-02-08 19:22:17,179 ERROR [java.logging] (main) at clojure.main.main(main.java:37)
2013-02-08 19:22:17,179 ERROR [java.logging] (main) Caused by: org.jclouds.rest.ResourceNotFoundException: Unable to find group 'jclouds#supervisor-amplcluster#us-east-1'
2013-02-08 19:22:17,179 ERROR [java.logging] (main) at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:109)
2013-02-08 19:22:17,179 ERROR [java.logging] (main) at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:92)
2013-02-08 19:22:17,179 ERROR [java.logging] (main) at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:69)
2013-02-08 19:22:17,179 ERROR [java.logging] (main) at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.shouldContinue(BaseHttpCommandExecutorService.java:191)
2013-02-08 19:22:17,179 ERROR [java.logging] (main) at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:161)
2013-02-08 19:22:17,180 ERROR [java.logging] (main) at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:130)
2013-02-08 19:22:17,180 ERROR [java.logging] (main) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
2013-02-08 19:22:17,180 ERROR [java.logging] (main) at java.util.concurrent.FutureTask.run(FutureTask.java:166)
2013-02-08 19:22:17,180 ERROR [java.logging] (main) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
2013-02-08 19:22:17,180 ERROR [java.logging] (main) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
2013-02-08 19:22:17,180 ERROR [java.logging] (main) at java.lang.Thread.run(Thread.java:679)
2013-02-08 19:22:17,180 ERROR [java.logging] (main) at org.jclouds.concurrent.config.ExecutorServiceModule$DescribingExecutorService.submit(ExecutorServiceModule.java:184)
2013-02-08 19:22:17,180 ERROR [java.logging] (main) at org.jclouds.http.internal.BaseHttpCommandExecutorService.submit(BaseHttpCommandExecutorService.java:127)
2013-02-08 19:22:17,181 ERROR [java.logging] (main) at org.jclouds.http.TransformingHttpCommandExecutorServiceImpl.submit(TransformingHttpCommandExecutorServiceImpl.java:54)
2013-02-08 19:22:17,181 ERROR [java.logging] (main) at org.jclouds.http.TransformingHttpCommandImpl.execute(TransformingHttpCommandImpl.java:73)
2013-02-08 19:22:17,181 ERROR [java.logging] (main) at org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFutureForHttpRequestMappedToMethodAndArgs(AsyncRestClientProxy.java:198)
2013-02-08 19:22:17,181 ERROR [java.logging] (main) at org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:134)
2013-02-08 19:22:17,181 ERROR [java.logging] (main) at $Proxy74.authorizeSecurityGroupIngressInRegion(Unknown Source)
2013-02-08 19:22:17,181 ERROR [java.logging] (main) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2013-02-08 19:22:17,181 ERROR [java.logging] (main) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
2013-02-08 19:22:17,182 ERROR [java.logging] (main) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2013-02-08 19:22:17,182 ERROR [java.logging] (main) at java.lang.reflect.Method.invoke(Method.java:616)
2013-02-08 19:22:17,182 ERROR [java.logging] (main) at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:137)
2013-02-08 19:22:17,182 ERROR [java.logging] (main) at $Proxy75.authorizeSecurityGroupIngressInRegion(Unknown Source)
2013-02-08 19:22:17,182 ERROR [java.logging] (main) at backtype.storm.security$authorize_group.invoke(security.clj:116)
2013-02-08 19:22:17,182 ERROR [java.logging] (main) at backtype.storm.security$authorize_group.invoke(security.clj:111)
2013-02-08 19:22:17,182 ERROR [java.logging] (main) at backtype.storm.provision$start_with_nodes_BANG_.invoke(provision.clj:81)
2013-02-08 19:22:17,182 ERROR [java.logging] (main) at backtype.storm.provision$start_BANG_.invoke(provision.clj:95)
2013-02-08 19:22:17,183 ERROR [java.logging] (main) at backtype.storm.provision$_main.doInvoke(provision.clj:147)
2013-02-08 19:22:17,183 ERROR [java.logging] (main) at clojure.lang.RestFn.invoke(RestFn.java:483)
2013-02-08 19:22:17,183 ERROR [java.logging] (main) at clojure.lang.Var.invoke(Var.java:381)
2013-02-08 19:22:17,183 ERROR [java.logging] (main) at user$eval8251.invoke(NO_SOURCE_FILE:1)
2013-02-08 19:22:17,183 ERROR [java.logging] (main) at clojure.lang.Compiler.eval(Compiler.java:5424)
2013-02-08 19:22:17,183 ERROR [java.logging] (main) at clojure.lang.Compiler.eval(Compiler.java:5415)
2013-02-08 19:22:17,183 ERROR [java.logging] (main) at clojure.lang.Compiler.eval(Compiler.java:5391)
2013-02-08 19:22:17,183 ERROR [java.logging] (main) at clojure.core$eval.invoke(core.clj:2382)
2013-02-08 19:22:17,184 ERROR [java.logging] (main) at clojure.main$eval_opt.invoke(main.clj:235)
2013-02-08 19:22:17,184 ERROR [java.logging] (main) at clojure.main$initialize.invoke(main.clj:254)
2013-02-08 19:22:17,184 ERROR [java.logging] (main) at clojure.main$null_opt.invoke(main.clj:279)
2013-02-08 19:22:17,184 ERROR [java.logging] (main) at clojure.main$main.doInvoke(main.clj:354)
2013-02-08 19:22:17,184 ERROR [java.logging] (main) at clojure.lang.RestFn.invoke(RestFn.java:422)
2013-02-08 19:22:17,184 ERROR [java.logging] (main) at clojure.lang.Var.invoke(Var.java:369)
2013-02-08 19:22:17,184 ERROR [java.logging] (main) at clojure.lang.AFn.applyToHelper(AFn.java:165)
2013-02-08 19:22:17,184 ERROR [java.logging] (main) at clojure.lang.Var.applyTo(Var.java:482)
2013-02-08 19:22:17,185 ERROR [java.logging] (main) at clojure.main.main(main.java:37)
2013-02-08 19:22:17,185 ERROR [java.logging] (main) at org.jclouds.concurrent.config.ExecutorServiceModule$DescribingExecutorService.submit(ExecutorServiceModule.java:184)
2013-02-08 19:22:17,185 ERROR [java.logging] (main) at org.jclouds.http.internal.BaseHttpCommandExecutorService.submit(BaseHttpCommandExecutorService.java:127)
2013-02-08 19:22:17,185 ERROR [java.logging] (main) at org.jclouds.http.TransformingHttpCommandExecutorServiceImpl.submit(TransformingHttpCommandExecutorServiceImpl.java:54)
2013-02-08 19:22:17,185 ERROR [java.logging] (main) at org.jclouds.http.TransformingHttpCommandImpl.execute(TransformingHttpCommandImpl.java:73)
2013-02-08 19:22:17,185 ERROR [java.logging] (main) at org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFutureForHttpRequestMappedToMethodAndArgs(AsyncRestClientProxy.java:198)
2013-02-08 19:22:17,185 ERROR [java.logging] (main) at org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:134)
2013-02-08 19:22:17,185 ERROR [java.logging] (main) at $Proxy74.authorizeSecurityGroupIngressInRegion(Unknown Source)
2013-02-08 19:22:17,186 ERROR [java.logging] (main) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2013-02-08 19:22:17,186 ERROR [java.logging] (main) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
2013-02-08 19:22:17,186 ERROR [java.logging] (main) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2013-02-08 19:22:17,186 ERROR [java.logging] (main) at java.lang.reflect.Method.invoke(Method.java:616)
2013-02-08 19:22:17,186 ERROR [java.logging] (main) at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:137)
2013-02-08 19:22:17,186 ERROR [java.logging] (main) at $Proxy75.authorizeSecurityGroupIngressInRegion(Unknown Source)
2013-02-08 19:22:17,186 ERROR [java.logging] (main) at backtype.storm.security$authorize_group.invoke(security.clj:116)
2013-02-08 19:22:17,187 ERROR [java.logging] (main) at backtype.storm.security$authorize_group.invoke(security.clj:111)
2013-02-08 19:22:17,187 ERROR [java.logging] (main) at backtype.storm.provision$start_with_nodes_BANG_.invoke(provision.clj:81)
2013-02-08 19:22:17,187 ERROR [java.logging] (main) at backtype.storm.provision$start_BANG_.invoke(provision.clj:95)
2013-02-08 19:22:17,187 ERROR [java.logging] (main) at backtype.storm.provision$_main.doInvoke(provision.clj:147)
2013-02-08 19:22:17,187 ERROR [java.logging] (main) at clojure.lang.RestFn.invoke(RestFn.java:483)
2013-02-08 19:22:17,187 ERROR [java.logging] (main) at clojure.lang.Var.invoke(Var.java:381)
2013-02-08 19:22:17,187 ERROR [java.logging] (main) at user$eval8251.invoke(NO_SOURCE_FILE:1)
2013-02-08 19:22:17,188 ERROR [java.logging] (main) at clojure.lang.Compiler.eval(Compiler.java:5424)
2013-02-08 19:22:17,188 ERROR [java.logging] (main) ... 12 more
2013-02-08 19:22:17,188 ERROR [java.logging] (main) Caused by: org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='31001939-f631-402a-a040-7ce66a25f41a', requestToken='null', code='InvalidGroup.NotFound', message='Unable to find group 'jclouds#supervisor-amplcluster#us-east-1'', context='{Response=, Errors=}'}
2013-02-08 19:22:17,189 ERROR [java.logging] (main) at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:78)
2013-02-08 19:22:17,189 ERROR [java.logging] (main) at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:69)
2013-02-08 19:22:17,189 ERROR [java.logging] (main) at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.shouldContinue(BaseHttpCommandExecutorService.java:191)
2013-02-08 19:22:17,189 ERROR [java.logging] (main) at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:161)
2013-02-08 19:22:17,189 ERROR [java.logging] (main) at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:130)
2013-02-08 19:22:17,189 ERROR [java.logging] (main) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
2013-02-08 19:22:17,189 ERROR [java.logging] (main) at java.util.concurrent.FutureTask.run(FutureTask.java:166)
2013-02-08 19:22:17,189 ERROR [java.logging] (main) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
2013-02-08 19:22:17,190 ERROR [java.logging] (main) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
2013-02-08 19:22:17,190 ERROR [java.logging] (main) at java.lang.Thread.run(Thread.java:679)

Deploy error on signer information not matching

First I had issues with #62, so I tried using the update-clojure-jclouds branch, as suggested. That resolved the issue with #62, but now I'm getting stuck at the following step:

...
ERROR compute - << problem customizing node(us-west-2/i-531b885e):
org.jclouds.ssh.SshException:
(ubuntu:rsa[fingerprint(<... cut for brevity...> )])
error acquiring {hostAndPort=<cut>, loginUser=ubuntu, ssh=null, connectTimeout=60000,
sessionTimeout=60000} (not retryable): class "org.bouncycastle.crypto.prng.VMPCRandomGenerator"'s signer
information does not match signer information of other classes in the same package
        at org.jclouds.sshj.SshjSshClient.propagate(SshjSshClient.java:381)
        at org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:211)
        at org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:221)
        ...

It seems as though maybe I don't have my rsa keys set up properly, or some such. But I can't figure out what would actually be wrong there (I did create a key pair, point to the correct locations in ~/.pallet/config.clj and loaded the public key onto AWS. Is there anything I'm missing? Or is this perhaps a bug?

"com.jcraft.jsch.JSchException: fromBase64: invalid base64 data" while trying to deploy in EC2

I tried this with the latest commits of the 3 branches of the project. All give me the same problems.

I am trying to deploy storm cluster in aws. The 4 virtual machines are created in my amazon account. However, when the message "com.jcraft.jsch.JSchException: fromBase64: invalid base64 data" apperas in the logs the execution stops and the 4 created virtual machines remain running in ec2. These machines have not storm installed.

I am running it OS X: 10.9.2 and java -version output looks like:

java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)

My clusters.yaml looks like this

################################################################################
# CLUSTERS CONFIG FILE
################################################################################

nimbus.image: "us-east-1/ami-d726abbe"         #64-bit ubuntu
nimbus.hardware: "m1.large"

supervisor.count: 2
supervisor.image: "us-east-1/ami-d726abbe"         #64-bit ubuntu on eu-east-1
supervisor.hardware: "m1.large"
#supervisor.spot.price: 1.60


zookeeper.count: 1
zookeeper.image: "us-east-1/ami-d726abbe"         #64-bit ubuntu
zookeeper.hardware: "m1.large"

I am launching my cluster using the following command

lein deploy-storm --start --name clusterhodei --branch master --commit 0.9.0-rc2

The first line of the output looks like this:

DEBUG jclouds - Failed to find credentials annotation, using credentials supplier

It doesn't seem affected and the execution goes in its way. After a while these messages are logged:

DEBUG core - p-f-s server environment null
DEBUG core - p-f-s server environment null
DEBUG core - p-f-s server environment null
DEBUG core - p-f-s server environment null
DEBUG core - p-f-s server environment null
DEBUG core - p-f-s server environment null
DEBUG core - p-f-s server environment null
DEBUG core - p-f-s server environment null
INFO  core - parallel-apply-phase :pallet.phase/pre-configure for :zookeeper-mycluster
INFO  core - parallel-apply-phase-to-target :node  :pallet.phase/pre-configure for :zookeeper-mycluster with 1 nodes
INFO  core - parallel-apply-phase :pallet.phase/pre-configure for :supervisor-mycluster
INFO  core - parallel-apply-phase-to-target :node  :pallet.phase/pre-configure for :supervisor-mycluster with 2 nodes
INFO  core - parallel-apply-phase :pallet.phase/pre-configure for :nimbus-mycluster
INFO  core - parallel-apply-phase-to-target :node  :pallet.phase/pre-configure for :nimbus-mycluster with 1 nodes
DEBUG core - apply-phase-to-node: phase :pallet.phase/pre-configure group :zookeeper-mycluster target 174.129.175.130
DEBUG core - apply-phase-to-node: phase :pallet.phase/pre-configure group :supervisor-mycluster target 54.205.42.201
DEBUG core - apply-phase-to-node: phase :pallet.phase/pre-configure group :supervisor-mycluster target 54.163.157.22
DEBUG core - apply-phase-to-node: phase :pallet.phase/pre-configure group :nimbus-mycluster target 54.167.202.34
INFO  execute - execute-with-ssh on supervisor-mycluster "54.163.157.22"
INFO  execute - execute-with-ssh on nimbus-mycluster "54.167.202.34"
INFO  execute - execute-with-ssh on supervisor-mycluster "54.205.42.201"
INFO  execute - execute-with-ssh on zookeeper-mycluster "174.129.175.130"
ERROR logging - Exception in thread "main"
ERROR logging - java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.jcraft.jsch.JSchException: fromBase64: invalid base64 data (form-init8016119195565937948.clj:1)
ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5440)
ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5415)
ERROR logging - at clojure.lang.Compiler.load(Compiler.java:5857)
ERROR logging - at clojure.lang.Compiler.loadFile(Compiler.java:5820)
ERROR logging - at clojure.main$load_script.invoke(main.clj:221)
ERROR logging - at clojure.main$init_opt.invoke(main.clj:226)
ERROR logging - at clojure.main$initialize.invoke(main.clj:254)
ERROR logging - at clojure.main$null_opt.invoke(main.clj:279)
ERROR logging - at clojure.main$main.doInvoke(main.clj:354)
ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:422)
ERROR logging - at clojure.lang.Var.invoke(Var.java:369)
ERROR logging - at clojure.lang.AFn.applyToHelper(AFn.java:165)
ERROR logging - at clojure.lang.Var.applyTo(Var.java:482)
ERROR logging - at clojure.main.main(main.java:37)
ERROR logging - Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.jcraft.jsch.JSchException: fromBase64: invalid base64 data
ERROR logging - at clojure.lang.LazySeq.sval(LazySeq.java:47)
ERROR logging - at clojure.lang.LazySeq.seq(LazySeq.java:56)
ERROR logging - at clojure.lang.RT.seq(RT.java:450)
ERROR logging - at clojure.core$seq.invoke(core.clj:122)
ERROR logging - at clojure.core$dorun.invoke(core.clj:2450)
ERROR logging - at clojure.core$doall.invoke(core.clj:2465)
ERROR logging - at pallet.core$parallel_lift.invoke(core.clj:757)
ERROR logging - at pallet.core$lift_phase$fn__5442.invoke(core.clj:779)
ERROR logging - at clojure.lang.ArrayChunk.reduce(ArrayChunk.java:58)
ERROR logging - at clojure.core$r.invoke(core.clj:797)
ERROR logging - at pallet.core$lift_phase.invoke(core.clj:781)
ERROR logging - at pallet.core$lift_nodes$fn__5445.invoke(core.clj:794)
ERROR logging - at clojure.lang.ArrayChunk.reduce(ArrayChunk.java:58)
ERROR logging - at clojure.core$r.invoke(core.clj:797)
ERROR logging - at pallet.core$lift_nodes.invoke(core.clj:798)
ERROR logging - at pallet.core$lift_STAR_.invoke(core.clj:1310)
ERROR logging - at pallet.core$converge_STAR_.invoke(core.clj:1335)
ERROR logging - at pallet.core$converge.doInvoke(core.clj:1506)
ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:440)
ERROR logging - at backtype.storm.provision$start_with_nodes_BANG_.invoke(provision.clj:69)
ERROR logging - at backtype.storm.provision$start_BANG_.invoke(provision.clj:91)
ERROR logging - at backtype.storm.provision$_main$fn__8907.invoke(provision.clj:144)
ERROR logging - at backtype.storm.provision$_main.doInvoke(provision.clj:130)
ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:552)
ERROR logging - at clojure.lang.Var.invoke(Var.java:390)
ERROR logging - at user$eval5.invoke(form-init8016119195565937948.clj:1)
ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5424)
ERROR logging - ... 13 more
ERROR logging - Caused by: java.util.concurrent.ExecutionException: com.jcraft.jsch.JSchException: fromBase64: invalid base64 data
ERROR logging - at java.util.concurrent.FutureTask.report(FutureTask.java:122)
ERROR logging - at java.util.concurrent.FutureTask.get(FutureTask.java:188)
ERROR logging - at clojure.core$future_call$reify__5500.deref(core.clj:5399)
ERROR logging - at clojure.core$deref.invoke(core.clj:1765)
ERROR logging - at clojure.core$map$fn__3695.invoke(core.clj:2096)
ERROR logging - at clojure.lang.LazySeq.sval(LazySeq.java:42)
ERROR logging - ... 39 more
ERROR logging - Caused by: com.jcraft.jsch.JSchException: fromBase64: invalid base64 data
ERROR logging - at com.jcraft.jsch.Util.fromBase64(Util.java:63)
ERROR logging - at com.jcraft.jsch.KnownHosts.setKnownHosts(KnownHosts.java:222)
ERROR logging - at com.jcraft.jsch.KnownHosts.setKnownHosts(KnownHosts.java:60)
ERROR logging - at com.jcraft.jsch.JSch.setKnownHosts(JSch.java:303)
ERROR logging - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
ERROR logging - at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
ERROR logging - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
ERROR logging - at java.lang.reflect.Method.invoke(Method.java:606)
ERROR logging - at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:90)
ERROR logging - at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
ERROR logging - at clj_ssh.ssh$ssh_agent.invoke(ssh.clj:103)
ERROR logging - at pallet.execute$default_agent$fn__4689.invoke(execute.clj:217)
ERROR logging - at clojure.lang.Atom.swap(Atom.java:37)
ERROR logging - at clojure.core$swap_BANG_.invoke(core.clj:1790)
ERROR logging - at pallet.execute$default_agent.invoke(execute.clj:213)
ERROR logging - at pallet.execute$execute_with_ssh$execute_with_ssh_fn__4858.invoke(execute.clj:604)
ERROR logging - at pallet.core$raise_on_error$fn__5216.invoke(core.clj:519)
ERROR logging - at pallet.core$middleware_handler$fn__5209.invoke(core.clj:495)
ERROR logging - at pallet.core$apply_phase_to_node.invoke(core.clj:656)
ERROR logging - at pallet.core$eval5379$fn__5380$iter__5381__5385$fn__5386$fn__5391.invoke(core.clj:721)
ERROR logging - at clojure.lang.AFn.call(AFn.java:18)
ERROR logging - at java.util.concurrent.FutureTask.run(FutureTask.java:262)
ERROR logging - at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
ERROR logging - at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
ERROR logging - at java.lang.Thread.run(Thread.java:744)
ERROR logging - Caused by: java.lang.ArrayIndexOutOfBoundsException: 437
ERROR logging - at com.jcraft.jsch.Util.fromBase64(Util.java:51)
ERROR logging - ... 24 more

Unable to sudo

The VMs generated by when running storm-deploy are not allowing for "sudo" access but have no problems with this when creating one manually. "sudo" is required to check the logs as I can access Ganglia on port 80 but not the Storm UI (8080) and all daemon process appear to be running.

Can't sudo on instances as user ubuntu

There may be something I am missing here, but after launching the cluster, and ssh'ing to the nimbus, I am prompted for a password when trying to sudo.

Certain deployment configurations not booting Nimbus properly.

Details of cause are sketchy but it's definitely a problem, going to try and troubleshoot / get a fix out today. May also be the cause of issue #21.

You can spot the problem by attempting to submit a jar to the cluster, you'll get a connection refused error as a result of Nimbus not allowing the connection.

Also attempting to the attach to the cluster results in AssertionErrors for running nodes.

AWS region seems to be ignored when creating instances

I tried setting the "jclouds.regions" to "us-west-1" in ~/.pallet/config.clj and when I ran the deploy start cluster command instances were started in us-east-1. Not sure if this is a bug in storm-deploy, jclouds, or pallet...

Error in deployment on Amazon EC2 using image based on ubuntu-trusty-14.04-amd64-server

I had tried to deploy a cluster using the defaults i.e., 1 Nimbus, 1 Zookeeper and 2 Supervisor nodes.

The instances were started successfully (I could see from Amazon console) but there was some error w.r.t building storm.

Then I tried to stop the cluster using the command "lein deploy-storm --stop --name ..." and what I found is that it could only stop 3 instances out of the 4 launched. It failed to stop one supervisor instance.

The error that was displayed on the console is:

/bin/mkdir -p '/usr/local/share/java'\n /usr/bin/install -c -m 644 zmq.jar '/usr/local/share/java'\nmake[3]: Leaving directory /opt/local/zeromq/rfLHkBB/jzmq/src'\nmake[2]: Leaving directory/opt/local/zeromq/rfLHkBB/jzmq/src'\nmake[1]: Leaving directory /opt/local/zeromq/rfLHkBB/jzmq/src'\nMaking install in perf\nmake[1]: Entering directory/opt/local/zeromq/rfLHkBB/jzmq/perf'\nmake install-am\nmake[2]: Entering directory /opt/local/zeromq/rfLHkBB/jzmq/perf'\nmake[3]: Entering directory/opt/local/zeromq/rfLHkBB/jzmq/perf'\nmake[3]: Nothing to be done for install-exec-am'.\nmake[3]: Nothing to be done forinstall-data-am'.\nmake[3]: Leaving directory /opt/local/zeromq/rfLHkBB/jzmq/perf'\nmake[2]: Leaving directory/opt/local/zeromq/rfLHkBB/jzmq/perf'\nmake[1]: Leaving directory /opt/local/zeromq/rfLHkBB/jzmq/perf'\nmake[1]: Entering directory/opt/local/zeromq/rfLHkBB/jzmq'\nmake[2]: Entering directory /opt/local/zeromq/rfLHkBB/jzmq'\nmake[2]: Nothing to be done forinstall-exec-am'.\nmake[2]: Nothing to be done for install-data-am'.\nmake[2]: Leaving directory/opt/local/zeromq/rfLHkBB/jzmq'\nmake[1]: Leaving directory `/opt/local/zeromq/rfLHkBB/jzmq'\n...done\nDirectory /mnt/storm...\n...done\nclean up home...\n...done\nBuild storm...\nAlready up-to-date.\nbash: bin/build_release.sh: No such file or directory\nBuild storm failed\nlogout\n", :server "54.67.52.187"}]
ERROR core - errors found [{:message "Unexpected exception: throw+: {:type :pallet/ssh-connection-failure, :message "ssh-fail: server 54.183.168.178, port 22, user storm, group :zookeeper-albeado-storm-cluster", :cause #}", :type :pallet/action-execution-error, :cause #<ExceptionInfo slingshot.ExceptionInfo: throw+: {:type :pallet/ssh-connection-failure, :message "ssh-fail: server 54.183.168.178, port 22, user storm, group :zookeeper-albeado-storm-cluster", :cause #}>}]
ERROR core - errors found [{:message "Unexpected exception: throw+: {:type :pallet/ssh-connection-failure, :message "ssh-fail: server 54.183.166.164, port 22, user storm, group :supervisor-albeado-storm-cluster", :cause #}", :type :pallet/action-execution-error, :cause #<ExceptionInfo slingshot.ExceptionInfo: throw+: {:type :pallet/ssh-connection-failure, :message "ssh-fail: server 54.183.166.164, port 22, user storm, group :supervisor-albeado-storm-cluster", :cause #}>}]
ERROR logging - Exception in thread "main"
ERROR core - errors found [{:message "Unexpected exception: throw+: {:type :pallet/ssh-connection-failure, :message "ssh-fail: server 54.183.167.245, port 22, user storm, group :nimbus-albeado-storm-cluster", :cause #}", :type :pallet/action-execution-error, :cause #<ExceptionInfo slingshot.ExceptionInfo: throw+: {:type :pallet/ssh-connection-failure, :message "ssh-fail: server 54.183.167.245, port 22, user storm, group :nimbus-albeado-storm-cluster", :cause #}>}]
ERROR logging - java.lang.RuntimeException: java.util.concurrent.ExecutionException: slingshot.ExceptionInfo: Error prevented completion of phase: Unexpected exception: throw+: {:type :pallet/ssh-connection-failure, :message "ssh-fail: server 54.183.168.178, port 22, user storm, group :zookeeper-albeado-storm-cluster", :cause #} (form-init5676622805172385725.clj:1)
ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5440)
ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5415)
ERROR logging - at clojure.lang.Compiler.load(Compiler.java:5857)
ERROR logging - at clojure.lang.Compiler.loadFile(Compiler.java:5820)
ERROR logging - at clojure.main$load_script.invoke(main.clj:221)
ERROR logging - at clojure.main$init_opt.invoke(main.clj:226)
ERROR logging - at clojure.main$initialize.invoke(main.clj:254)
ERROR logging - at clojure.main$null_opt.invoke(main.clj:279)
ERROR logging - at clojure.main$main.doInvoke(main.clj:354)
ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:422)
ERROR logging - at clojure.lang.Var.invoke(Var.java:369)
ERROR logging - at clojure.lang.AFn.applyToHelper(AFn.java:165)
ERROR logging - at clojure.lang.Var.applyTo(Var.java:482)
ERROR logging - at clojure.main.main(main.java:37)
ERROR logging - Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: slingshot.ExceptionInfo: Error prevented completion of phase: Unexpected exception: throw+: {:type :pallet/ssh-connection-failure, :message "ssh-fail: server 54.183.168.178, port 22, user storm, group :zookeeper-albeado-storm-cluster", :cause #}
ERROR logging - at clojure.lang.LazySeq.sval(LazySeq.java:47)
ERROR logging - at clojure.lang.LazySeq.seq(LazySeq.java:56)
ERROR logging - at clojure.lang.RT.seq(RT.java:450)
ERROR logging - at clojure.core$seq.invoke(core.clj:122)
ERROR logging - at clojure.core$dorun.invoke(core.clj:2450)
ERROR logging - at clojure.core$doall.invoke(core.clj:2465)
ERROR logging - at pallet.core$parallel_lift.invoke(core.clj:757)
ERROR logging - at pallet.core$lift_phase$fn__5430.invoke(core.clj:779)
ERROR logging - at clojure.lang.ArrayChunk.reduce(ArrayChunk.java:60)
ERROR logging - at clojure.core$r.invoke(core.clj:797)
ERROR logging - at pallet.core$lift_phase.invoke(core.clj:781)
ERROR logging - at pallet.core$lift_nodes$fn__5433.invoke(core.clj:794)
ERROR logging - at clojure.lang.ArrayChunk.reduce(ArrayChunk.java:58)
ERROR logging - at clojure.core$r.invoke(core.clj:797)
ERROR logging - at pallet.core$lift_nodes.invoke(core.clj:798)
ERROR logging - at pallet.core$lift_STAR_.invoke(core.clj:1310)
ERROR logging - at pallet.core$converge_STAR_.invoke(core.clj:1335)
ERROR logging - at pallet.core$converge.doInvoke(core.clj:1506)
ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:440)
ERROR logging - at backtype.storm.provision$converge_BANG_.invoke(provision.clj:41)
ERROR logging - at backtype.storm.provision$stop_BANG_.invoke(provision.clj:114)
ERROR logging - at backtype.storm.provision$_main$fn__8422.invoke(provision.clj:143)
ERROR logging - at backtype.storm.provision$_main.doInvoke(provision.clj:130)
ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:437)
ERROR logging - at clojure.lang.Var.invoke(Var.java:373)
ERROR logging - at user$eval5.invoke(form-init5676622805172385725.clj:1)
ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5424)
ERROR logging - ... 13 more
ERROR logging - Caused by: java.util.concurrent.ExecutionException: slingshot.ExceptionInfo: Error prevented completion of phase: Unexpected exception: throw+: {:type :pallet/ssh-connection-failure, :message "ssh-fail: server 54.183.168.178, port 22, user storm, group :zookeeper-albeado-storm-cluster", :cause #}
ERROR logging - at java.util.concurrent.FutureTask.report(FutureTask.java:122)
ERROR logging - at java.util.concurrent.FutureTask.get(FutureTask.java:188)
ERROR logging - at clojure.core$future_call$reify__5500.deref(core.clj:5399)
ERROR logging - at clojure.core$deref.invoke(core.clj:1765)
ERROR logging - at clojure.core$map$fn__3695.invoke(core.clj:2096)
ERROR logging - at clojure.lang.LazySeq.sval(LazySeq.java:42)
ERROR logging - ... 39 more
ERROR logging - Caused by: slingshot.ExceptionInfo: Error prevented completion of phase: Unexpected exception: throw+: {:type :pallet/ssh-connection-failure, :message "ssh-fail: server 54.183.168.178, port 22, user storm, group :zookeeper-albeado-storm-cluster", :cause #}
ERROR logging - at pallet.core$raise_on_error$fn__5204.invoke(core.clj:526)
ERROR logging - at pallet.core$middleware_handler$fn__5197.invoke(core.clj:495)
ERROR logging - at pallet.core$apply_phase_to_node.invoke(core.clj:656)
ERROR logging - at pallet.core$eval5367$fn__5368$iter__5369__5373$fn__5374$fn__5379.invoke(core.clj:721)
ERROR logging - at clojure.lang.AFn.call(AFn.java:18)
ERROR logging - at java.util.concurrent.FutureTask.run(FutureTask.java:262)
ERROR logging - at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
ERROR logging - at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
ERROR logging - at java.lang.Thread.run(Thread.java:744)
ERROR logging - Caused by: slingshot.ExceptionInfo: throw+: {:type :pallet/ssh-connection-failure, :message "ssh-fail: server 54.183.168.178, port 22, user storm, group :zookeeper-albeado-storm-cluster", :cause #}
ERROR logging - at pallet.execute$ensure_ssh_connection$fn__4714.invoke(execute.clj:383)
ERROR logging - at pallet.execute$ensure_ssh_connection.invoke(execute.clj:380)
ERROR logging - at pallet.execute$ssh_bash_on_target.invoke(execute.clj:481)
ERROR logging - at pallet.core$executor.invoke(core.clj:389)
ERROR logging - at pallet.action_plan$execute_action.invoke(action_plan.clj:535)
ERROR logging - at pallet.action_plan$execute$fn__2363.invoke(action_plan.clj:554)
ERROR logging - at clojure.core$r.invoke(core.clj:799)
ERROR logging - at pallet.action_plan$execute.invoke(action_plan.clj:551)
ERROR logging - at pallet.action_plan$execute_for_target.invoke(action_plan.clj:649)
ERROR logging - at pallet.core$execute.invoke(core.clj:502)
ERROR logging - at pallet.core$translate_action_plan$fn__5192.invoke(core.clj:488)
ERROR logging - at pallet.execute$ssh_user_credentials$fn__4860.invoke(execute.clj:655)
ERROR logging - at pallet.execute$execute_with_ssh$execute_with_ssh_fn__4846.invoke(execute.clj:603)
ERROR logging - at pallet.core$raise_on_error$fn__5204.invoke(core.clj:519)
ERROR logging - ... 8 more
ERROR logging - Caused by: com.jcraft.jsch.JSchException: java.net.ConnectException: Connection timed out
ERROR logging - at com.jcraft.jsch.Util.createSocket(Util.java:344)
ERROR logging - at com.jcraft.jsch.Session.connect(Session.java:194)
ERROR logging - at com.jcraft.jsch.Session.connect(Session.java:162)
ERROR logging - at clj_ssh.ssh$connect.invoke(ssh.clj:300)
ERROR logging - at pallet.execute$ensure_ssh_connection$fn__4714.invoke(execute.clj:381)
ERROR logging - ... 21 more
ERROR logging - Caused by: java.net.ConnectException: Connection timed out
ERROR logging - at java.net.PlainSocketImpl.socketConnect(Native Method)
ERROR logging - at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
ERROR logging - at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
ERROR logging - at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
ERROR logging - at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
ERROR logging - at java.net.Socket.connect(Socket.java:579)
ERROR logging - at java.net.Socket.connect(Socket.java:528)
ERROR logging - at java.net.Socket.(Socket.java:425)
ERROR logging - at java.net.Socket.(Socket.java:208)
ERROR logging - at com.jcraft.jsch.Util.createSocket(Util.java:338)
ERROR logging - ... 25 more

I don't know what is happening. Could someone please help?

BTW, the Java SDK I was using was JDK 1.7.045 64 bit.

Final IP Address Output is borked

This is a minor issue which I will fix within 48 hours. Somebody must have changed the data structure storing the ip addresses for the nodes and so now the code that prints them is outputting an object address instead of a value.

Build storm failed

I am getting the following error when using storm-deploy, which seems to occur while attempting to build Storm on the nimbus-instance:

Build storm...
Cloning into 'storm'...
remote: Reusing existing pack: 22107, done.
Receiving objects:   0% (1/22107)   
[...]
Receiving objects: 100% (22107/22107), 5.79 MiB, done.
Resolving deltas:   0% (0/13280)
[...]
Resolving deltas: 100% (13280/13280), done.
Already up-to-date.
bash: bin/build_release.sh: No such file or directory
Build storm failed
logout

Re-running the script on the same instance(s) give the same error-message.

I am launching storm-deploy with this command:
lein deploy-storm --start --name storm

and I am using the latest commit on the storm-deploy master branch.

Update version of com.jcraft/jsch.agentproxy.*

Using storm-deploy out of the box (on an old JRE 1.6.0_45), I encountered a known bug in jsch.agentproxy that prevented me from deploying a cluster. Here's a reference to the jsch.agentproxy bug-fix: ymnk/jsch-agent-proxy@58ccf74

In https://github.com/nathanmarz/storm-deploy/blob/master/project.clj

             [com.jcraft/jsch.agentproxy.usocket-jna "0.0.5"]
             [com.jcraft/jsch.agentproxy.usocket-nc "0.0.5"]
             [com.jcraft/jsch.agentproxy.sshagent "0.0.5"]
             [com.jcraft/jsch.agentproxy.pageant "0.0.5"]
             [com.jcraft/jsch.agentproxy.core "0.0.5"]
             [com.jcraft/jsch.agentproxy.jsch "0.0.5"]

I changed all of the "0.0.5" to "0.0.6". The next cluster I launched brought in the new version of jsch.agentproxy and everything ran as expected.

Even if the bug is only made manifest by my old version of Java, it's still worth keeping the dependencies up to date. (Note that the "0.0.5" -> "0.0.6" version update only touches this one issue.)

UI and/or DRPC sometimes fail to daemonize

This issue started happening when the DRPC daemonization was added

pip install on executors

Is there any way to pip install packages that are required by the executor? I would like to include a few like cjson for example.

Daemons should start on reboot

Install OpenJDK 7 by default?

I couldn't find a better place to post this and, as someone relatively new to Java & Clojure, hacking on this repo is slow-going.

Anyhow, is there a way to configure things such that OpenJDK 7 is installed and used by default? Specifically, I'm using a JRuby DSL (https://github.com/colinsurprenant/redstorm) to prototype something and it requires OpenJDK 7.

deploy fails with storm 0.9+ because build moved to maven3

the latest versions of storm use maven3 to build, but storm-deploy is relying on bin/build-release.sh, which has now been removed.

Storm UI is not launched

I just launched the storm cluster by:
$ lein deploy-storm --start --name mycluster --branch 0.8.2
Then I run to find cluster information:
$ lein deploy-storm --attach --name mycluster

I can ssh into the Nimbus instance. I cannot see the port 8080 is open by:
$ netstat -lptu
(No info could be read for "-p": geteuid()=1001 but you should be root.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 :ssh *: LISTEN -
tcp 0 0 :8649 *: LISTEN -
tcp 0 0 :8651 *: LISTEN -
tcp 0 0 :8652 *: LISTEN -
tcp 0 0 :http *: LISTEN -
tcp6 0 0 [::]:ssh [::]:* LISTEN -
udp 0 0 239.2.11.71:8649 : -
udp 0 0 :bootpc *: -

My env is:
$ lein -v
Leiningen 2.3.4 on Java 1.6.0_24 Java HotSpot(TM) 64-Bit Server VM

What should I do? I tried latest storm verion and 0.8.3. All do not work. By the way, I used m1.medium for all instances. And I set 2 supervisors.

Thanks.

Windows Cygwin

Ran into a couple of issues using Cygwin. I had to run:

eval `ssh-agent`

to enable ssh to the instances that were started.

Had to install pageant.exe:

http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

to get past the error:

Could not initialize class com.jcraft.jsch.agentproxy.usocket.JNAUSocketFactory$CLibrary

thx to: http://logs.lazybot.org/irc.freenode.net/%23pallet/2013-04-11.txt

I also ran into the "Security Group Does Not Exist" error but haven't worked through that one yet.

Hope this helps someone else...

Storm deploy Default JDK - java 1.6

Trying to install Storm Cluster using storm-deploy branch (install-0.9.1) ( storm commit 78d431828e88c35e55e2a7a9ad66d2ce6f8bcd07). By default the Storm-deploy using java 1.6 open jdk in all the nodes. Basically I'm trying to use kinesis storm spout (https://github.com/awslabs/kinesis-storm-spout) to connect kinesis stream. This project was built using java 1.7 or later. So I'm not able connect to kinesis stream. So please specify, what are the changes to be made in storm deploy to install java 1.7 open JDK in all nodes.

Document how storm-deploy works with release tags

The documentation covered in #45 suggests that to work with release 0.9.0-rc2 you run the following:

lein deploy-storm --start --name mycluster --branch master --commit 32098d5b2694434ea43d430a4703fbe51bab268f

However there is a more natural way:

lein deploy-storm --start --name mycluster --commit 0.9.0-rc2

The storm-deploy method signatures and variable names contained in #45 constantly refer to SHA1s Given that storm-deploy safely takes tags as well as commit SHA1s I would like to update the source code to reflect that. I'll submit a pull-request if there is any buy-in. This would be a non-functional change.

No such file or directoy

The following error was found when deploying the default configuration.

The exact timing is right after it installs jzmq, then the following lines print after "INVOKE make (install)":

WARN deprecate - DEPRECATED [backtype/storm/crate/storm.clj:94] pallet.resource.exec-script/exec-checked-script is deprecated, use pallet.action.exec-script/exec-checked-script
WARN deprecate - DEPRECATED [backtype/storm/crate/storm.clj:89] pallet.resource.exec-script/exec-checked-script is deprecated, use pallet.action.exec-script/exec-checked-script
Exception in thread "main" java.io.FileNotFoundException: /Users/nathan/.ssh/storm (No such file or directory) (NO_SOURCE_FILE:1)
........
at clojure.java.io$reader.doInvoke(io.clj:96)
at clojure.lang.RestFn.invoke(RestFn.java:411)
at clojure.lang.AFn.applyToHelper(AFn.java:163)
at clojure.lang.RestFn.applyTo(RestFn.java:133)
at clojure.core$apply.invoke(core.clj:542)
at clojure.core$slurp.doInvoke(core.clj:5372)
at clojure.lang.RestFn.invoke(RestFn.java:411)
at backtype.storm.crate.storm$install_github_key.invoke(storm.clj:40)
at backtype.storm.crate.storm$make.invoke(storm.clj:88)
at backtype.storm.crate.storm$install_supervisor.invoke(storm.clj:109)
at backtype.storm.node$supervisor_server_spec$fn__8573.invoke(node.clj:96)

"lein deploy-storm --start --name mycluster" does not work

Hi,
I tried installing Storm on ec2 with "lein deploy-storm --start --name mycluster --branch 0.8.3" Whilt installing it show everything OK but when I tailed nohup.out. It continuously throws this error.
Exception in thread "main" java.lang.NoClassDefFoundError: backtype/storm/command/config_value
Caused by: java.lang.ClassNotFoundException: backtype.storm.command.config_value
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: backtype.storm.command.config_value. Program will exit.
Exception in thread "main" java.lang.NoClassDefFoundError: backtype/storm/command/config_value
Caused by: java.lang.ClassNotFoundException: backtype.storm.command.config_value
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: backtype.storm.command.config_value. Program will exit.
Running: java -server -Dstorm.options= -Dstorm.home=/home/storm/storm-0.8.2 -Djava.library.path= -Dstorm.conf.file= -cp /home/storm/storm-0.8.2/log4j:/home/storm/storm-0.8.2/conf -Dlogfile.name=nimbus.log -Dlog4j.configuration=storm.log.properties backtype.storm.daemon.nimbus
Exception in thread "main" java.lang.NoClassDefFoundError: backtype/storm/daemon/nimbus
Caused by: java.lang.ClassNotFoundException: backtype.storm.daemon.nimbus
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: backtype.storm.daemon.nimbus. Program will exit

And When I tried this "lein deploy-storm --start --name my cluster" there is some maven build error.

Thanks,
Vipul

cluster compute instances can't be found

When trying to start up a cc2.8xlarge we get the following error. I think it might be related to the jclouds version, but I'm having trouble upgrading it.

java.util.NoSuchElementException: no hardware profiles support images matching params: [biggest=false, fastest=false, imageName=ubuntu/images/hvm/ubuntu-quantal-12.10-amd64-server-20130206, imageDescription=^099720109477/ubuntu/images/hvm/ubuntu-quantal-12.10-amd64-server-20130206$, imageId=null, imagePredicate=null, imageVersion=^20130206$, location=[id=us-east-1, scope=REGION, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA], metadata={}], minCores=16.0, minRam=61952, osFamily=ubuntu, osName=null, osDescription=099720109477/ubuntu/images/hvm/ubuntu-quantal-12.10-amd64-server-20130206, osVersion=, osArch=hvm, os64Bit=true, hardwareId=null, hypervisor=null]
at org.jclouds.compute.domain.internal.TemplateBuilderImpl.throwNoSuchElementExceptionAfterLoggingHardwareIds(TemplateBuilderImpl.java:683)
at org.jclouds.compute.domain.internal.TemplateBuilderImpl.resolveHardware(TemplateBuilderImpl.java:717)
at org.jclouds.compute.domain.internal.TemplateBuilderImpl.build(TemplateBuilderImpl.java:634)
at org.jclouds.ec2.compute.strategy.EC2CreateNodesInGroupThenAddToSet.execute(EC2CreateNodesInGroupThenAddToSet.java:136)
at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:200)
at org.jclouds.aws.ec2.compute.AWSEC2ComputeService.createNodesInGroup(AWSEC2ComputeService.java:130)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:90)
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
at org.jclouds.compute2$create_nodes.invoke(compute2.clj:166)
at pallet.compute.jclouds.JcloudsService.run_nodes(jclouds.clj:477)
at pallet.core$create_nodes.invoke(core.clj:845)
at pallet.core$adjust_node_count.invoke(core.clj:912)
at pallet.core$parallel_adjust_node_counts$p_a_n_c_future__5497$fn__5498.invoke(core.clj:1003)
at clojure.lang.AFn.call(AFn.java:18)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:680)

IOException getting whatsmyip.akamai.com kills deploy

After executing lein deploy-storm --start --name my-storm and letting it run for about 8-10 minutes I am continuing to run into the following exception. I've tried this with the master branch as well as the install-0.9.1 branch of storm-deploy, both without luck.

################################################################################
# CLUSTERS CONFIG FILE
################################################################################
nimbus.image: "us-east-1/ami-d726abbe"         #64-bit ubuntu
nimbus.hardware: "m1.large"

supervisor.count: 2
supervisor.image: "us-east-1/ami-d726abbe"         #64-bit ubuntu on us-east-1
supervisor.hardware: "m1.large"
#supervisor.spot.price: 1.60

zookeeper.count: 1
zookeeper.image: "us-east-1/ami-d726abbe"         #64-bit ubuntu
zookeeper.hardware: "m1.large"

# nimbus.image: "us-west-2/ami-fa9cf1ca"         #64-bit ubuntu
# nimbus.hardware: "m1.large"

# supervisor.count: 2
# supervisor.image: "us-west-2/ami-fa9cf1ca"         #64-bit ubuntu on us-west-2
# supervisor.hardware: "m1.large"
# #supervisor.spot.price: 1.60

# zookeeper.count: 1
# zookeeper.image: "us-west-2/ami-fa9cf1ca"         #64-bit ubuntu
# zookeeper.hardware: "m1.large"

As I've not yet been able to actually deploy storm, any guidance or suggestions are appreciated. Here's the exception:

ERROR logging - Exception in thread "main"
ERROR logging - java.io.IOException: Server returned HTTP response code: 403 for URL: http://whatismyip.akamai.com/ (form-init4146737830589478912.clj:1)
ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5440)
ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5415)
ERROR logging - at clojure.lang.Compiler.load(Compiler.java:5857)
ERROR logging - at clojure.lang.Compiler.loadFile(Compiler.java:5820)
ERROR logging - at clojure.main$load_script.invoke(main.clj:221)
ERROR logging - at clojure.main$init_opt.invoke(main.clj:226)
ERROR logging - at clojure.main$initialize.invoke(main.clj:254)
ERROR logging - at clojure.main$null_opt.invoke(main.clj:279)
ERROR logging - at clojure.main$main.doInvoke(main.clj:354)
ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:422)
ERROR logging - at clojure.lang.Var.invoke(Var.java:369)
ERROR logging - at clojure.lang.AFn.applyToHelper(AFn.java:165)
ERROR logging - at clojure.lang.Var.applyTo(Var.java:482)
ERROR logging - at clojure.main.main(main.java:37)
ERROR logging - Caused by: java.io.IOException: Server returned HTTP response code: 403 for URL: http://whatismyip.akamai.com/
ERROR logging - at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1626)
ERROR logging - at java.net.URL.openStream(URL.java:1037)
ERROR logging - at backtype.storm.security$fn__1784.invoke(security.clj:13)
ERROR logging - at clojure.lang.AFn.applyToHelper(AFn.java:159)
ERROR logging - at clojure.lang.AFn.applyTo(AFn.java:151)
ERROR logging - at clojure.core$apply.invoke(core.clj:540)
ERROR logging - at clojure.core$memoize$fn__4542.doInvoke(core.clj:5093)
ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:398)
ERROR logging - at backtype.storm.security$authorizeme.invoke(security.clj:21)
ERROR logging - at backtype.storm.provision$attach_BANG_.invoke(provision.clj:62)
ERROR logging - at backtype.storm.provision$start_with_nodes_BANG_.invoke(provision.clj:89)
ERROR logging - at backtype.storm.provision$start_BANG_.invoke(provision.clj:100)
ERROR logging - at backtype.storm.provision$_main$fn__8875.invoke(provision.clj:156)
ERROR logging - at backtype.storm.provision$_main.doInvoke(provision.clj:136)
ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:437)
ERROR logging - at clojure.lang.Var.invoke(Var.java:373)
ERROR logging - at user$eval5$fn__7.invoke(form-init4146737830589478912.clj:1)
ERROR logging - at user$eval5.invoke(form-init4146737830589478912.clj:1)
ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5424)
ERROR logging - ... 13 more

Could not find artifact org.clojure:clojure-contrib

I get the following error when running lein deps

Could not find artifact org.clojure:clojure-contrib:pom:1.1.0 in central (http://repo1.maven.org/maven2)
Could not find artifact org.clojure:clojure-contrib:pom:1.1.0 in clojars (https://clojars.org/repo/)
Could not find artifact org.clojure:clojure-contrib:pom:1.1.0 in jclouds-snapshot (https://oss.sonatype.org/content/repositories/snapshots)
Could not find artifact org.clojure:clojure-contrib:pom:1.1.0 in sonatype (https://oss.sonatype.org/content/repositories/releases)
Could not transfer artifact org.clojure:clojure-contrib:pom:1.1.0 from/to sonatype-nexus-releases (http://oss.sonatype.org/content/repositories/releases): Connection to http://oss.sonatype.org refused
Could not transfer artifact org.clojure:clojure-contrib:pom:1.1.0 from/to clojure (http://build.clojure.org/releases): Connection to http://build.clojure.org refused
Could not transfer artifact org.clojure:clojure-contrib:pom:1.1.0 from/to clojure-snapshots (http://build.clojure.org/snapshots): Connection to http://build.clojure.org refused
Check :dependencies and :repositories for typos.
It's possible the specified jar is not in any repository.
If so, see "Free-floating Jars" under http://j.mp/repeatability

Unable to connect to Nimbus using Storm-Deploy on AWS Cloud

We are following Storm-Deploy ( https://github.com/nathanmarz/storm-deploy ) to deploy storm cluster on AWS Cloud. Our cluster is being successfully deployed and the separate EC2 instances are being spawned for Nimbus, Zookeeper and Supervisors.
However we are not being able to connect to Storm UI. Further we are not able to deploy the topology in the cluster as it cannot connect to Nimbus using ssh.
Each time storm-deploy launches a new cluster, it creates a new key-pair. We are not able to locate where the pem file of the new key pair is being saved.
We have tried connecting with

authorized_keys
id_rsa
id_rsa.pub
as pem file using ssh command with Nimbus, Zookeeper and Supervisors machines. Each time we are getting the same error:
"Permission denied (public key)"

Please suggest how to login to Nimbus and submit topology.

Our pallet.clj file:
(defpallet
:services
{
:default {
:blobstore-provider "aws-s3"
:provider "aws-ec2"
:environment {:user {:username "storm"
:private-key-path "/.ssh/id_rsa"
:public-key-path "/.ssh/id_rsa.pub"}
:aws-user-id "XXXX-XXXX-XXXX"}
:identity "XXXXXXXXXXXX"
:credential "XXXXXXXXXX"
:jclouds.regions "us-west-2"
}
})

Storm-deploy with Storm >= 0.9.0

I spent about the entire day yesterday trying to do the basic storm deploy
commands. I've tried on ubuntu, mac and a aws box. The deploy works for
storm 0.8.2 but our topoligies are written with storm 0.9.0.1.

The error I'm getting is (I think its the first one)

-------begin error
2014-02-09 18:57:39,833 INFO pallet.execute Output: 54.184.12.237
Already up-to-date.
bash: bin/build_release.sh: No such file or directory
Build storm failed
logout

2014-02-09 18:57:39,842 INFO pallet.execute Output: 54.184.41.36
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking dependency style of g++... gcc3
checking how to run the C++ preprocessor... g++ -E

2014-02-09 18:57:39,936 ERROR pallet.execute Exit status : 1
-------end error

The commands Im using are (Ive tried numerous branches)

lein deploy-storm --start --name test
lein deploy-storm --start --name test --branch 0.9.0.1
lein deploy-storm --start --name test --branch 0.9.0

My config is as follows. I was testing with t1.micros but now on m1.smalls.

clusters.yaml
nimbus.image: "us-west-2/ami-84d7b7b4"
nimbus.hardware: "m1.small"

supervisor.count: 2
supervisor.image: "us-west-2/ami-84d7b7b4"
supervisor.hardware: "m1.small"

supervisor.spot.price: 1.60

zookeeper.count: 1
zookeeper.image: "us-west-2/ami-84d7b7b4"
zookeeper.hardware: "m1.small"

config.clj
(defpallet
:services
{
:default {
:blobstore-provider "aws-s3"
:provider "aws-ec2"
:environment {:user {:username "storm"
:private-key-path "/path/to/key.pem"
:public-key-path "/path/to/key.pub"}
:aws-user-id "xxxxxxxxxxxxxx"}
:identity "xxxxxxx"
:credential "xxxxxxx"
:jclouds.regions "us-west-2"
}
})

Is there something obvious I'm missing? After getting this error, the process hangs and doesn't exit.

deleted

Cant install storm deploy on EC2

I keep getting following exception when I start storm cluster...Please advise

Exception in thread "main" org.jclouds.rest.AuthorizationException: POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 -> HTTP/1.1 401 Unauthorized (NO_SOURCE_FILE:1)

I am running nimbus, supervisor, zookeeper on the same machine...

here is the storm.yaml

storm.local.dir: "/mnt/storm"
storm.zookeeper.servers:
- "localhost"
java.library.path: "/usr/local/lib/"
nimbus.host: "127.0.0.1"
nimbus.task.launch.secs: 240
supervisor.worker.start.timeout.secs: 240
supervisor.worker.timeout.secs: 240
storm.zookeeper.servers:
- "127.0.0.1"
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703

here is the cluster.yaml

nimbus.image: "us-east-1a/ami-idxxxx" #64-bit ubuntu
nimbus.hardware: "m1.large"
supervisor.count: 1
supervisor.image: "us-east-1a/ami-idxxxx" #64-bit ubuntu on us-east
supervisor.hardware: "c1.xlarge"

supervisor.spot.price: 1.60

zookeeper.count: 1
zookeeper.image: "us-east-1a/ami-idxxxx" #64-bit ubuntu
zookeeper.hardware: "m1.large"

here is the config.clj - (I have not setup any ssh keys .. all you need is username password to get in...)

(defpallet
:services
{
:default {
:blobstore-provider "aws-s3"
:provider "aws-ec2"
:environment {:user {:username "storm" ; this must be "storm"
:private-key-path ""
:public-key-path ""}
:aws-user-id "id i see on aws dashboard"}
:identity "userid"
:credential "password"
:jclouds.regions "us-east-1a"
}
})

Deploy storm cluster EC2 instances in a AWS VPC

Hi,

I am trying to understand if this functionality is available or not, using Pallet one can create EC2 instances within a AWS VPC by defining the subnet id in ~/.pallet/config.clj.

:location {:subnet-id "subxxxx"}

Can the same be used for creating storm cluster instances ?

icedtea-netx error when start new storm cluster

I ran the command for start new cluster on ec2 and i faced with this issue: Errors were encountered while processing: icedtea-netx:amd64
Details are :
2014-09-24 10:17:29,273 INFO pallet.execute Output: 54.220.40.192
Setting up icedtea-netx:amd64 (1.5.1-1ubuntu1) ...
update-alternatives: warning: forcing reinstallation of alternative /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/itweb-settings because link group itweb-settings is broken
update-alternatives: warning: not replacing /usr/share/man/man1/itweb-settings.1.gz with a link
update-alternatives: warning: forcing reinstallation of alternative /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/itweb-settings because link group itweb-settings is broken
update-alternatives: warning: not replacing /usr/share/man/man1/itweb-settings.1.gz with a link
update-alternatives: error: alternative path /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/javaws doesn't exist
dpkg: error processing package icedtea-netx:amd64 (--configure):
subprocess installed post-installation script returned error exit status 2
Errors were encountered while processing:
icedtea-netx:amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)
Failed to perform requested operation on package. Trying to recover:
Setting up icedtea-netx:amd64 (1.5.1-1ubuntu1) ...
update-alternatives: warning: forcing reinstallation of alternative /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/itweb-settings because link group itweb-settings is broken
update-alternatives: warning: not replacing /usr/share/man/man1/itweb-settings.1.gz with a link
update-alternatives: warning: forcing reinstallation of alternative /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/itweb-settings because link group itweb-settings is broken
update-alternatives: warning: not replacing /usr/share/man/man1/itweb-settings.1.gz with a link

2014-09-24 10:17:29,870 INFO pallet.execute Output: 54.228.129.82

...done
Packages...
Reading package lists...
Building dependency tree...
Reading state information...

2014-09-24 10:17:30,041 INFO pallet.execute Output: 54.216.86.253
Building tag database...
The following partially installed packages will be configured:
icedtea-netx
No packages will be installed, upgraded, or removed.

Can everyone help me to solve out this case ?
deploy command is: lein deploy-storm --start --name Test --branch 0.9.0.1

Thanks

Always launching m1.small instance on EC2

Hi :)

I have an issue with the EC2 Instances deployed... I set my clusters.yaml to:

nimbus.image: "us-west-1/ami-827252c7"         #64-bit ubuntu
nimbus.hardware: "m3.large"

supervisor.count: 2
supervisor.image: "us-west-1/ami-827252c7"         #64-bit ubuntu on eu-west-1
supervisor.hardware: "c3.xlarge"
#supervisor.spot.price: 1.60


zookeeper.count: 1
zookeeper.image: "us-west-1/ami-827252c7"         #64-bit ubuntu
zookeeper.hardware: "m3.large"

But it always launch m1.small instances... I updated the project.clj to use
[com.palletops/pallet-jclouds "1.7.3"]
but nothing changed.

Debug logs:

DEBUG compute - >> searching params({imageId=us-west-1/ami-827252c7, hardwareId=m3.large})
DEBUG compute - >> searching params({imageId=us-west-1/ami-827252c7, hardwareId=m3.large})
DEBUG compute - >> searching params({imageId=us-west-1/ami-827252c7, hardwareId=c3.xlarge})
DEBUG compute - <<   matched image(us-west-1/ami-827252c7) hardware(m3.large) location(us-west-1)
DEBUG compute - <<   matched image(us-west-1/ami-827252c7) hardware(c3.xlarge) location(us-west-1)
DEBUG compute - <<   matched image(us-west-1/ami-827252c7) hardware(m3.large) location(us-west-1)
...
DEBUG jclouds -   options {:image-id "us-west-1/ami-827252c7", :inbound-ports (6627 22)}
DEBUG jclouds -   options {:image-id "us-west-1/ami-827252c7", :inbound-ports (6700 6701 6702 6703 22)}
DEBUG jclouds -   options {:image-id "us-west-1/ami-827252c7", :inbound-ports (2181 22)}
DEBUG compute - >> searching params({imageId=us-west-1/ami-827252c7})
DEBUG compute - >> searching params({imageId=us-west-1/ami-827252c7})
DEBUG compute - >> searching params({imageId=us-west-1/ami-827252c7})
DEBUG compute - <<   matched image(us-west-1/ami-827252c7) hardware(m1.small) location(us-west-1)
DEBUG compute - <<   matched image(us-west-1/ami-827252c7) hardware(m1.small) location(us-west-1)
DEBUG compute - <<   matched image(us-west-1/ami-827252c7) hardware(m1.small) location(us-west-1)

Do you have any idea ?

Thanks

deploy to AWS error

cmd used to run:

sudo lein deploy-storm --start --name mycluster --branch master --commit 0.9.0-rc2

Starting cluster with storm branch master and commit 0.9.0-rc2 INFO provision - Provisioning nodes [nn=1, sn=2, zn=1] DEBUG core - pallet version: 0.7.3 INFO core - lift-nodes phases [:destroy-server], groups [] INFO core - lift-group-phase phases [:destroy-group], groups [] INFO core - lift-group-phase phases [:create-group], groups [:zookeeper-mycluster :supervisor-mycluster :nimbus-mycluster] INFO core - parallel-apply-phase :pallet.phase/pre-create-group for :zookeeper-mycluster INFO core - parallel-apply-phase-to-target :group :pallet.phase/pre-create-group for :zookeeper-mycluster INFO core - parallel-apply-phase :pallet.phase/pre-create-group for :supervisor-mycluster INFO core - parallel-apply-phase-to-target :group :pallet.phase/pre-create-group for :supervisor-mycluster INFO core - parallel-apply-phase :pallet.phase/pre-create-group for :nimbus-mycluster INFO core - parallel-apply-phase-to-target :group :pallet.phase/pre-create-group for :nimbus-mycluster INFO core - parallel-apply-phase :create-group for :zookeeper-mycluster INFO core - parallel-apply-phase-to-target :group :create-group for :zookeeper-mycluster INFO core - parallel-apply-phase :create-group for :supervisor-mycluster INFO core - parallel-apply-phase-to-target :group :create-group for :supervisor-mycluster INFO core - parallel-apply-phase :create-group for :nimbus-mycluster INFO core - parallel-apply-phase-to-target :group :create-group for :nimbus-mycluster INFO core - parallel-apply-phase :pallet.phase/post-create-group for :zookeeper-mycluster INFO core - parallel-apply-phase-to-target :group :pallet.phase/post-create-group for :zookeeper-mycluster INFO core - parallel-apply-phase :pallet.phase/post-create-group for :supervisor-mycluster INFO core - parallel-apply-phase-to-target :group :pallet.phase/post-create-group for :supervisor-mycluster INFO core - parallel-apply-phase :pallet.phase/post-create-group for :nimbus-mycluster INFO core - parallel-apply-phase-to-target :group :pallet.phase/post-create-group for :nimbus-mycluster INFO core - adjust-server-count :zookeeper-mycluster INFO core - adjust-server-count :supervisor-mycluster INFO core - adjust-server-count :nimbus-mycluster INFO core - Starting 1 nodes for :nimbus-mycluster, os-family null INFO core - Starting 1 nodes for :zookeeper-mycluster, os-family null INFO core - Starting 2 nodes for :supervisor-mycluster, os-family null ERROR futures - Adjust node count exception: No implementation of method: :ensure-os-family of protocol: #'pallet.compute/ComputeService found for class: nil java.lang.IllegalArgumentException: No implementation of method: :ensure-os-family of protocol: #'pallet.compute/ComputeService found for class: nil at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:471) at pallet.compute$eval1063$fn__1092$G__1046__1099.invoke(compute.clj:49) at pallet.core$create_nodes$fn__5444.invoke(core.clj:837) at clojure.lang.AFn.applyToHelper(AFn.java:163) at clojure.lang.AFn.applyTo(AFn.java:151) at clojure.core$apply.invoke(core.clj:542) at clojure.core$update_in.doInvoke(core.clj:4959) at clojure.lang.RestFn.invoke(RestFn.java:446) at pallet.core$create_nodes.invoke(core.clj:836) at pallet.core$adjust_node_count.invoke(core.clj:912) at pallet.core$parallel_adjust_node_counts$p_a_n_c_future__5506$fn__5507.invoke(core.clj:1003) at clojure.lang.AFn.call(AFn.java:18) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) DEBUG futures - Adjust node count exception java.lang.IllegalArgumentException: No implementation of method: :ensure-os-family of protocol: #'pallet.compute/ComputeService found for class: nil at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:471) at pallet.compute$eval1063$fn__1092$G__1046__1099.invoke(compute.clj:49) at pallet.core$create_nodes$fn__5444.invoke(core.clj:837) at clojure.lang.AFn.applyToHelper(AFn.java:163) at clojure.lang.AFn.applyTo(AFn.java:151) at clojure.core$apply.invoke(core.clj:542) at clojure.core$update_in.doInvoke(core.clj:4959) at clojure.lang.RestFn.invoke(RestFn.java:446) at pallet.core$create_nodes.invoke(core.clj:836) at pallet.core$adjust_node_count.invoke(core.clj:912) at pallet.core$parallel_adjust_node_counts$p_a_n_c_future__5506$fn__5507.invoke(core.clj:1003) at clojure.lang.AFn.call(AFn.java:18) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) ERROR futures - Adjust node count exception: No implementation of method: :ensure-os-family of protocol: #'pallet.compute/ComputeService found for class: nil java.lang.IllegalArgumentException: No implementation of method: :ensure-os-family of protocol: #'pallet.compute/ComputeService found for class: nil at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:471) at pallet.compute$eval1063$fn__1092$G__1046__1099.invoke(compute.clj:49) at pallet.core$create_nodes$fn__5444.invoke(core.clj:837) at clojure.lang.AFn.applyToHelper(AFn.java:163) at clojure.lang.AFn.applyTo(AFn.java:151) at clojure.core$apply.invoke(core.clj:542) at clojure.core$update_in.doInvoke(core.clj:4959) at clojure.lang.RestFn.invoke(RestFn.java:446) at pallet.core$create_nodes.invoke(core.clj:836) at pallet.core$adjust_node_count.invoke(core.clj:912) at pallet.core$parallel_adjust_node_counts$p_a_n_c_future__5506$fn__5507.invoke(core.clj:1003) at clojure.lang.AFn.call(AFn.java:18) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) DEBUG futures - Adjust node count exception java.lang.IllegalArgumentException: No implementation of method: :ensure-os-family of protocol: #'pallet.compute/ComputeService found for class: nil at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:471) at pallet.compute$eval1063$fn__1092$G__1046__1099.invoke(compute.clj:49) at pallet.core$create_nodes$fn__5444.invoke(core.clj:837) at clojure.lang.AFn.applyToHelper(AFn.java:163) at clojure.lang.AFn.applyTo(AFn.java:151) at clojure.core$apply.invoke(core.clj:542) at clojure.core$update_in.doInvoke(core.clj:4959) at clojure.lang.RestFn.invoke(RestFn.java:446) at pallet.core$create_nodes.invoke(core.clj:836) at pallet.core$adjust_node_count.invoke(core.clj:912) at pallet.core$parallel_adjust_node_counts$p_a_n_c_future__5506$fn__5507.invoke(core.clj:1003) at clojure.lang.AFn.call(AFn.java:18) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) ERROR futures - Adjust node count exception: No implementation of method: :ensure-os-family of protocol: #'pallet.compute/ComputeService found for class: nil java.lang.IllegalArgumentException: No implementation of method: :ensure-os-family of protocol: #'pallet.compute/ComputeService found for class: nil at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:471) at pallet.compute$eval1063$fn__1092$G__1046__1099.invoke(compute.clj:49) at pallet.core$create_nodes$fn__5444.invoke(core.clj:837) at clojure.lang.AFn.applyToHelper(AFn.java:163) at clojure.lang.AFn.applyTo(AFn.java:151) at clojure.core$apply.invoke(core.clj:542) at clojure.core$update_in.doInvoke(core.clj:4959) at clojure.lang.RestFn.invoke(RestFn.java:446) at pallet.core$create_nodes.invoke(core.clj:836) at pallet.core$adjust_node_count.invoke(core.clj:912) at pallet.core$parallel_adjust_node_counts$p_a_n_c_future__5506$fn__5507.invoke(core.clj:1003) at clojure.lang.AFn.call(AFn.java:18) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) DEBUG futures - Adjust node count exception java.lang.IllegalArgumentException: No implementation of method: :ensure-os-family of protocol: #'pallet.compute/ComputeService found for class: nil at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:471) at pallet.compute$eval1063$fn__1092$G__1046__1099.invoke(compute.clj:49) at pallet.core$create_nodes$fn__5444.invoke(core.clj:837) at clojure.lang.AFn.applyToHelper(AFn.java:163) at clojure.lang.AFn.applyTo(AFn.java:151) at clojure.core$apply.invoke(core.clj:542) at clojure.core$update_in.doInvoke(core.clj:4959) at clojure.lang.RestFn.invoke(RestFn.java:446) at pallet.core$create_nodes.invoke(core.clj:836) at pallet.core$adjust_node_count.invoke(core.clj:912) at pallet.core$parallel_adjust_node_counts$p_a_n_c_future__5506$fn__5507.invoke(core.clj:1003) at clojure.lang.AFn.call(AFn.java:18) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) DEBUG core - pallet version: 0.7.3 INFO core - lift-nodes phases [:settings :configure], groups [:zookeeper-mycluster :supervisor-mycluster :nimbus-mycluster] INFO core - parallel-apply-phase :pallet.phase/pre-settings for :zookeeper-mycluster INFO core - parallel-apply-phase-to-target :node :pallet.phase/pre-settings for :zookeeper-mycluster with 0 nodes INFO core - parallel-apply-phase :pallet.phase/pre-settings for :supervisor-mycluster INFO core - parallel-apply-phase-to-target :node :pallet.phase/pre-settings for :supervisor-mycluster with 0 nodes INFO core - parallel-apply-phase :pallet.phase/pre-settings for :nimbus-mycluster INFO core - parallel-apply-phase-to-target :node :pallet.phase/pre-settings for :nimbus-mycluster with 0 nodes INFO core - parallel-apply-phase :settings for :zookeeper-mycluster INFO core - parallel-apply-phase-to-target :node :settings for :zookeeper-mycluster with 0 nodes INFO core - parallel-apply-phase :settings for :supervisor-mycluster INFO core - parallel-apply-phase-to-target :node :settings for :supervisor-mycluster with 0 nodes INFO core - parallel-apply-phase :settings for :nimbus-mycluster INFO core - parallel-apply-phase-to-target :node :settings for :nimbus-mycluster with 0 nodes INFO core - parallel-apply-phase :pallet.phase/post-settings for :zookeeper-mycluster INFO core - parallel-apply-phase-to-target :node :pallet.phase/post-settings for :zookeeper-mycluster with 0 nodes INFO core - parallel-apply-phase :pallet.phase/post-settings for :supervisor-mycluster INFO core - parallel-apply-phase-to-target :node :pallet.phase/post-settings for :supervisor-mycluster with 0 nodes INFO core - parallel-apply-phase :pallet.phase/post-settings for :nimbus-mycluster INFO core - parallel-apply-phase-to-target :node :pallet.phase/post-settings for :nimbus-mycluster with 0 nodes INFO core - parallel-apply-phase :pallet.phase/pre-configure for :zookeeper-mycluster INFO core - parallel-apply-phase-to-target :node :pallet.phase/pre-configure for :zookeeper-mycluster with 0 nodes INFO core - parallel-apply-phase :pallet.phase/pre-configure for :supervisor-mycluster INFO core - parallel-apply-phase-to-target :node :pallet.phase/pre-configure for :supervisor-mycluster with 0 nodes INFO core - parallel-apply-phase :pallet.phase/pre-configure for :nimbus-mycluster INFO core - parallel-apply-phase-to-target :node :pallet.phase/pre-configure for :nimbus-mycluster with 0 nodes INFO core - parallel-apply-phase :configure for :zookeeper-mycluster INFO core - parallel-apply-phase-to-target :node :configure for :zookeeper-mycluster with 0 nodes INFO core - parallel-apply-phase :configure for :supervisor-mycluster INFO core - parallel-apply-phase-to-target :node :configure for :supervisor-mycluster with 0 nodes INFO core - parallel-apply-phase :configure for :nimbus-mycluster INFO core - parallel-apply-phase-to-target :node :configure for :nimbus-mycluster with 0 nodes INFO core - parallel-apply-phase :pallet.phase/post-configure for :zookeeper-mycluster INFO core - parallel-apply-phase-to-target :node :pallet.phase/post-configure for :zookeeper-mycluster with 0 nodes INFO core - parallel-apply-phase :pallet.phase/post-configure for :supervisor-mycluster INFO core - parallel-apply-phase-to-target :node :pallet.phase/post-configure for :supervisor-mycluster with 0 nodes INFO core - parallel-apply-phase :pallet.phase/post-configure for :nimbus-mycluster INFO core - parallel-apply-phase-to-target :node :pallet.phase/post-configure for :nimbus-mycluster with 0 nodes DEBUG provision - Finished converge ERROR logging - Exception in thread "main" ERROR logging - java.lang.NullPointerException (form-init3395142206682442835.clj:1) ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5440) ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5415) ERROR logging - at clojure.lang.Compiler.load(Compiler.java:5857) ERROR logging - at clojure.lang.Compiler.loadFile(Compiler.java:5820) ERROR logging - at clojure.main$load_script.invoke(main.clj:221) ERROR logging - at clojure.main$init_opt.invoke(main.clj:226) ERROR logging - at clojure.main$initialize.invoke(main.clj:254) ERROR logging - at clojure.main$null_opt.invoke(main.clj:279) ERROR logging - at clojure.main$main.doInvoke(main.clj:354) ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:422) ERROR logging - at clojure.lang.Var.invoke(Var.java:369) ERROR logging - at clojure.lang.AFn.applyToHelper(AFn.java:165) ERROR logging - at clojure.lang.Var.applyTo(Var.java:482) ERROR logging - at clojure.main.main(main.java:37) ERROR logging - Caused by: java.lang.NullPointerException ERROR logging - at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:263) ERROR logging - at backtype.storm.security$authorize_group.invoke(security.clj:111) ERROR logging - at backtype.storm.provision$start_with_nodes_BANG_.invoke(provision.clj:77) ERROR logging - at backtype.storm.provision$start_BANG_.invoke(provision.clj:91) ERROR logging - at backtype.storm.provision$_main$fn__8422.invoke(provision.clj:144) ERROR logging - at backtype.storm.provision$_main.doInvoke(provision.clj:130) ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:552) ERROR logging - at clojure.lang.Var.invoke(Var.java:390) ERROR logging - at user$eval5$fn__7.invoke(form-init3395142206682442835.clj:1) ERROR logging - at user$eval5.invoke(form-init3395142206682442835.clj:1) ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5424) ERROR logging - ... 13 more

storm-deploy needs to point to the apache storm repo

Storm has moved to apache storm. See https://github.com/nathanmarz/storm/issues/772

At a bare minimum the storm crate needs to be updated here to point to the new repo.

public-key-path in ~/.pallet/config.clj seems to be ignored

I have ssh keys that may be named a little non standard (e.g. key.pem and key.pub) and when I setup my ~/.pallet/config.clj with the full path to these keys there was a FileNotFoundException because it could not find "key.pem.pub". It seems that the public-key-path is not used and the private key name + ".pub" is used instead. I was able to get this working by creating a soft link from key.pub to key.pem.pub, but this seems to be a bug. I am not sure if this is a bug in storm-deploy, pallet, or jclouds...

`lein deploy-storm` stalls

The lein deploy-storm ... command "stalls" towards the end.

After showing the following messages, it isn't doing much. I don't know if I should kill it.

Is this a bug in lein deploy-storm?

lein deploy-storm --start --name my-cluster --release 0.8.2

...

DEBUG compute - >> listing node details matching(ALWAYS_TRUE)
DEBUG compute - << list(5)
DEBUG compute - >> listing node details matching(ALWAYS_TRUE)
DEBUG compute - << list(5)
DEBUG compute - >> listing node details matching(ALWAYS_TRUE)
DEBUG compute - << list(5)
DEBUG compute - >> listing node details matching(ALWAYS_TRUE)
DEBUG compute - << list(5)
INFO  provision - Attaching Complete.
INFO  provision - Provisioning Complete.
DEBUG compute - >> listing node details matching(ALWAYS_TRUE)
DEBUG compute - << list(5)
INFO  provision - TAG:     zookeeper-my-cluster
INFO  provision - PUBLIC:  23.99.9.999
INFO  provision - PRIVATE: 10.99.999.9
DEBUG compute - >> listing node details matching(ALWAYS_TRUE)
DEBUG compute - << list(5)
INFO  provision - TAG:     nimbus-my-cluster
INFO  provision - PUBLIC:  54.999.9.999
INFO  provision - PRIVATE: 10.999.999.99
DEBUG compute - >> listing node details matching(ALWAYS_TRUE)
DEBUG compute - << list(5)
INFO  provision - TAG:     supervisor-my-cluster
INFO  provision - PUBLIC:  184.99.999.999, 23.99.999.99
INFO  provision - PRIVATE: 10.99.999.999, 10.999.999.999

On Linux must have a null passphrase on the keys

See here for more details: https://groups.google.com/group/storm-user/browse_thread/thread/e7ccf0c0f09ab342

failed with code 500

I have tried many different setups with older JDK 6, creating my own PEM or using the AWS PEM and creating a .pub. For some reason when I try to deploy I get this,

INFO execute - Output:
/home/stormdb/.ssh/pscinternal.pem.pub

DEBUG execute - out
=> "/home/stormdb/.ssh/pscinternal.pem.pub\n"
INFO execute - Output:
/home/stormdb/.ssh/pscinternal.pem

DEBUG execute - out
=> "/home/stormdb/.ssh/pscinternal.pem\n"
INFO execute - Output:
storm

DEBUG execute - out
=> "storm\n"
INFO execute - Output:
/home/stormdb/.ssh/pscinternal.pem.pub

DEBUG execute - out
=> "/home/stormdb/.ssh/pscinternal.pem.pub\n"
INFO execute - Output:
/home/stormdb/.ssh/pscinternal.pem

DEBUG execute - out
=> "/home/stormdb/.ssh/pscinternal.pem\n"
INFO execute - Output:
/home/stormdb/.ssh/pscinternal.pem.pub

DEBUG execute - out
=> "/home/stormdb/.ssh/pscinternal.pem.pub\n"
INFO execute - Output:
/home/stormdb/.ssh/pscinternal.pem

DEBUG execute - out
=> "/home/stormdb/.ssh/pscinternal.pem\n"
DEBUG jclouds - Found jclouds sshj driver
DEBUG jclouds - extensions (:log4j :slf4j :sshj)
DEBUG jclouds - options [:jclouds.regions "us-east-1" :blobstore-provider "aws-s3"]
ERROR logging - Exception in thread "main"
ERROR logging - com.google.inject.CreationException: Guice creation errors:

org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot be used as a key; It is not fully specified.

1 error (form-init1617305676035910732.clj:1)
ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5440)
ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5415)
ERROR logging - at clojure.lang.Compiler.load(Compiler.java:5857)
ERROR logging - at clojure.lang.Compiler.loadFile(Compiler.java:5820)
ERROR logging - at clojure.main$load_script.invoke(main.clj:221)
ERROR logging - at clojure.main$init_opt.invoke(main.clj:226)
ERROR logging - at clojure.main$initialize.invoke(main.clj:254)
ERROR logging - at clojure.main$null_opt.invoke(main.clj:279)
ERROR logging - at clojure.main$main.doInvoke(main.clj:354)
ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:422)
ERROR logging - at clojure.lang.Var.invoke(Var.java:369)
ERROR logging - at clojure.lang.AFn.applyToHelper(AFn.java:165)
ERROR logging - at clojure.lang.Var.applyTo(Var.java:482)
ERROR logging - at clojure.main.main(main.java:37)
ERROR logging - Caused by: com.google.inject.CreationException: Guice creation errors:

org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot be used as a key; It is not fully specified.

1 error
ERROR logging - at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435)
ERROR logging - at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:154)
ERROR logging - at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106)
ERROR logging - at com.google.inject.Guice.createInjector(Guice.java:95)
ERROR logging - at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:324)
ERROR logging - at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:262)
ERROR logging - at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:524)
ERROR logging - at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:504)
ERROR logging - at org.jclouds.compute2$compute_service.doInvoke(compute2.clj:92)
ERROR logging - at clojure.lang.RestFn.applyTo(RestFn.java:147)
ERROR logging - at clojure.core$apply.doInvoke(core.clj:548)
ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:562)
ERROR logging - at pallet.compute.jclouds$eval5952$fn__5954.invoke(jclouds.clj:720)
ERROR logging - at clojure.lang.MultiFn.invoke(MultiFn.java:167)
ERROR logging - at pallet.compute$compute_service.doInvoke(compute.clj:36)
ERROR logging - at clojure.lang.RestFn.applyTo(RestFn.java:140)
ERROR logging - at clojure.core$apply.invoke(core.clj:542)
ERROR logging - at pallet.configure$compute_service_from_map.invoke(configure.clj:176)
ERROR logging - at clojure.lang.Var.invoke(Var.java:365)
ERROR logging - at clojure.lang.AFn.applyToHelper(AFn.java:163)
ERROR logging - at clojure.lang.Var.applyTo(Var.java:482)
ERROR logging - at clojure.core$apply.invoke(core.clj:540)
ERROR logging - at pallet.compute$compute_service_from_map.doInvoke(compute.clj:185)
ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:409)
ERROR logging - at backtype.storm.provision$mk_aws.invoke(provision.clj:120)
ERROR logging - at backtype.storm.provision$_main.doInvoke(provision.clj:123)
ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:483)
ERROR logging - at clojure.lang.Var.invoke(Var.java:381)
ERROR logging - at user$eval5$fn__7.invoke(form-init1617305676035910732.clj:1)
ERROR logging - at user$eval5.invoke(form-init1617305676035910732.clj:1)
ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5424)
ERROR logging - ... 13 more

Anyone else experiencing this issue?

JDK 1.7.0_51 breaks `deploy-storm` (JClouds/Guice/Guava)

JDK 1.7.0_u51 breaks lein deploy-storm. The issue resides in jclouds/guice/guava (see: Upgrading JDK from 1.7.0_45 to 1.7.0_51 causes problems when searching for interface implementations). You will get errors that look like:

org.jclouds.rest.config.SyncToAsyncHttpApiProvider<org.jclouds.rest.HttpClient, A> cannot be used as a key; It is not fully specified.

This has been fixed in versions 1.8.0/1.7.1/1.6.2 of jclouds, however it looks like there were some api changes incompatible with storm-deploy's current use of jclouds.

PS - if you are a hitting this bug, an interim workaround is to downgrade your JDK to 1.7.0_u45. This worked for me and seems to have worked for others.

pip install required packages on worker machines

Is there any way to do something like a pip install on the worked machines to get required python packages? Or, is the way to do this to use the machine running the storm-deploy to install / setup anything and everything under multilang/resources that the workers will need?

Cannot connect to the Storm UI with Storm 0.9.x

I have encountered many issues attempting to deploy storm to EC2. I have tried several different branches with varying failing results. My last attempt was without specifying a branch (apache-storm-0.9.1-incubating-SNAPSHOT). It appears that the UI daemon is running on the nimbus although it appears that port 8080 is not open for communications. According to the running processes, nimbus appears to be fine but it is difficult to verify without the UI. Additionally, it appears that only one of the two supervisor machines actually is running the daemon.

I noticed 6 "java.lang.ClassNotFoundException: backtype.storm.LocalDRPC" being thrown during the build process, but it really does not seem like this would be the root cause. Please see my commands and config which may help determine whether or not I have made some error.

Lein commands:
lein deploy-storm --start --name test
lein deploy-storm --attach --name test

.pallet/config.clj:
(defpallet
:services
{
:default {
:blobstore-provider "aws-s3"
:provider "aws-ec2"
:environment {:user {:username "storm" ; this must be "storm"
:private-key-path "/.ssh/aws_rsa"
:public-key-path "/.ssh/aws_rsa.pub"}
:aws-user-id "xxxx-xxxx-xxxx"}
:identity "xxxxxxxxxxxxxxxxxxxxx"
:credential "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
:jclouds.regions "us-west-2"
}
})

conf/clusters.yaml:
nimbus.image: "us-west-2/ami-xxxxxxxx" #64-bit ubuntu
nimbus.hardware: "t1.micro"
supervisor.count: 2
supervisor.image: "us-west-2/ami-xxxxxxxx" #64-bit ubuntu
supervisor.hardware: "t1.micro"
zookeeper.count: 1
zookeeper.image: "us-west-2/ami-xxxxxxxx" #64-bit ubuntu
zookeeper.hardware: "t1.micro"

conf/storm.yaml
storm.local.dir: "/mnt/storm"

storm-deploy exception

I've tried running storm-deploy a few times and it is consistently failing both from my laptop (osx) and from an ec2 rhel 6.4 instance. The exception occurs directly after storm-deploy appears to be starting ganglia on all the nodes. The last few messages before the exception are as follows:

DEBUG core - p-f-s server environment null
DEBUG core - p-f-s server environment null
INFO core - parallel-apply-phase :pallet.phase/post-exec for :supervisor-stormClusterX
INFO core - parallel-apply-phase-to-target :node :pallet.phase/post-exec for :supervisor-stormClusterX with 2 nodes
DEBUG core - apply-phase-to-node: phase :pallet.phase/post-exec group :supervisor-stormClusterX target 54.226.164.194
DEBUG core - apply-phase-to-node: phase :pallet.phase/post-exec group :supervisor-stormClusterX target 54.224.119.37
INFO execute - execute-with-ssh on supervisor-stormClusterX "54.226.164.194"
INFO execute - execute-with-ssh on supervisor-stormClusterX "54.224.119.37"
INFO execute - Admin user storm /Users/benmackenzie/.ssh-aws/BenMac.pem /Users/benmackenzie/.ssh-aws/BenMac.pem.pub
INFO execute - Admin user storm /Users/benmackenzie/.ssh-aws/BenMac.pem /Users/benmackenzie/.ssh-aws/BenMac.pem.pub
DEBUG provision - Finished post-configure and exec phases
INFO provision - Attaching to Available Cluster...
DEBUG compute - >> listing node details matching(ALWAYS_TRUE)
DEBUG compute - << list(10)
DEBUG compute - >> listing node details matching(ALWAYS_TRUE)
DEBUG compute - << list(10)
DEBUG compute - >> listing node details matching(ALWAYS_TRUE)
DEBUG compute - << list(10)
DEBUG compute - >> listing node details matching(ALWAYS_TRUE)
DEBUG compute - << list(10)
ERROR logging - Exception in thread "main"
ERROR logging - java.lang.RuntimeException: request: POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1; cause: java.lang.NullPointerException (NO_SOURCE_FILE:1)

**config.clj is as follows:

(defpallet
:services
{
:default {
:blobstore-provider "aws-s3"
:provider "aws-ec2"
:environment {:user {:username "storm" ; this must be "storm"
:private-key-path "/.ssh-aws/BenMac.pem"
:public-key-path "/.ssh-aws/BenMac.pem.pub"}
:aws-user-id "xxxxxx"}
:identity "xxxxxxx"
:credential "xxxxxxx"
:jclouds.regions "us-east-1"
}
})

any ideas?

With the example AMIs, doesn't install the php / apache module, and thus the ganglia Web doesn't work

I've solved it with the following code change:

--- a/src/clj/backtype/storm/crate/ganglia.clj
+++ b/src/clj/backtype/storm/crate/ganglia.clj
@@ -17,7 +17,7 @@
[session]
(-> session
(package/packages

  :aptitude ["rrdtool" "librrds-perl" "librrd-dev" "php5-gd"

  :aptitude ["rrdtool" "librrds-perl" "librrd-dev" "php5-gd" "libapache2-mod-php5"
           "ganglia-monitor" "ganglia-webfrontend" "gmetad"])

(file/symbolic-link
"/usr/share/ganglia-webfrontend" "/var/www/ganglia")))

Security group does not exist

This used to work a week or two ago. Now I'm running against 237270c and get this error way at the end.

My security groups do not end with "#us-east-1" in the UI; they just look like: "jclouds#nimbus-my-cluster"

DEBUG provision - Finished converge
ERROR logging - Exception in thread "main"
ERROR logging - org.jclouds.rest.ResourceNotFoundException: The security group 'jclouds#nimbus-my-cluster#us-east-1' does not exist (NO_SOURCE_FILE:1)
ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5440)
ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5415)
ERROR logging - at clojure.lang.Compiler.eval(Compiler.java:5391)
ERROR logging - at clojure.core$eval.invoke(core.clj:2382)
ERROR logging - at clojure.main$eval_opt.invoke(main.clj:235)
ERROR logging - at clojure.main$initialize.invoke(main.clj:254)
ERROR logging - at clojure.main$null_opt.invoke(main.clj:279)
ERROR logging - at clojure.main$main.doInvoke(main.clj:354)
ERROR logging - at clojure.lang.RestFn.invoke(RestFn.java:422)
ERROR logging - at clojure.lang.Var.invoke(Var.java:369)
ERROR logging - at clojure.lang.AFn.applyToHelper(AFn.java:165)
ERROR logging - at clojure.lang.Var.applyTo(Var.java:482)
ERROR logging - at clojure.main.main(main.java:37)
ERROR logging - Caused by: org.jclouds.rest.ResourceNotFoundException: The security group 'jclouds#nimbus-my-cluster#us-east-1' does not exist
ERROR logging - at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:109)
ERROR logging - at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:92)
ERROR logging - at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:69)
ERROR logging - at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.shouldContinue(BaseHttpCommandExecutorService.java:197)

storm-deploy for "new generation" ec2 hardware

I'm looking to deploy storm (and zookeeper) into our VPC with the new i2 and c3 hardware that has recently come available to see if it makes sense for our setup.

I know there may be some issues with VPC and having to set up an IPSEC VPN, and I'm fully okay with that, however I cannot find a way (since I'm new to leiningen and storm-deploy) to get these new generation machines up and running.

When I try to deploy, I obviously get these errors:

DEBUG compute - >> searching params({imageId=us-east-1/ami-xxxxxxxx, hardwareId=c3.large})
DEBUG compute - >> searching params({imageId=us-east-1/ami-xxxxxxxx, hardwareId=c3.large})
WARN compute - hardware ids that didn't match: [cc1.4xlarge, cg1.4xlarge, cc2.8xlarge, t1.micro, c1.medium, c1.xlarge, m1.large, m1.small, m1.medium, m1.xlarge, m2.xlarge, m2.2xlarge, m2.4xlarge, hi1.4xlarge]
java.util.NoSuchElementException: hardwareId(c3.large) not found

Set environment variable on executor

I am using python for my spout and bolts. I have a dependency which is a swig'd wrapper around a C++ program. I have been successful getting maven to run my setup.py and install the dependency, but I have not figured out how to get the PYTHONPATH set on the executor nodes.

I could hard code the path to the egg, but the egg name changes depending on the os / architecture, so I don't really want to do that.

Is there a way to set an environment variable, like PYTHONPATH before submitting the topology that will be picked up by the executors?

Local mode ?

Hi,
Is it possible to use storm-deploy to deploy on local ubuntu machines instead of EC2?

deployment fails when using a branch named with the format "apache-0.9.1"

It seems like the code is trying to parse the version number:

ERROR logging - Caused by: java.lang.NumberFormatException: For input string: "apache-0"
ERROR logging - at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
ERROR logging - at java.lang.Integer.parseInt(Integer.java:492)
ERROR logging - at java.lang.Integer.parseInt(Integer.java:527)
ERROR logging - at backtype.storm.branch$parse_branch$fn__6476.invoke(branch.clj:4)
ERROR logging - at clojure.core$map$fn__3695.invoke(core.clj:2096)
ERROR logging - at clojure.lang.LazySeq.sval(LazySeq.java:42)

nathanmarz / storm-deploy Goto Github PK

storm-deploy's People

Stargazers

Watchers

Forkers

storm-deploy's Issues

supervisor.spot.price: 1.60

supervisor.spot.price: 1.60

Recommend Projects

Recommend Topics

Recommend Org