hdinsight / hdinsight-storm-examples Goto Github PK

SubmitStormTopology.ps1 does not work

I executed SubmitStormTopology.ps1 script and I got an error:

PS D:\GIT\hdinsight-storm-examples\scripts\storm> .\SubmitStormTopology.ps1 -ClusterUrl "https://server_storm_cluster.azurehdinsight.net" -ClusterUsername admin -ClusterPassword "my_password" -JarPath "D:\apache-storm-0.9.3\examples\storm-starter\target\storm-starter-0.9.3-jar-with-dependencies.jar" -ClassName "storm.starter.WordCountTopology"

This is the output:

2015-09-04T00:21:47.9364419-04:00 [ERROR] [SubmitStormTopology.ps1(ln:65)] - Exception encountered while invoking the [POST] rest method at: https://server_storm_cluster.azurehdinsight.net/StormDashboard/SubmitWasbJar
Invoke-RestMethod : Server Error
401 - Unauthorized: Access is denied due to invalid credentials.
You do not have permission to view this directory or page using the credentials that you supplied.
At D:\GIT\hdinsight-storm-examples\scripts\storm\SubmitStormTopology.ps1:51 char:17

• $response = Invoke-RestMethod -Uri $clusterSubmitJarUri.AbsoluteUri -Method  ...
  

• - CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  - FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
  
  

2015-09-04T00:21:47.9964437-04:00 [ERROR] [SubmitStormTopology.ps1(ln:70)] - Topology submission encountered an error, please check logs for error information.
Topology submission encountered an error, please check logs for error information.
At D:\GIT\hdinsight-storm-examples\scripts\storm\SubmitStormTopology.ps1:71 char:5

• throw "Topology submission encountered an error, please check logs for error ...
  

• - CategoryInfo          : OperationStopped: (Topology submis...or information.:String) [], RuntimeException
  - FullyQualifiedErrorId : Topology submission encountered an error, please check logs for error information.
  
  

The user/password is the same that I provided when I created Storm cluster in Azure HDInsight.

I'm able to login https://server_storm_cluster.azurehdinsight.net with the same credentials.

A user may get a HttpStatus 404 while trying to submit a topology if the topology package size is larger than 30 MB.

This is happening due to default IIS upload limit restrictions.
Refer: http://stackoverflow.com/questions/4548305/maximum-value-of-maxrequestlength

The issue will be fixed in next update of HDInsight Storm.

Workaround:
You can workaround this problem by modifying its Web.config.
a. Step 1: Remote Desktop into your cluster's headnode by enaling RDP and find out your active headnode via the Desktop Shortcut: Hadoop Service Availability (you may have to refresh the page)
b. Step 2: Go to C:\apps\dist\storm-\SCPAPI\SCPAPI and open Web.config for editing
c. Step 3: Update the value for httpRuntime maxRequestLength under system.Web from "51200" to "2097152"

    <system.web>
        <httpRuntime targetFramework="4.5" maxRequestLength="2097152" executionTimeout="300" />

d. Step 4: Add a requestLimits maxAllowedContentLength under security in system.WebServer

    <system.webServer>
        <security>
            <requestFiltering>
                <requestLimits maxAllowedContentLength="2147483648" />
            </requestFiltering>

e. The change is instant and you don't need to restart IIS. If you get HttpStatus 500 check the event logs if you did a mistake in configuration changes.

The Java library allows specifying complex types as message ids when emitting tuples form spout. At the very least, can the support for specifying string content in messageId be extended to the SCM.NET SDK?

I'm using Visual Studio 2015 Update 3 with Azure Data Lake tools for Visual Studio (2.2.2100.0v).
Created new storm application from File->New Project->Templates->Azure Data Lake->Storm (HDInsight).
Created new cluster with HDP 3.4 (Linux).
Checked nuget reference of Microsoft.SCP.Net.SDK is 0.10.0.6.
Right clicked project and asked to "Submit to Storm on HDInsight". Selected correct cluster name and clicked Submit.

Right after, it gets stuck with the following text in the output window (under Topology Submission). What is wrong with it.

Also is there any manual steps to do this without Visual Studio?

ScpC Start: 0 : ScpC Version: 1.0.6098.38783
ScpC Information: 0 : Executing Assembly Path: C:\Users\admin\Documents\Visual Studio 2015\Projects\StormSample2\packages\Microsoft.SCP.Net.SDK.0.10.0.6\tools
ScpC Information: 0 : Executing Assembly's Path Parent: C:\Users\admin\Documents\Visual Studio 2015\Projects\StormSample2\packages\Microsoft.SCP.Net.SDK.0.10.0.6
ScpC Information: 0 : Executing Assembly's Parent's Parent: C:\Users\admin\Documents\Visual Studio 2015\Projects\StormSample2\packages
ScpC Start: 0 : GenerateSpec
ScpC Information: 0 : Load configuration file: C:\Users\admin\Documents\Visual Studio 2015\Projects\StormSample2\StormSample2\bin\Debug\SCPHost.exe.config as default configuartion file.
ScpC Information: 0 : Found active TopologyDescriptor: "StormSample2.HelloWorld"
ScpC Information: 0 : Generating spec file "C:\Users\admin\Documents\Visual Studio 2015\Projects\StormSample2\StormSample2\bin\Debug\StormSample2.spec"
ScpC Information: 0 : Succeed to generate spec file "C:\Users\admin\Documents\Visual Studio 2015\Projects\StormSample2\StormSample2\bin\Debug\StormSample2.spec"
2/9/2017 7:04:38 PM Begin to generate zip package...
2/9/2017 7:04:54 PM ScpC Start: 0 : ScpC Version: 1.0.6098.38783
ScpC Information: 0 : Executing Assembly Path: C:\Users\admin\Documents\Visual Studio 2015\Projects\StormSample2\packages\Microsoft.SCP.Net.SDK.0.10.0.6\tools
ScpC Information: 0 : Executing Assembly's Path Parent: C:\Users\admin\Documents\Visual Studio 2015\Projects\StormSample2\packages\Microsoft.SCP.Net.SDK.0.10.0.6
ScpC Information: 0 : Executing Assembly's Parent's Parent: C:\Users\admin\Documents\Visual Studio 2015\Projects\StormSample2\packages
ScpC Information: 0 : Create temp folder: C:\Users\admin\AppData\Local\Temp\Temp-ScpC-package-636222440789034139 ...
ScpC Information: 0 : Copy C# files from "C:\Users\admin\Documents\Visual Studio 2015\Projects\StormSample2\StormSample2\bin\Debug" to resources folder ...

Hello; I'm currently working on an IoT project that's exploring EventHubs + HDInsight/Storm for processing large numbers of device telemetry events.

I've found the various "develop Storm topologies in C#" examples that refer to EventHubs and have experimented with the C# SDK, but can't find an example of a (custom) EventHub spout in C# that's able to work with multiple EventHub partitions.

It seems like using the id of the spout instance to determine which partition to read from is the way to go, but this information (context.getThisComponentId() in Java) doesn't seem to be present in the C# SDK.

Are there further samples available/planned using C# in this scenario? Adding Java to our current technology stack isn't feasible right now. (Also, is there a better forum for these kinds of discussions?)

Many thanks in advance!

Do you have any sample code where Storm writes out to an Azure blob, rather than HBase? We are having trouble getting any examples to work that do not write to HBase.

When using the evenhubspout there is no option to configure a certain consumergroup. However, this option is available in the Java implementation. It is not included in the constructor, but it is available as a property. Can this property be made available through SCP?

On top of that, the eventhubspout only returns the eventdata's message, the user-properties, partitionkey, etc. are not included in the tuple. However these are certainly transmitted. This is probably due to the Java implementation of the spout.

Suggest to open source the scp.net not only the sample

IotExample\build.bat

2017-01-19T07:52:30.4452615+13:00 [INFO] [buildJava.ps1(ln:44)] - Building Java
Project: C:\Users\russeda\Documents\Visual Studio 2015\Projects\hdinsight-storm-
examples\IotExample\iot
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building iot 1.0
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.562 s
[INFO] Finished at: 2017-01-19T07:52:32+13:00
[INFO] Final Memory: 9M/245M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project iot: Could not resolve dependencies for project com.microsoft.hdinsight.storm.examples:iot:jar:1.0: Failed to collectdependencies at org.apache.hadoop:hadoop-common:jar:3.0.0-SNAPSHOT: Failed to read artifact descriptor for org.apache.hadoop:hadoop-common:jar:3.0.0-SNAPSHOT: Could not find artifact org.apache.hadoop:hadoop-project-dist:pom:3.0.0-SNAPSHOT-> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

If I check my local repository hadoop-project-dist does not exist in org.apache.hadoop

In installMvnLib.ps1 line 53:

cmd /c "mvn -q org.apache.maven.plugins:maven-install-plugin:2.5.2:install-file -Dfile=""$libDir\hadoop\hadoop-project-dist-3.0.0-SNAPSHOT.pom"" ""-DpomFile=""$libDir\hadoop\hadoop-project-dist-3.0.0-SNAPSHOT.pom"""

There is a stray "" before -DpomFile. Removing this and rebuilding fixed the issue

Am trying to stand up the cluster using the PS prepare.ps1 script and am seeing the following error. Have tried multiple times. Any ideas?

2016-07-07T07:46:36.7815892-07:00 [INFO] [CreateStorageAccountARM.ps1(ln:54)] - Getting Storage Key for iot201607070742
2016-07-07T07:46:37.0940803-07:00 [INFO] [CreateStorageAccountARM.ps1(ln:59)] - Storage Account: iot201607070742 successfully created in Resource Group: iot-group-201607
070742 at Location: Central US. Time: 33.2957405 secs
Cannot get Storage Key
At C:\projects\hdinsight-storm-examples\scripts\azure\CreateAzureResources.ps1:207 char:5
+     throw "Cannot get Storage Key"
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (Cannot get Storage Key:String) [], RuntimeException
    + FullyQualifiedErrorId : Cannot get Storage Key

PS C:\projects\hdinsight-storm-examples\IotExample>

is posible créate a Trident topology wiht SCP .NET and C#? If yes, any sample?

Thanks

Lately I've been having a lot of issues running topologies. Two errors occur quite often, giving no indication as to what's wrong.

2015-08-12 08:58:08 b.s.d.executor [ERROR] 
java.lang.RuntimeException: com.microsoft.eventhubs.client.EventHubException: org.apache.qpid.amqp_1_0.client.ConnectionClosedException: Connection closed for unknown reason
    at com.microsoft.eventhubs.spout.EventHubSpout.open(EventHubSpout.java:156) ~[stormjar.jar:na]
    at backtype.storm.daemon.executor$fn__5064$fn__5079.invoke(executor.clj:542) ~[storm-core-0.9.3.2.2.7.1-0004.jar:0.9.3.2.2.7.1-0004]
    at backtype.storm.util$async_loop$fn__550.invoke(util.clj:463) ~[storm-core-0.9.3.2.2.7.1-0004.jar:0.9.3.2.2.7.1-0004]
    at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]
Caused by: com.microsoft.eventhubs.client.EventHubException: org.apache.qpid.amqp_1_0.client.ConnectionClosedException: Connection closed for unknown reason
    at com.microsoft.eventhubs.client.EventHubConsumerGroup.ensureSessionCreated(EventHubConsumerGroup.java:64) ~[stormjar.jar:na]
    at com.microsoft.eventhubs.client.EventHubConsumerGroup.createReceiver(EventHubConsumerGroup.java:39) ~[stormjar.jar:na]
    at com.microsoft.eventhubs.client.ResilientEventHubReceiver.initialize(ResilientEventHubReceiver.java:63) ~[stormjar.jar:na]
    at com.microsoft.eventhubs.spout.EventHubReceiverImpl.open(EventHubReceiverImpl.java:74) ~[stormjar.jar:na]
    at com.microsoft.eventhubs.spout.SimplePartitionManager.open(SimplePartitionManager.java:77) ~[stormjar.jar:na]
    at com.microsoft.eventhubs.spout.EventHubSpout.preparePartitions(EventHubSpout.java:134) ~[stormjar.jar:na]
    at com.microsoft.eventhubs.spout.EventHubSpout.open(EventHubSpout.java:153) ~[stormjar.jar:na]
    ... 4 common frames omitted
Caused by: org.apache.qpid.amqp_1_0.client.ConnectionClosedException: Connection closed for unknown reason
    at org.apache.qpid.amqp_1_0.client.Connection.checkNotClosed(Connection.java:338) ~[stormjar.jar:na]
    at org.apache.qpid.amqp_1_0.client.Connection.createSession(Connection.java:322) ~[stormjar.jar:na]
    at com.microsoft.eventhubs.client.EventHubConsumerGroup.ensureSessionCreated(EventHubConsumerGroup.java:61) ~[stormjar.jar:na]
    ... 10 common frames omitted

2015-08-12 08:59:44 b.s.d.executor [ERROR] 
java.lang.NullPointerException: null
    at org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint.getFirstFreeChannel(ConnectionEndpoint.java:327) ~[stormjar.jar:na]
    at org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint.createSession(ConnectionEndpoint.java:230) ~[stormjar.jar:na]
    at org.apache.qpid.amqp_1_0.client.Session.<init>(Session.java:58) ~[stormjar.jar:na]
    at org.apache.qpid.amqp_1_0.client.Connection.createSession(Connection.java:323) ~[stormjar.jar:na]
    at com.microsoft.eventhubs.client.EventHubConsumerGroup.ensureSessionCreated(EventHubConsumerGroup.java:61) ~[stormjar.jar:na]
    at com.microsoft.eventhubs.client.EventHubConsumerGroup.createReceiver(EventHubConsumerGroup.java:39) ~[stormjar.jar:na]
    at com.microsoft.eventhubs.client.ResilientEventHubReceiver.initialize(ResilientEventHubReceiver.java:63) ~[stormjar.jar:na]
    at com.microsoft.eventhubs.spout.EventHubReceiverImpl.open(EventHubReceiverImpl.java:74) ~[stormjar.jar:na]
    at com.microsoft.eventhubs.spout.SimplePartitionManager.open(SimplePartitionManager.java:77) ~[stormjar.jar:na]
    at com.microsoft.eventhubs.spout.EventHubSpout.preparePartitions(EventHubSpout.java:134) ~[stormjar.jar:na]
    at com.microsoft.eventhubs.spout.EventHubSpout.open(EventHubSpout.java:153) ~[stormjar.jar:na]
    at backtype.storm.daemon.executor$fn__5064$fn__5079.invoke(executor.clj:542) [storm-core-0.9.3.2.2.7.1-0004.jar:0.9.3.2.2.7.1-0004]
    at backtype.storm.util$async_loop$fn__550.invoke(util.clj:463) [storm-core-0.9.3.2.2.7.1-0004.jar:0.9.3.2.2.7.1-0004]
    at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]

Please update SCP.Net to use the lastest version of Newtonsoft.Json, or drop the restriction on version number. Adding external packages that depend on a newer version of Newtonsoft.Json is a hassle.

Workaround for now is to force the update of the package.

Cluster created before June 4th 2015 may exhibit an issue where a user can get HttpStatus 502 while submitting their topology.

The issue was with Azure OS Reboot deleting our SCPAPI website from IIS. The setup code for HDInsight cluster was updated with version 3.2.4.583 that now has a fix for the issue.

Please create newer Storm 3.2 clusters to get the fix.

I have an issue with running the Kafka Creationscripts. I always get the CommandNotFoundException.

• CategoryInfo : ObjectNotFound: (Switch-AzureMode:String) [UploadFileToStorage.ps1], CommandNotFoundException
  • FullyQualifiedErrorId : CommandNotFoundException,UploadFileToStorage.ps1

Any ideas?

Hi, more of a question, is there a way to see what Context.Logger outputs in topology running on Storm?