storm jar uber-dataingest-1.0.0.jar com.xavient.storm.dataingest.DataIngestionTopology about data-ingestion-platform HOT 8 OPEN

Hi ,
Thanks for writing me back ,However I have resolved the issue and I have one question ,after I submit the jar then I need to do Open the UI for the application by visiting the URL "http://tomcat-server:port/DataIngestGUI/UI.jsp",I am not finding the UI.jsp file ,could you please help me in this ,

One question If I directly put the csv's JSON's and XML directly into the KAFKA_TOPIC, will the storm spout process it to the next bolt,Kindly let me know.

Thanks
Prosenjit

from data-ingestion-platform.

Greetings.... will look into it on Tuesday..

from data-ingestion-platform.

Hi ,

I have resolved the issue however when I pass messages to Kafka_topic like
bin/kafka-console-producer.sh --broker-list sandbox-hdp.hortonworks.com:6667 --topic kafka_topic
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
{"id":"XIND10000","author":"Jack White","title":"Hadoop Made Easy","genre":"Programming","price":"50000","publish_date":"2001-09-10","description":"ebook"}
{"id":"XIND10002","author":"Steven Berg","title":"Apache Source","genre":"Programming","price":"45000","publish_date":"2001-10-10","description":"ebook"}

and I receive the messages in consumer as well
but when I see them in storm topology ,I get the below error
ava.lang.NullPointerException: topic cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:226) at org.apache.storm.kafka.DynamicBrokersReader.(DynamicBrokersReader.java:55) at org.apache.storm.kafka.trident.ZkBrokerReader.(ZkBrokerReader.java:43) at org.apache.storm.kafka.KafkaUtils.makeBrokerReader(KafkaUtils.java:58) at org.apache.storm.kafka.KafkaSpout.open(KafkaSpout.java:77) at org.apache.storm.daemon.executor$fn__10182$fn__10197.invoke(executor.clj:603) at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:482) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:748)

Could you please help me with this ,waiting for your kind reply ,if you want we can have a session remote as well.

Thanks
Prosenjit

from data-ingestion-platform.

Hi ,

As agreed can we please look into it , had resolved the above error
but getting the below error java.lang.ExceptionInInitializerError at kafka.metrics.KafkaMetricsGroup$class.newTimer(KafkaMetricsGroup.scala:85) at kafka.consumer.FetchRequestAndResponseMetrics.newTimer(FetchRequestAndResponseStats.scala:26) at kafka.consumer.FetchRequestAndResponseMetrics.<init>(FetchRequestAndResponseStats.scala:35) at kafka.consumer.FetchRequestAndResponseStats.<init>(FetchRequestAndResponseStats.scala:47) at kafka.consumer.FetchRequestAndResponseStatsRegistry$$anonfun$2.apply(FetchRequestAndResponseStats.scala:60) at kafka.consumer.FetchRequestAndResponseStatsRegistry$$anonfun$2.apply(FetchRequestAndResponseStats.scala:60) at kafka.utils.Pool.getAndMaybePut(Pool.scala:59) at kafka.consumer.FetchRequestAndResponseStatsRegistry$.getFetchRequestAndResponseStats(FetchRequestAndResponseStats.scala:64) at kafka.consumer.SimpleConsumer.<init>(SimpleConsumer.scala:44) at kafka.javaapi.consumer.SimpleConsumer.<init>(SimpleConsumer.scala:34) at org.apache.storm.kafka.DynamicPartitionConnections.register(DynamicPartitionConnections.java:60) at org.apache.storm.kafka.PartitionManager.<init>(PartitionManager.java:70) at org.apache.storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98) at org.apache.storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69) at org.apache.storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:129) at org.apache.storm.daemon.executor$fn__10182$fn__10197$fn__10228.invoke(executor.clj:647) at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:66) at java.lang.Runtime.addShutdownHook(Runtime.java:211) at com.yammer.metrics.Metrics.<clinit>(Metrics.java:21) ... 19 more

Can you please help today is tuesday.

Thanks
Prosenjit

from data-ingestion-platform.

Hi ,

Can someone please look into the error .

Thanks
Prosenjit

from data-ingestion-platform.

@BigData-Engineer FYI.,

from data-ingestion-platform.

This seems to be versioning issue. As you are using newer version of HDP please change the versions of storm, Kafka etc in pom file. Rebuild and run.

from data-ingestion-platform.

@mohnkhan @BigData-Developer @rskumar @nsabharwal I am changed the POM.xml as per the version I am using in Ambari ,please find the pom.xml and the screenshot with the component versions attached .

I am attaching both kindly let me know where I am going wrong.

KIndly help me here .

Thanks
Prosenjit
Components Version as defined in Ambari Services.docx
pom.txt

from data-ingestion-platform.