This is a distributed streaming application that reads covid19 data using Python, and pipes it to Kafka streams. Jobs are run using Yarn, Samza and Zookeeper on Hadoop 2.0

a) make sure JAVA_HOME is set in .bashrc.

echo $JAVA_HOME

b) Make sure you have write persmission in /home directory or wherever you are cloning the repo.

git clone https://github.com/Shivansh1805/KWHY.git

chmod +x /home/path/to/KWHY/deploy-script/grid
chmod +x /home/path/to/KWHY/deploy/bin/run-job.sh
chmod +x /home/path/to/KWHY/deploy/bin/run-class.sh

If you face permission related problem with any script, just make it executable by above mentioned commands.

mvn clean install

Install and Start yarn, zookeeper, kafka

deploy-script/grid bootstrap

To start/stop zookeeper,yarn,kafka by a single command

deploy-script/grid start all
deploy-script/grid stop all

To start/stop zookeeper,yarn,kafka separately

deploy-script/grid start/stop kafka
deploy-script/grid start/stop yarn
deploy-script/grid start/stop zookeeper

Note - If you face some error while starting kafka just remove the kafka-logs directory from /tmp/ and start it again.

Update path in both files present in src/resources directory

yarn.package.path=/home/path/to/KWHY/deploy/deploy.tar.gz

/home/path/to/KWHY/deploy/bin/run-job.sh --config-factory=org.apache.samza.config.factories.PropertiesConfigFactory --config-path=/home/path/to/KWHY/deploy/conf/covid-parser.properties

Open browser and go to local yarn resource manager: http://localhost:8088/cluster to check status

/home/path/to/KWHY/deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic district-data

/home/path/to/KWHY/deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic state-data

Pull data from covid19 API using python script and pipe it to kafka producer and bind to 'district-data' & 'state-data' topic Go to producer directory and run the below command.

python3 covid-stream.py 

If you encounter error due to impropoer yarn shut down, simply delete /tmp/kafka-logs directory

Before pushing the code

rm -rf deploy
mvn clean