Scrape content from Orange forums. Made with scala-scraper.

Because it uses Scala 2.11, a version of Apache Spark supporting this version of Scala has to be built. To do this, first download Spark source code.

Then make pom.xml use Scala 2.11:

./dev/change-scala-version.sh 2.11

Build Spark with Maven:

./build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Dscala-2.11 -DskipTests clean package

Use ml.ScrapeMessages as:

./bin/spark-submit --class ml.ScrapeMessages scraper-assembly.jar

Messages are then stored in $SPARK_DIR/ml/data in partitioned files: part-00000, part-00001, .... Merge them into a single text file to import it in Excel and save it as a csv file. This file alread exists in ml at the root of this directory.

Run ml.Train with the csv file to save a trained model:

./bin/spark-submit --class ml.Train scraper-assembly.jar <csv training file> <trained model save folder>

Use sbt assembly to compile it and run:

./bin/spark-submit --class Scraper scraper-assembly.jar

Deploy the Spark - Cassandra cluster with t3g7/deployer.

Then get Swarm nodes IPs with docker-machine ip swarm-node-{1,2,3} and replace "Swarm node 1 IP,Swarm node 2 IP,Swarm node 3 IP" with them as spark.cassandra.connection.host in Scraper.scala.

Copy the jar in t3g7/deployer/data and run it:

docker exec -it swarm-master/master /usr/local/spark/bin/spark-submit --class Scraper --master spark://master:7077 /usr/local/spark/data/scraper-assembly-$VERSION.jar