soscler / spark-boot-camp Goto Github PK

First, create a user-defined bridge network to allow the containers to communicate with each other: sudo docker network create spark-network.

Start the Spark master: sudo docker run -d --rm --name spark-master -p 4040:4040 -p 8080-8081:8080-8081 -p 7077:7077 --network spark-network gettyimages/spark bin/spark-class org.apache.spark.deploy.master.Master.

Finally, start one or more workers: sudo docker run -d --rm --name spark-worker1 --network spark-network gettyimages/spark bin/spark-class org.apache.spark.deploy.worker.Worker spark://spark-master:7077 (change the name of the container and the allocated resources when running multiple Spark workers).