First time I wanted to install and learn spark I just said to myself I'll use windows locally, and skip using an Ubuntu VM for this. Turns out installing Spark on windows ain't that easy, hence why I made this script.
This is a powershell script used to install Spark and everything else (so it will actually work). This script also sets the necessary environment variables on the User-level.
This was made for development purposes, so I don't recommend using it for production
It will install the following (if the command has not been found, don't worry, it will ask before continueing):
- chocolatey: A package manager for windows, used to install dependencies
- python2: Installed through chocolatey
- Java 1.8.0 (jdk8, jre8): Installed through chocolatey
- Scala 2.12.1: Installed from the official installer package
- GnuWin: GNU Coreutils for windows (used for the tar & gzip commands); installed through chocolatey
- Spark 2.1.0 with Hadoop 2.7: binaries downloaded directly from the official website
- winutils.exe: Required by spark to work (it's added in the
SPARK_HOME/bin
dir)
Actually, it's easy as pie ๐ช. Turn on a powershell console with Administrator privileges and navigate to your script file, then:
PS> ./installer.ps1
At the moment there is one command argument for this script:
-Where "C:\"
the path were spark will be installed. In this example you will find spark in the dirC:\spark
- Hive needs a temporary scratch dir (defualt is
/tmp/hive
) with write permissions - Because of how the file system works on Windows this path is relative the current root you run the command (so if you're on C: drive, the folder will be available at
C:\tmp\hive
)
- Even though I'm using chmod to change the dir permissions spark may still fail to boot up complaining with the error
The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw-
. Your Spark installation will not work until this is solved - Scala not detected even after we install it & refreshenv (although it is installed); Opening another session and re-executing the script might fix the path var
- If you happen to run into any issue at all, don't hesitate to file a issue, here on Github ๐
- There are quite a lot of improvements to be made, please check the issue tracker or file your own improvement issue