Take in a list of web addresses, and download the files to a target location. Location can be either a local filesystem, or in HDFS.

There will be a compiled binary included in the repository, downloader, that can be used out of the box. However, if you wish to compile the package yourself, you will need to use the following steps.

1. Download Go and follow the installation instructions

2. Set up your install

3. Clone the downloader repository, and change into the directory

4. Compile the binary by running the following command:

   go build

5. The downloader binary will have been generated

If ran without any arguments, the downloader command will display the help message. The same as if it was ran with downloader -h, downloader --help, and downloader help.

Running downloader -v or downloader --version will display the version.

There are two subcommands:

• local - for when you want to download files into a local directory
• hdfs - for when you want to download files directly into HDFS

Both of the subcommands make use of the same flags:

There may be issues downloading files directly to HDFS in a kerberized environment. This is something that will be worked on, and updated accordingly.

For any issues that are found, please raise an issue on Github.