This repository contains instructions to set up and run the Apache SystemML API in a Spark Shell with the IBM Analytics Engine to complete computations in Scala.
@MadisonJMyers In general, this is great! It gets the user using Spark, SystemML, and IAE. Here are some thoughts as I went through it:
It's subject to preference, but I would lowercase "Shell", and slightly alter the wording from "on a Spark Shell" to "in a Spark shell".
I might alter the wording of "You can log into Spark Shell, load SystemML on the shell, ..." to "You start a Spark shell, import SystemML, ....".
I would remove the assumptions for SystemML section with the brew install commands since IAE will already have Spark installed.
I'm not sure opening the Ambari webpage necessary. I think it should be fine to ssh in directly.
It's not yet clear where the ssh command is from the readme -- I found it under "Connection Details".
Under the "Our next step is to parallelize the information, read in two matrices as RDDs, getting the sum of the first, the sum of the second and a message." section, can you remove the scala> prefixes so that the user can simply copy & paste?
In that same section, there is an issue with the last quotation mark (") in the lines, such as in val data1 = sc.parallelize(Array("1.0,2.0", "3.0,4.0โ)) in which that final " has been encoded as โ, and so it doesn't work at the command line. Can you update it to a " mark? There are a few lines in the section with that problem.
Also, for all of the code sections, can you wrap them up as code blocks (``` marks on the lines before and after each block -> https://guides.github.com/features/mastering-markdown/#examples)? That will probably help with the encoding and formatting issues, and make it easier to copy & paste.
In the "Now print your script info. You should see:" section, can you split this into the code to run (println(script.info)) and the expected output the user should see? Code blocks will help a lot as well.
When I run the code in "Execute your script and get your results!", I get an error. I think it is due to the formatting of val s = ... in "Our next step is to parallelize the information, read in two matrices as RDDs, getting the sum of the first, the sum of the second and a message.". The vertical bars (|) shouldn't be present.
The line val script = dmlFromUrl(scriptUrl).in("A", habermanRDD, habermanMetadata).in("K", typesRDD, typesMetadata).in("$CONSOLE_OUTPUT", true) results in an error due to typesMetadata not being available.
Feature [1]: Scale ... and the next few sections are bolded by accident.