Cartesian Product (CP) is a knowledge management platform that allows you to visualize data, explore deeper insight, and use your own data to help you answer your most pressing questions and contribute to clean energy development around the globe.

What really distinguishes CP is its backend system that gives users the power to contribute their own data and insights or download the current state of our ever growing open source datasets. By crowd sourcing our knowledge management system, we strive to convert the linear process of one-off data science consulting into an iterative cycle of community contribution and engagement, driving data-informed decision making.

Most of us are in the Advanced Data Engineering class at GalavanizeU Data Science Master's Program in San Francisco. We also have other members coming from the Galvanize Web Development Immersive program, but all of us are part of the Galvanize community.

We're doing this because climate change is one the paramount issues of our generation. Decision makers need to have open and easy access to data that comes together in a single place.

Class Project for DSCI-6009-SU17

So far: Kafka broker that ingests data from plume.io [work in progress]

Start Zookeeper. If you have installed zookeeper, start it, or run the command:
> bin/zookeeper-server-start.sh config/zookeeper.properties

Start Kafka with default configuration
> bin/kafka-server-start.sh config/server.properties

Create a Topic - 'plume_pollution'
> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 10 --topic plume_pollution

To show all content in a topic (and to check the Producer is working), have the command line consumer running: bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic plume_pollution --from-beginning

Package project
sbt assembly It will package compiled classes and its dependencies into a jar.

Run the Producer The Producer is a Scala file located at src\main\scala\KafkaBroker.Scala Once packaged in a JAR you can run the producer with the following arguments: scala cartesianproduct2-assembly-1.0.jar plume_pollution localhost:9092 48.85 2.294 1000 where:

• title: plume_pollution
• broker: localhost:9092
• latitude: 48.85
• longitude: 2.294
• sleepTime: 1000

For additional information about Kafka, Plume API or Plume token check the WIKI

Output If all is working, you should see the following output:

List topics

Install pip and python-dev:

sudo apt-get install python-pip python-dev

Install IPython. Since the machine will most likely be running Python 2.7, it is necessary to install an older version:

sudo pip install IPython=5.0

Install Jupyter:

sudo pip install jupyter

In order to run Jupyter with PySpark, the drivers need to be changed. Create a bash script called jupyter-pyspark with the following code:

#!/usr/bin/env bash

export PYSPARK_DRIVER_PYTHON=which jupyter

export PYSPARK_DRIVER_PYTHON_OPTS="notebook --NotebookApp.open_browser=False --NotebookApp.ip='*' --NotebookApp.port=8888"

dse pyspark

Change permissions on the file:

chmod a+x jupyter-pyspark

Move jupyter-pyspark to /usr/bin. Now running the jupyter-pyspark command from anywhere in the machine will begin a browserless Jupyter notebook running with a PySpark driver.

In order to access the notebook remotely, ports must be opened to the relevant IPs. This may be done using the Google Cloud networking dashboard.

Having a single build.sbt file is an issue if you want to develop multiple scala projects, as it becomes a dependency nightmare. Instead, we have a Build.scala file in /project which specifies the dependencies on a per-project level. This makes development much simpler.