Read this in other languages: 日本語.
Visualize and analyze San Francisco crime incidents using a Jupyter Notebook, PixieDust, and PixieApps
Data Science Experience is now Watson Studio. Although some images in this code pattern may show the service as Data Science Experience, the steps and processes will still work.
In this Code Pattern we will use PixieDust running on IBM Watson Studio to analyze traffic calming and crime data from the City of San Francisco. Watson Studio is an interactive, collaborative, cloud-based environment where data scientists, developers, and others interested in data science can use tools (e.g., RStudio, Jupyter Notebooks, Spark, etc.) to collaborate, share, and gather insight from their data.
When the reader has completed this Code Pattern, they will understand how to:
- Use Jupyter Notebooks to load, visualize, and analyze data
- Run Notebooks in IBM Watson Studio
- Leverage PixieDust as a python notebook helper
- Build a dashboard using PixieApps
- Fetch data from City of San Francisco Open Data
- Create an interactive map with Mapbox GL
The intended audience for this Code Pattern is application developers and other stakeholders who wish to utilize the power of Data Science quickly and effectively.
- Load the provided notebook into the Watson Studio platform.
- DataSF Open Data crime info is loaded into the Jupyter Notebook.
- The notebook analyzes the crime info.
- You can interactively change charts and graphs.
- A PixieApp dashboard is created and can be interacted with.
- IBM Watson Studio: Analyze data using RStudio, Jupyter, and Python in a configured, collaborative environment that includes IBM value-adds, such as managed Spark.
-
Jupyter Notebooks: An open-source web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text.
-
PixieDust Python helper library for python notebooks
-
PixieApps: Python library used to write UI elements for analytics, and run them directly in a Jupyter notebook.
-
Mapbox GL: JavaScript library that uses WebGL to render interactive maps.
- Get a Mapbox Token for use in the notebook
Follow these steps to setup and run this Code Pattern. The steps are described in detail below.
- Sign up for the Watson Studio
- Create the Spark Service
- Create the notebook
- Run the notebook
- Analyze the results
- Save and Share
Sign up for IBM's Watson Studio. By creating a project in Watson Studio a free tier Object Storage
service will be created in your IBM Cloud account. Take note of your service names as you will need to select them in the following steps.
Note: When creating your Object Storage service, select the
Free
storage type in order to avoid having to pay an upgrade fee.
- In your project go to the
Settings
tab, scroll down toAssociated Services
and choose+ Add service
->Spark
- Either choose and
Existing
Spark service, or create aNew
one
- In Watson Studio, click on
Create notebook
to create a notebook. - Create a project if necessary, provisioning an object storage service if required.
- In the
Assets
tab, select theCreate notebook
option. - Select the
From URL
tab. - Enter a name for the notebook.
- Optionally, enter a description for the notebook.
- Enter this Notebook URL: https://raw.githubusercontent.com/IBM/pixiedust-traffic-analysis/master/notebooks/pixiedust-traffic-analysis.ipynb
- Select the Spark runtime you've associated with this project:
- Click the
Create
button.
NOTE: See the points in the notebook where you will have to enter your Mapbox Token to render the map.
When a notebook is executed, what is actually happening is that each code cell in the notebook is executed, in order, from top to bottom.
Each code cell is selectable and is preceded by a tag in the left margin. The tag
format is In [x]:
. Depending on the state of the notebook, the x
can be:
- A blank, this indicates that the cell has never been executed.
- A number, this number represents the relative order this code step was executed.
- A
*
, this indicates that the cell is currently executing.
There are several ways to execute the code cells in your notebook:
- One cell at a time.
- Select the cell, and then press the
Play
button in the toolbar.
- Select the cell, and then press the
- Batch mode, in sequential order.
- From the
Cell
menu bar, there are several options available. For example, you canRun All
cells in your notebook, or you canRun All Below
, that will start executing from the first cell under the currently selected cell, and then continue executing all cells that follow.
- From the
- At a scheduled time.
- Press the
Schedule
button located in the top right section of your notebook panel. Here you can schedule your notebook to be executed once at some future time, or repeatedly at your specified interval.
- Press the
After running each cell of the notebook, the results will display. When we use PixieDust display()
to create an interactive dataset, we are able to change the visualization using tables, graphs, and charts.
After running cell #3 display(accidents)
, we can see by clicking the Options
button that we are able to manipulate the keys and values for the fields used in the chart:
Following the instructions, we use DaysOfWeek and IncidntNum, but the user can change the keys and value to see how the chart will look with different inputs.
We use Spark SQL to isolate data to the Taraval district:
accidents.registerTempTable("accidents")
taraval = sqlContext.sql("SELECT * FROM accidents WHERE PdDistrict='TARAVAL'")
We then get an interactive map of the Taraval district:
With PixieApps, we can create a dashboard with map layers that can be used to visualize various datasets (i.e. Traffic Calming, Police Districts, and Crimes):
from pixiedust.display.app import *
@PixieApp
class SFDashboard():
def mainScreen(self):
return """
<div class="well">
<center><span style="font-size:x-large">Analyzing San Francisco Public Safety data with PixieDust</span></center>
<center><span style="font-size:large"><a href="https://datasf.org/opendata" target="new">https://datasf.org/opendata</a></span></center>
</div>
<div class="row">
<div class="form-group col-sm-2" style="padding-right:10px;">
<div><strong>Layers</strong></div>
{% for layer in this.layers %}
<div class="rendererOpt checkbox checkbox-primary">
<input type="checkbox" pd_refresh="map{{prefix}}" pd_script="self.toggleLayer({{loop.index0}})">
<label>{{layer["name"]}}</label>
</div>
{%endfor%}
</div>
<div class="form-group col-sm-10">
<div id="map{{prefix}}" pd_entity pd_options="{{this.formatOptions(this.mapJSONOptions)}}"/>
</div>
</div>
"""
<div id="map{{prefix}}" pd_entity pd_options="{{this.formatOptions(this.mapJSONOptions)}}"/>
pd_entity
: Tell PixieDust which dataset to work on.
pd_options
: Contains the PixieDust options for the map.
The best way to generate the pd_options
for a PixieDust visualization is to:
- Call
display()
on a new cell - Graphically select the options for your chart
- Select
View/Cell Toobar/Edit metadata
menu - Click on the
Edit Metadata
button and copy the PixieDust metadata
To conform to the pd_options
notation, we need to transform the PixieDust JSON metadata into an attribute string with the following format: “key1=value1;key2=value2;…”
To make it easier, we use a simple Python transform function:
def formatOptions(self, options):
return ';'.join(["{}={}".format(k,v) for (k, v) in iteritems(options)])
The formatOptions
is then invoked using JinJa2 notation from within the html:
pd_options = “{{this.formatOptions(this.mapJSONOptions)}}”
Note: setup is a special method that will be called automatically when the PixieApp is initialized.
A Mapbox token is included in the notebook, for now. To get your own visit Mapbox
def setup(self):
self.mapJSONOptions = {
"mapboxtoken": "pk.eyJ1IjoicmFqcnNpbmdoIiwiYSI6ImNqM2s4ZDg4djAwcGYyd3BwaGxwaDV3bWoifQ.d5Rklkdu5MeGAnXu1GMNYw",
"chartsize": "90",
"aggregation": "SUM",
"rowCount": "500",
"handlerId": "mapView",
"rendererId": "mapbox",
"valueFields": "IncidntNum",
"keyFields": "X,Y",
"basemap": "light-v9"
}
from pixiedust.display.app import *
from pixiedust.apps.mapboxBase import MapboxBase
@PixieApp
class SFDashboard(MapboxBase):
def setup(self):
...<snip>...
self.setLayers([
{
"name": "Traffic calming",
"url": "https://data.sfgov.org/api/geospatial/ddye-rism?method=export&format=GeoJSON",
"type": "symbol",
"layout": {
"icon-image": "police-15",
"icon-size": 1.5
}
},
...<snip>...
...<snip>...
{% for layer in this.layers %}
<div class="rendererOpt checkbox checkbox-primary">
<input type="checkbox" pd_refresh="map{{prefix}}" pd_script="self.toggleLayer({{loop.index0}})">
<label>{{layer["name"]}}</label>
</div>
{%endfor%}
...<snip>...
The user can now select layers and the map will dynamically add or remove them.
Under the File
menu, there are several ways to save your notebook:
Save
will simply save the current state of your notebook, without any version information.Save Version
will save your current state of your notebook with a version tag that contains a date and time stamp. Up to 10 versions of your notebook can be saved, each one retrievable by selecting theRevert To Version
menu item.
You can share your notebook by selecting the “Share” button located in the top right section of your notebook panel. The end result of this action will be a URL link that will display a “read-only” version of your notebook. You have several options to specify exactly what you want shared from your notebook:
Only text and output
: will remove all code cells from the notebook view.All content excluding sensitive code cells
: will remove any code cells that contain a sensitive tag. For example,# @hidden_cell
is used to protect your dashDB credentials from being shared.All content, including code
: displays the notebook as is.- A variety of
download as
options are also available in the menu.
There is a sample of the output in data/examples/pixiedust-traffic-analysis.html, it is best viewed via rawgit.
Note: Some interactive map functionality, like
Options
andLayers
will not work. To see these, you must run the notebook itself.
- Artificial Intelligence Code Patterns: Enjoyed this Code Pattern? Check out our other AI Code Patterns.
- Data Analytics Code Patterns: Enjoyed this Code Pattern? Check out our other Data Analytics Code Patterns
- AI and Data Code Pattern Playlist: Bookmark our playlist with all of our Code Pattern videos
- With Watson: Want to take your Watson app to the next level? Looking to utilize Watson Brand assets? Join the With Watson program to leverage exclusive brand, marketing, and tech resources to amplify and accelerate your Watson embedded commercial solution.
- Watson Studio: Master the art of data science with IBM's Watson Studio
This code pattern is licensed under the Apache Software License, Version 2. Separate third party code objects invoked within this code pattern are licensed by their respective providers pursuant to their own separate licenses. Contributions are subject to the Developer Certificate of Origin, Version 1.1 (DCO) and the Apache Software License, Version 2.