Project - EDA with Pandas Using the Boston Housing Data

Introduction

In this section, you've learned a lot about importing, cleaning up, analyzing (using descriptive statistics) and visualizing data. In this more free-form project, you'll get a chance to practice all of these skills with the Boston Housing dataset, which contains housing values in the suburbs of Boston. The Boston housing data is commonly used by aspiring Data Scientists.

Objectives

You will be able to:

• Perform a full exploratory data analysis process to gain insight about a dataset

Goals

Use your data munging and visualization skills to conduct an exploratory analysis of the dataset below. At a minimum, this should include:

• Loading the data (which is stored in the file 'train.csv')
• Use built-in Python functions to explore measures of centrality and dispersion for at least 3 variables
• Create meaningful subsets of the data using selection operations like .loc, .iloc, or related operations. Explain why you used the chosen subsets and do this for three possible 2-way splits. State how you think the two measures of centrality and/or dispersion might be different for each subset of the data. Examples of potential splits:
  • Create two new DataFrames based on your existing data, where one contains all the properties next to the Charles river, and the other one contains properties that aren't
  • Create two new DataFrames based on a certain split for crime rate
• Next, use histograms and scatter plots to see whether you observe differences for the subsets of the data. Make sure to use subplots so it is easy to compare the relationships.

Variable Descriptions

This DataFrame contains the following columns:

• crim: per capita crime rate by town
• zn: proportion of residential land zoned for lots over 25,000 sq.ft
• indus: proportion of non-retail business acres per town
• chas: Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)
• nox: nitrogen oxide concentration (parts per 10 million)
• rm: average number of rooms per dwelling
• age: proportion of owner-occupied units built prior to 1940
• dis: weighted mean of distances to five Boston employment centers
• rad: index of accessibility to radial highways
• tax: full-value property-tax rate per $10,000
• ptratio: pupil-teacher ratio by town
• b: 1000(Bk - 0.63)^2 where Bk is the proportion of African American individuals by town
• lstat: lower status of the population (percent)
• medv: median value of owner-occupied homes in $10000s

Source

• Harrison, D. and Rubinfeld, D.L. (1978) Hedonic prices and the demand for clean air. J. Environ. Economics and Management 5, 81–102.

• Belsley D.A., Kuh, E. and Welsch, R.E. (1980) Regression Diagnostics. Identifying Influential Data and Sources of Collinearity. New York: Wiley.

Important: We mentioned in our data ethics lesson earlier that data can include offensive or inappropriate language at times. This is already becoming very apparent in the Boston Data, which includes a racial variable. Note that the Boston Housing data was constructed in the 1970’s, and is based on US census Data. On census.gov, you can read the following note:

"Census statistics date back to 1790 and reflect the growth and change of the United States. Past census reports contain some terms that today's readers may consider obsolete and inappropriate. As part of our goal to be open and transparent with the public, we are improving access to all Census Bureau original publications and statistics, which serve as a guide to the nation's history."

With that note, it is important to absolutely handle with care, and it is strongly adviced to not use the racial variable when creating a predictive model.

Summary

Congratulations, you've completed your first "free form" exploratory data analysis of a popular dataset!