We keep seeing a common phrase that 80% of the work of a data scientist is data cleaning. We have no idea whether this number is accurate but a data scientist indeed spends lots of time and effort in collecting, cleaning and preparing the data for analysis. This is because datasets are usually messy and complex in nature. It is a very important ability for a data scientist to refine and restructure datasets into a usable state in order to proceed to the data analysis stage.
In this exercise, you will both practice the data cleaning techniques we discussed in the lesson and learn new techniques by looking up documentations and references. You will work on your own but remember the teaching staff is at your service whenever you encounter problems.
In this lab you will be working on main.ipynb. To launch it, first navigate to the directory that contains main.ipynb
in Terminal, then execute jupyter notebook
. In the webpage that is automatically opened, click the main.ipynb
link to launch it.
When you are on main.ipynb
, read the instructions for each cell and provide your answers. Make sure to test your answers in each cell and save. Jupyter Notebook should automatically save your work progress. But it's a good idea to periodically save your work manually just in case.
Do you remember the SQL class? In this lab, you will examine some BigQuery tables.
main.ipynb
with your responses to each of the questions above.outliers.csv
containing the outliers of the dataset.weather.ipynb
containing the additional challenge code and results.
Upon completion, add your deliverables to git. Then commit git, push to your forked repo, and create the pull request as in the previous labs. **REMEMBER
-
Upon completion, commit your code and submit to github. REMEMBER YOU HAVE ALREADY FORKED THE REPO BEFORE!!
git add . git commit -m "<lab or project name>" git push origin master
-
Navigate to your repo and create a Pull Request.
-
Create a pull request with title following this format: "[<your_campus>][<bootcamp_code>] [<lab/project_name>]<your_name>"
- For instance, if you are doing data bootcamp in Madrid, your name is Marc Pomar and the lab you are working on is
lab-numpy
, your pull request should be named like this: "[MAD][datamad10108] [lab-numpy] Marc Pomar"
- For instance, if you are doing data bootcamp in Madrid, your name is Marc Pomar and the lab you are working on is
-
If you have successfully created the pull request you are done! CONGRATS :)
Data Cleaning with Numpy and Pandas
If you have completed the main challenge without much difficulty, you can try to tidy the data you will find in the lab folder weather. This dataset is a subset of a global historical climatology network dataset. The data represents the daily weather records for a weather station (MX17004) in Mexico for five months in 2010. The goal of this additional challenge is to get the most tidy dataset you are able to produce. Hint:Variables are stored in both rows and columns.
To accomplish this challenge, you will need to do some research on tidying and melt&pivot. Feel free to reference any resources you consider appropiate.