This repo contains scripts to do web scraping of e-commerce grocery web pages.
This is done merely as a school project and we adhere to the robots.txt and ethics of being a good webscrapper.
The project directory would look something like this after a few rounds of scraping.
├── config
├── data
│ ├── fairprice
│ │ └── raw
│ │ └── 20180820_links.csv
│ └── redmart
│ ├── processed
│ │ └── data.csv
│ └── raw
│ ├── 20180809_data.json
│ ├── 20180811_data.json
│ ├── 20180812_data.json
│ ├── 20180813_data.json
│ ├── 20180814_data.json
│ ├── 20180818_data.json
│ └── 20180820_data.json
├── environment.yml
├── LICENSE
├── log
├── notebook
│ ├── FairPrice Exploratory.ipynb
│ ├── MongoDB.ipynb
│ ├── narrative-python.ipynb
│ └── Redmart Exploratory .ipynb
├── pictures
├── README.md
├── report
├── robots
│ ├── fairprice_robots.txt
│ └── redmart_robots.txt
├── seleniumdrivers
│ └── chromedriver
└── src
├── fairprice.py
├── make_dataset.py
├── make_mongodb_redmart.py
├── redmart.py
└── utils.py
In order to run the code, you would need to have Anaconda3 installed.
- Clone the repo
git clone https://github.com/notha99y/Grocery.git
- Set up conda environment
cd Grocery
conda env create -f=environment.yml
With this, you are set to do webscraping and some simple data analysis
TODO
Currently, these are some of the things you do can with the Redmart Scripts
- Scrape Redmart
- This would create a project directory called data/raw, scrapes the redmart webpage and saves it into a .json file
- This would take roughly 10 mins (depending on your internet speed and redmart servers) The .json file would be rougly 200 MB in size
python src/redmart.py
The redmart collection is roughly 60 MB in size and have ~ 62,000 unique items
- Extract data into MongoDB
- This would extract the relevant information from the raw json data and add it into MongoDB.
- The extracted data will be saved into a db called Grocery and in a collection called redmart
python src/make_mongodb_redmart.py
For those who are familiar with Tableau could connect the MongoDB to Tableau.
How to connect MongoDB to Tableau
We have came up with an Interactive Dashboard ( <-- Click )
Go Full Screen for maximum viewing pleasure.
The FairPrice Script takes roughly 4 hours to run and would output a .csv file called $TODAY_DATE_links.csv (which contains all the products listing links of fairprice) create collection called fairprice of ~6 MB in size with ~6,500 unique items.
In order to run the script, you would need to do the following:
- Get Selenium Chromedriver download here
- Unzip the chromedriver
- Set the chrome_driver variable in line 65 of utils.py to the path of your downloaded chromedriver
With these, you are ready to run. The script would given you the following outputs:
- date_links.csv (36 mins): csv file which contains all the product links
- A Grocery MongoDB with a fairprice collection of product documents
python src/fairprice.py
We have came up with an Interactive Dashboard ( <-- Click )
Go Full Screen for maximum viewing pleasure.
TODO
We could run some commands using the MongoDB shell.
Alternatively, the default MongoDB GUI called Compass could do the job.
Screenshot of the GroceryDB and its collections
- Open MongoDB shell
mongo
- Show databases
- You should see a db called Grocery
show dbs
- Show collections
- This would display the two collections:
- fairprice
- redmart
- This would display the two collections:
use Grocery
show collections
- Count Query
- We can count the number of items in the collections
db.redmart.find().count()
- Get Data Size
db.redmart.dataSize()
- Drop collections
db.redmart.drop()