Giter VIP home page Giter VIP logo

hive-on-yelp's Introduction

Hive-on-Yelp

Introduction

This repository is hosting our analysis of Yelp Dataset. Yelp Dataset is available here: https://www.yelp.com/dataset/download

For the sentiment analysis, we have used a dictionary dataset found here: https://s3.amazonaws.com/hipicdatasets/dictionary.tsv

state_locations.txt file is used for producing better location detection in Excel 3D Map; it can be found here: https://drive.google.com/uc?id=1dFrIcQuBhaANRHHvnzbthfU3HHVDRy7Y&export=download

Complete tutorial is available here: https://tinyurl.com/hive-on-yelp

There are five JSON files in this dataset: business, checkin, review, user, and tip. Our downloaded version of the dataset is 9.8 GB when uncompressed.

We have run descriptive analytics on the Yelp dataset to find out how different features of Yelp evolved throughout last decades, how businesses in different states receive different level of ratings, what people express in their tips by doing sentiment analysis.

Structure

Codes

This directory contains all the final codes in one place. Please replace UNAME with appropriate username.

Deliverables

All the final submissions are saved in this directory.

Outputs

This directory holds all the final output files.

preparation.sh

This file prepares the user machine by extracting JSON files from the archive and moving the JSON files to HDFS filesystem.

primary_tables.hql

This file creates initial raw tables based on the five JSON files, and standard tables based on the raw tables.

state_locations.txt

Maps state acronyms used in the Yelp Dataset to complete state names with country names for better map rendering.

Workflow Chart.jpg

This flow chart explains the top-level view of overall operation.

drafts

This directory contains all the previous versions and ongoing works

hive-on-yelp's People

Contributors

mohsenualam avatar lunettorandall avatar ben17er avatar

Stargazers

 avatar  avatar

Watchers

James Cloos avatar  avatar  avatar

hive-on-yelp's Issues

Count of distinct dates seems wrong

After running count(distinct(checkin_dates)), the output found is 348, which is weird because the date range in check-ins is from January 2010 to December 2019. Suspect that some days went missing in one of the tables. This issue also caused a gap between # of check-in per day in this chart:
unknown

standard_tables.hql: creating review table issue

Unequal number of rows in raw_review and review table

--Creating review table
CREATE TABLE review (review_id string, rev_user_id string, rev_business_id string, rev_stars int, rev_useful int, rev_funny int, rev_cool int, rev_text string, rev_date string);
--Populating review table from raw_review
FROM raw_review INSERT OVERWRITE TABLE review SELECT get_json_object(json_response, '$.review_id'), get_json_object(json_response, '$.user_id'), get_json_object(json_response, '$.business_id'), get_json_object(json_response, '$.stars'), get_json_object(json_response, '$.useful'), get_json_object(json_response, '$.funny'), get_json_object(json_response, '$.cool'), get_json_object(json_response, '$.text'), get_json_object(json_response, '$.date');

Some of the reviews have quotation marks and escape characters. These reviews are not being parsed correctly. For example, one review's text attributes look like:

"As someone who has worked with many museums, I was eager to visit this gallery on my most recent trip to Las Vegas. When I saw they would be showing infamous eggs of the House of Faberge from the Virginia Museum of Fine Arts (VMFA), I knew I had to go!\n\nTucked away near the gelateria and the garden, the Gallery is pretty much hidden from view. It's what real estate agents would call "cozy" or "charming" - basically any euphemism for small.\n\nThat being said, you can still see wonderful art at a gallery of any size, so why the two s you ask? Let me tell you:\n\n pricing for this, while relatively inexpensive for a Las Vegas attraction, is completely over the top. For the space and the amount of art you can fit in there, it is a bit much.\n* it's not kid friendly at all. Seriously, don't bring them.\n* the security is not trained properly for the show. When the curating and design teams collaborate for exhibitions, there is a definite flow. That means visitors should view the art in a certain sequence, whether it be by historical period or cultural significance (this is how audio guides are usually developed). When I arrived in the gallery I could not tell where to start, and security was certainly not helpful. I was told to "just look around" and "do whatever." \n\nAt such a fine institution, I find the lack of knowledge and respect for the art appalling."

After parsing, the review table's rev_text column has only this portion:
As someone who has worked with many museums, I was eager to visit this gallery on my most recent trip to Las Vegas. When I saw they would be showing infamous eggs of the House of Faberge from the Virginia Museum of Fine Arts (VMFA), I knew I had to go!

And, the rev_date column became null.

It took 11 rows to completely parse one review. Similar things happened for all the reviews with \n with it.

As a result, 8,021,122 rows from the raw_review table became 25,129,230 rows in the review table.

We can fix this issue by somehow ignoring '\n' when populating the review table.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.