Comments (15)
@dessertalready : The dataset 'yelp_dataset_challenge_round9' is kinda a double zipped file. If you try to rename the file by adding a ".tar" extension to it, you'll be able to see different .json files for business, check-ins, reviews, tips and users. Then you could use those files as input for the converter.
Hoping this helps.
from dataset-examples.
@CAVIND46016: I am very grateful for your warm help!!
I was not sure it was a bouble zipped file, and you really make me clear!
Thanks a lot!!
from dataset-examples.
Hi,
I also faced trouble while using the python code to convert json files to csv. The error message is as follows:
usage: json_to_csv_converter.py [-h] json_file
json_to_csv_converter.py: error: the following arguments are required: json_file
C:\Users\Sajeev\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py:2889:
I am a novice at Python programming. Appreciate any help to fix this.
Thanks
from dataset-examples.
Hi,
I also faced similar trouble while using the python converter to convert the json to csv format:
This issue puzzled me for quite some time. Need help!
Thanks a lot.
from dataset-examples.
If you guys have R, you might want to try using that. It's worked pretty well for me. I believe R loads the the entire file into memory before doing the operation, so if it crashes it's probably because it ran out of RAM.
Here's an example script in R that I've used to convert the properties I need into a CSV file.
#makeEReviewFull.R
#Last updated December 14, 2016 12:59
#This script converts the a JSON file containing review data
#into an edge file for upload into System G Graph store.
#The JSON contains all reviews from the yelp
#academic dataset.
#Uses jsonlite to create a csv file for reviews.
library(jsonlite)
reviews <- stream_in(file("./yelp_academic_dataset_review.json"))
#reviews$text <- gsub('\n','',reviews$text)
edges <- data.frame(
source = reviews$user_id,
target = reviews$business_id,
type = reviews$type,
date = reviews$date,
stars = reviews$stars,
votes = reviews$votes,
review_id = reviews$review_id
)
write.csv(edges, "./eReviewsFull.csv", row.names=F)
from dataset-examples.
Thank you for your help!
In fact, I have another question about the dataset.
There is only one file without suffix in the download tar, named yelp_dataset_challenge_round9. Is it a json file with all types of data, e.g., business, tips.
As the file you have used seems to be independent only about the data of review, i.e., yelp_academic_dataset_review.json. And it is transformed according to its structure, while different types of data is with distinct structures.
I have no idea about the dataset.
Thanks!
from dataset-examples.
@Aneapiy : thanks a lot for offering the code in R. But, I keep getting the following error:
Error in (function (..., row.names = NULL, check.rows = FALSE, check.names = TRUE, :
arguments imply differing number of rows: 7, 2, 19, 6, 3, 5, 1, 0, 18, 4, 8, 20, 25, 14, 12, 13, 26, 11, 21, 10, 22, 16, 23, 27, 9, 17, 15, 24, 31, 28, 29, 30, 32, 34
I am trying to read a json file that is different from what you have given in your sample code. I am trying to convert the file: yelp_academic_dataset_business.json. Is this because some columns have missing or NULL values? I think the function stream_in works because I am able to see the data format in R. I guess the problem happens when we try to convert it to dataframe.
Another issue is that there are columns called attributes, categories, etc. which have multiple values. Is there is a way to read each of these values and output to separate columns in csv?
Thanks
from dataset-examples.
@svknair. If you're trying to read the business data you might one to try changing the names of the keys R is looking for. For example, within the dataframe parenthesis you might want to change
source = reviews$user_id,
to
id = reviews$business_id,
Note that the string before the $ sign has to match the name of the variable you're streaming the information into. The string after the $ is the JSON key that you want to look in. The keys in the business file is different than the keys in the reviews file.
Hmm, as far as columns like attributes etc are concern, I'm not sure if this would pose a problem since there's multiple different sub categories it could taken on. You might want to try changing the keys I mentioned above, importing in a small subset of the business day (say the first 10 lines) and seeing if R flattens it out automatically.
Here's an example of what I've used for the business json file:
#makeVBusinessFull.R
#Last updated December 14, 2016 09:12AM
#This script converts the a JSON file containing business data
#into a vertex file for upload into System G Graph store.
#The JSON is the original business data from the yelp
#academic dataset without any preprocessing.
library(jsonlite)
biz <- stream_in(file("./yelp_academic_dataset_business.json"))
biz$full_address <- gsub('\n',' ',biz$full_address)
nodes <- data.frame(
id = biz$business_id,
type = biz$type,
name = biz$name,
city = biz$city,
state = biz$state,
stars = biz$stars,
latitude = biz$latitude,
longitude = biz$longitude,
review_count = biz$review_count,
full_address = biz$full_address
)
write.csv(nodes, "./vBusinessFull.csv", row.names=F)
from dataset-examples.
the conversion in R is stuck after an hour.. does this happen with anybody else?
from dataset-examples.
@Aneapiy Thanks for your suggestions. The code works fine. Another query that I have: how can I split the fields named attributes, categories and hours? These have multiple sub-fields that I would like to split into separate columns. I am referring to the business data file here.
from dataset-examples.
I faced error when wanted to Run R script of #makeReviewFull
Loading required package: methods
opening file input connection.
Imported 4153150 records. Simplifying...
closing file input connection.
Error in data.frame(source = reviews$user_id, target = reviews$business_id, :
arguments imply differing number of rows: 4153150, 0
Execution halted
and for MakeVBusinessFull:
Loading required package: methods
opening file input connection.
Imported 144072 records. Simplifying...
closing file input connection.
Error in `$<-.data.frame`(`*tmp*`, "full_address", value = character(0)) :
replacement has 0 rows, data has 144072
Calls: $<- -> $<-.data.frame
Execution halted
from dataset-examples.
how to open file json if size more than 1 gb??
from dataset-examples.
I have no idea if you can open with any tools in Windows, but for Unix machines you can use grep commands to view for content matching a pattern, or use some split commands to view smaller chunks of the file. There might be some softwares available in Windows also that can do the split job.
Lots of discussion on the web for this can be found:
- https://news.ycombinator.com/item?id=12063626
- https://askubuntu.com/questions/28847/text-editor-to-edit-large-4-3-gb-plain-text-file
- Sublimetext (http://www.sublimetext.com/)
Thanks.
from dataset-examples.
I have the same issue with converting the business file:
> biz$full_address <- gsub('\n',' ',biz$full_address)
Error in `$<-.data.frame`(`*tmp*`, "full_address", value = character(0)) :
replacement has 0 rows, data has 144072
It looks like there is no "full_address" field, and that it has been replaced with "address". At least, that's how it is in my download.
from dataset-examples.
Cannot convert business.json to csv
error: TypeError: argument of type 'NoneType' is not iterable
from dataset-examples.
Related Issues (20)
- More/Fewer total reviews per business than in 'review_count'
- where is the label if review is positive or negative?
- json_to_csv_converter.py has a small issue for the new yelp dataset HOT 1
- json_to_csv_converter.py fixed for Python 3 HOT 7
- What's the meaing of each column heading ?
- question about date availability for business in NYC HOT 2
- Review_count HOT 1
- Get Yelp original json file HOT 1
- Check-in data is not user-related? HOT 1
- missing attribute in category_predictor.py HOT 1
- Is the first entry in the variable “categories” the superordinate category?
- business.json is not converted to business.csv HOT 2
- Broken links in README HOT 2
- Question regarding the time zones in the review.json file HOT 1
- 'NoneType' object is not subscriptable error when yelp_academic_dataset_business.json conversion
- Citation issue
- Including not recommended reviews
- Convert JSON to CSV: Missing module source and missing undefined variable
- Extract review.json file from yelp_academic_dataset.json
- Can I know which photo belongs to which review
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dataset-examples.