gregorut / vgchartzscrape Goto Github PK
View Code? Open in Web Editor NEWScrapy project for data capture of vgchartz
License: GNU General Public License v3.0
Scrapy project for data capture of vgchartz
License: GNU General Public License v3.0
Hi there,
I am very new to python, but I want to use this code to get a more current dataset than the one posted in Kaggle.
After the code has finished scraping, it gives this message:
Page: 1
Page: 2
Page: 3
Page: 4
Page: 5
Page: 6
Page: 7
Page: 8
Page: 9
Page: 10
Page: 11
Page: 12
Page: 13
Page: 14
Page: 15
Page: 16
Page: 17
Page: 18
0
Index(['Rank', 'Name', 'Platform', 'Year', 'Genre', 'Critic_Score',
'User_Score', 'Publisher', 'Developer', 'NA_Sales', 'PAL_Sales',
'JP_Sales', 'Other_Sales', 'Global_Sales'],
dtype='object')
Then when I open the csv it has written, all it has is the column headers but no data.
Any help would be much appreciated!!
Jules
Hi,
I found your Kaggle dataset very interesting. I've worked on this crawler fork to keep it up to date and operate with the latest version of the data. I also want to made a Pull Request when it will be finished.
I wanted to know if you thought about licensing the code, e.g. MIT? Since I'd like to apply it and respect your wishes about it.
Is it possible to apply MIT through generating a LICENSE.md file?
Greetings and thanks in advace for your reply.
Manuel
Hi,
I tried running the script (in Python 3) however I encountered an issue at line 39 (lambda x: x.attrs['href'].startswith('http://www.vgchartz.com/game/') ) which resulted in a KeyError 'href'.
Kindly advice what is needed to solve this?
Many thanks!
Hi,
I saw the dataset over on Kaggle (https://www.kaggle.com/gregorut/videogamesales) and was hoping to use it for a college project. I wanted to get the most recent copy of the data, so I came here. I would appreciate if you could add a license to this repository (or the dataset on Kaggle if you would prefer) so that I can make use of it. https://choosealicense.com/no-permission/
Thank,
Philip
Good Afternoon! I recently tried rerunning the script for a class project but have been unsuccessful in rerunning the script. I have tried running in on Linux (multiple python versions and windows). I have no experience coding python and was able to figure out how to fix a couple of other errors before this one completely stumped me.
Traceback (most recent call last):
File "vgchartsfull2.py", line 29, in
for row in chart.find_all('tr')[1:]:
AttributeError: 'NoneType' object has no attribute 'find_all'
Any help will be greatly appreciated :)
vgchartzfull.py:33: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("html.parser"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.
The code that caused this warning is on line 33 of the file vgchartzfull.py. To get rid of this warning, pass the additional argument 'features="html.parser"' to the BeautifulSoup constructor.
soup = BeautifulSoup(r)
Page: 1
1 Fetch data for game Wii Sports
2 Fetch data for game Super Mario Bros.
Traceback (most recent call last):
File "vgchartzfull.py", line 92, in
site_raw = urllib.request.urlopen(url_to_game).read()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 531, in open
response = meth(req, response)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 641, in http_response
'http', request, response, code, msg, hdrs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 569, in error
return self._call_chain(*args)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 429: Rate
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.