Comments (4)
Thank you for the report. Maybe we should provide limit
parameter in the request. Is it possible for you to tell me dataset name which has 100000 rows (to confirm the behavior) ?
https://www.e-stat.go.jp/api/api-info/e-stat-manual
from japandas.
@sinhrks Thank you for your reply. There are quite a lot of dataset that has over 100000 rows, e.g. 生産・出荷集中度調査 / 累積生産集中度(長期時系列データ)(statsDataId=0003280394).
One way I find to solve this problem is to specify the "cat01"/"cat02"/"cat03" (which can be found in meta information) when using API to request data and then to loop for all the categories. Thus it will be great appreciated if you can add the function of parameter-specification to jpd.DataReader().
from japandas.
I've fixed the issue and going to release new version (0.5.1).
e-stat limit
kw doesn't work for more than 100,000 records. Thus, you should specify startPosition
kw.
https://japandas.readthedocs.io/en/latest/jpdatareader.html#e-stat-api
from japandas.
@sinhrks Sorry, I didn't notice your comment. I solved the 100,000 limit by divid the data into several pieces based on the meta code of the category like industry and scale. But your approach is just better. Great job.
from japandas.
Related Issues (6)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from japandas.