e-Statは最大100000行データしか撮れません about japandas HOT 4 CLOSED

Thank you for the report. Maybe we should provide limit parameter in the request. Is it possible for you to tell me dataset name which has 100000 rows (to confirm the behavior) ?

https://www.e-stat.go.jp/api/api-info/e-stat-manual

from japandas.

@sinhrks Thank you for your reply. There are quite a lot of dataset that has over 100000 rows, e.g. 生産・出荷集中度調査 / 累積生産集中度（長期時系列データ）(statsDataId=0003280394).

One way I find to solve this problem is to specify the "cat01"/"cat02"/"cat03" (which can be found in meta information) when using API to request data and then to loop for all the categories. Thus it will be great appreciated if you can add the function of parameter-specification to jpd.DataReader().

from japandas.

I've fixed the issue and going to release new version (0.5.1).

e-stat limit kw doesn't work for more than 100,000 records. Thus, you should specify startPosition kw.

https://japandas.readthedocs.io/en/latest/jpdatareader.html#e-stat-api

from japandas.

@sinhrks Sorry, I didn't notice your comment. I solved the 100,000 limit by divid the data into several pieces based on the meta code of the category like industry and scale. But your approach is just better. Great job.

from japandas.