osmlab / csv-err Goto Github PK

Everything in csv-err goes through postgres for normalizing. Import and export is slow, installation is complicated and fragile; I don't like relying on it.

Eventually, would like to rely on ogr2ogr and something like csv-fix and keep everything simple (and hopefully faster) with cli tools.

• http://www.gdal.org/drv_csv.html

For osmi:
ogr2ogr -s_srs EPSG:4326 -t_srs EPSG:4326 -f CSV -lco GEOMETRY=AS_WKT the_csv.csv role_mismatch_hull.gml

Finding lots of dupes just manually going through things.

Some of the errors have geometry with them, we should use them for the possibility of filtering bounding boxes. These don't have to be full exact geometry just a representative point.

• Maybe we should have a consistent geometry column?
• What format should we standardize around? Right now everything is WKT because it's lightweight?

cc @sbma44

Currently everything is grouped by it's origin, for example keepright and a bunch of errors are gathered together into their own csvs.

So there is:

• keepright-tasks/error1.csv
• keepright-tasks/error2.csv
• keepright-tasks/error3.csv

and then that directory is zipped up.

But sometimes certain errors can't be generated because of random problems, the world is chaotic and such (#2). So, we only want to replace certain files, that isn't currently possible with the directory zip, these all need to be individual files on s3 and then replaced when possible.

Then, how do we know when something came from? Where can I store a timestamp?

There's a marginal difference between the 1, 2, 5 classification. I've been combining them this week and it's fine.

Previously: osmlab/to-fix#2

Right now the error is

Server error! We are sorry for the inconvenience. 35
Please contact info@geofabrik if the error persists.

but I've seen slightly different errors as well so we can't count on that. Best approach would be to check the file size after download and only proceed with importing those that actually exist. This means export scripts also need to be able to not find certain things that are missing and handle it.

So the goal is that some things might get updated some might not and that's ok. Everything is just ok.

Dump is here: http://osmose.openstreetmap.fr/export/

Need to look into how difficult this will be downstream in to-fix.

Goal is to bake vector tiles for easy styling; to see coverage. That's easiest with geojson.