Parser optimization: Seperate filtering and property handling and the two passes in OSMReader about graphhopper HOT 3 CLOSED

This would speed up parsing on expense of memory.

I don't like this idea. For me RAM is more important than speed, e.g. if you want to import planet.osm you already need 20GB and this is increasing by 40% per year due to OSM contributions! So, before increasing the RAM usage we should make it possible to have a fast but on-disc solution for GHLongIntBTree. But I didn't find a good solution. E.g. MapDB was too slow. I'll experiment with Lucene again as we probably need it elsewhere.

Of course you can make this speed-up optional, e.g. in a new OSMReaderHelper subclass or whatever

from graphhopper.

It would probably be worthwhile to have a technology that will run with limited memory for any size of file. In my map compiler I had a similar problem caching the nodes. I wrote my own storage class that is organized in pages which are loaded from disk, with a maximum number that can be loaded at one time.

Until then it can't hurt to make this optional, controlled by a property. If there is no cached data available, the ways are parsed again just like it happens now. Nothing lost. If you have enough memory, it should run faster.

from graphhopper.

This can be closed.

from graphhopper.