ericsun95 / spark-osm-pbf Goto Github PK

Setup parsers

• Setup PBFBlobParser
• Setup PBFFileParser
• Setup PBFStreamParser

Add Initial Tests

• Add test to PBFBlobParser
• Add test to PBFFileParser
• Add test to PBFStreamParser

Explore DataSourceV2 API and come up with a plan.

Data Source API V2 supports filter pushdown performance optimization for DataSourceReaders with SupportsPushDownFilters (that is applied when DataSourceV2Strategy execution planning strategy is requested to plan a DataSourceV2Relation logical operator).

(From Parquet Filter Pushdown in Apache Drill’s documentation) Filter pushdown is a performance optimization that prunes extraneous data while reading from a data source to reduce the amount of data to scan and read for queries with supported filter expressions. Pruning data reduces the I/O, CPU, and network overhead to optimize query performance.

When reading big files locally, it shows gc problem.
java.lang.OutOfMemoryError: Java heap space

It seems relevant to the readFully store lots of cache which requires more memory.

Add support to Spark 3.0. (May need a new branch as the DataSourceV2 style changed a lot compared with spark 2.4.x)

Support BBox Extraction when parsing the PBF file header

Make cumulative sum calculation more generic

Add support generate an assembly jar for Action workflow.

Support PBF Output so that we can compare file by file

Setup workflow with tests