apache / incubator-baremaps Goto Github PK

Create custom vector tiles from OpenStreetMap and other data sources with Postgis and Java.

License: Apache License 2.0

Java 74.14% PLpgSQL 0.28% Shell 0.82% Batchfile 0.05% HTML 1.53% JavaScript 21.46% TypeScript 1.37% Python 0.16% FreeMarker 0.18%

vector-tiles mapbox openstreetmap java postgresql postgis web-mapping spatial-data

incubator-baremaps's Introduction

Apache Baremaps (Incubating)

Apache Baremaps is a toolkit and a set of infrastructure components for creating, publishing, and operating online maps. It provides a data pipeline enabling developers to build maps with different data sources with live reload capabilities. It provides other services commonly used in online maps, such as location search and IP to location.

🔥 Live Demo

Live Demo

📖 How do I use Apache Baremaps?

You can find the documentation on the project's website. The following pages showcase the main uses of Apache Baremaps.

The OpenStreetMap example is a good introduction to Baremaps, it shows how to produce high resolution vector tiles.
The NaturalEarth example shows how to produce low resolution vector tiles.
The Contour example shows how to produce contour lines from a digital elevation model.
The IP to location example shows how to create and serve an IP to location service in a simple web application.
The Geocoding example shows how to create and serve a geocoding service in a simple web application.

👩‍💻 How do I contribute?

There are many places where you can contribute to Apache Baremaps such as the code, the documentation, the website or the examples.

The official documentation is located in a separate repository.

If you want to contribute to the code you can refer to the following developer guides available in the documentation.

You can also contribute in the following ways.

Create an issue: Report a bug or feature request
Join the mailing list: Initiate or participate in project discussions on the mailing list
Write a post: Write a post to share your use cases and experiences with Apache Baremaps

Finally, check out CONTRIBUTING and CODE_OF_CONDUCT.

📄 License

This project is licensed under Apache License 2.0 - see the LICENSE file for details.

incubator-baremaps's People

Contributors

Stargazers

Watchers

incubator-baremaps's Issues

Publish artifacts on maven central with github actions

Double quote PostgreSQL table names in the backend

I faced this error after starting the server with a custom layer having non standard SQL characters (must be double quoted):

[WARN ] 2021-02-21 14:54:26.213 [armeria-common-worker-epoll-2-5] HttpResponseSubscriber - [id: 0x1dea6d9c, L:/172.28.0.3:9000 - R:/10.193.21.89:42482] Unexpected exception from a service or a response publisher: com.baremaps.server.TileService@238a9e9c
java.lang.IllegalArgumentException: The SQL query malformed
	at com.baremaps.tile.postgres.PostgisQueryParser.parse(PostgisQueryParser.java:52) ~[baremaps-tile-postgres-0.4.0.jar:?]
	at com.baremaps.tile.postgres.PostgisTileStore.lambda$null$0(PostgisTileStore.java:137) ~[baremaps-tile-postgres-0.4.0.jar:?]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) ~[?:?]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274) ~[?:?]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
	at com.baremaps.tile.postgres.PostgisTileStore.query(PostgisTileStore.java:138) ~[baremaps-tile-postgres-0.4.0.jar:?]
	at com.baremaps.tile.postgres.PostgisTileStore.read(PostgisTileStore.java:110) ~[baremaps-tile-postgres-0.4.0.jar:?]
	at com.baremaps.server.TileService.lambda$doGet$0(TileService.java:58) ~[baremaps-server-0.4.0.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) ~[?:?]
	at com.linecorp.armeria.common.RequestContext.lambda$makeContextAware$3(RequestContext.java:522) ~[armeria-1.4.0.jar:?]
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44) ~[micrometer-core-1.6.3.jar:1.6.3]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]

I suspect the PostgreSQL layer name MUST be escaped in the SQL query defined in the ./config/layer.yaml file, especially when they contain non standard SQL characters such as capital letters or digits, because when I escape the layer name using double quotes, it doesn't print this error at startup.

...
FROM 1_differential_rabbiHousing${zoom}
---

-> goes bad...

...
FROM "1_differential_rabbiHousing"${zoom}
...

-> goes better...

Would it be possible to quote (/check for) double quote around the PostgreSQL table names in the backend?

Here's the config.yaml:

id: OpenStreetMapVecto
server:
  host: localhost
  port: 9000
center:
  lon: 6.5743
  lat: 46.5189
  zoom: 14
bounds:
  minLon: 5.8358
  minLat: 45.6591
  maxLon: 10.9794
  maxLat: 47.8700
  minZoom: 6
  maxZoom: 14 
layers:
  - config/rabbits/1_differential_rabbiHousing/layer.yaml

stylesheets:
  - config/rabbits/1_differential_rabbiHousing/stylesheet.yaml

/config/rabbits/1_differential_rabbiHousing/layer.yaml:

id: 1_differential_rabbiHousing
type: point
queries:
  - minZoom: 13
    maxZoom: 22
    sql: |
      SELECT id, rabbit_id, name, familiy, has_carot, speed, geometry
      FROM "1_differential_rabbiHousing"${zoom}
      WHERE geometrytype(geometry) LIKE '%POINT%'

/config/rabbits/1_differential_rabbiHousing/stylesheet.yaml:

id: 1_differential_rabbiHousing
styles:
  - id: ogc_fid
    layer: 1_differential_rabbiHousing
    type: fill
    layout:
      visibility: visible
    paint:
      fill-color: rgb(0, 0, 228)
      fill-antialias: true

I'm not sure whereas the layer and id in the stylesheet has to represent the same object and what they do refer to?

Allow file extensions matching when searching for layers and stylesheets

I faced this error after starting the server with a custom layer:

[ERROR] 2021-02-21 14:46:02.017 [main] Preview - Unable to read the configuration file.
com.fasterxml.jackson.databind.JsonMappingException: /config/custom/skating_mushrooms/layer.yaml (through reference chain: com.baremaps.config.Config["layers"]->java.util.ArrayList[0])
...
java.lang.NullPointerException
...

but my layer is actually named skating_mushrooms/layer.yml, therefore I suspect that the file extension is hard codded.

From here, there are probably two solutions:

baremaps should also allow for .yml extensions, e.g. within a regexp,
or it should get the full filename (including extension) so that files can be named with any extension.

Assume that the tiles are hosted at the origin

Behind a proxy, the tile source and the html pages may have different origin.

Replace GIST with SP-GIST

As tiles are a form of space partitionning, SP-GIST are probably a better fit than GIST for indexing geometries. The idea would be to investigate how an SP-GIST can be configured to match tile boundaries.

https://www.postgresql.org/docs/12/spgist.html

Examine the usage of a BufferedInputStream

https://github.com/baremaps/baremaps/blob/860396be5648d060712e5dedea0e5feb5f25a21a/baremaps-osm/src/main/java/com/baremaps/osm/reader/pbf/FileBlockStreamer.java#L20

If the consuming code of the stream is doing lots of small reads, using a BufferedInputStream might improve performance here.

Reorganize the command line interface

As new services will be added, the command line interface needs to be reorganized. The current idea is to have something similar to what aws-cli does, i.e., the sub command correspond to the context. In baremaps it could translate as follow:

baremaps [context] [action]
baremaps osm import
baremaps osm update
baremaps tiles export
baremaps tiles serve
baremaps tiles preview
baremaps location serve
baremaps search serve
baremaps route serve
...

Add filtering capabilities to the Tile Service

For instance, a subset of the layers could be returned by passing parameters in the URL or in the query parameters.

/z/x/y.pbf?layers=layer1+layer2+layer3

If no filter is specified, all the layers are returned in the vector tile.

WARN: Unable to build the geometry for some relations

During some imports, I got many of these warnings: Unable to build the geometry for relation XXXXXX

Here's a log dump when such even occurs:

(...)
Import openstreetmap
--2021-02-18 15:18:36--  https://download.geofabrik.de/europe/switzerland-latest.osm.pbf
Resolving download.geofabrik.de (download.geofabrik.de)... 116.202.112.212, 88.99.142.44
Connecting to download.geofabrik.de (download.geofabrik.de)|116.202.112.212|:443... connected.
HTTP request sent, awaiting response... 304 Not Modified
File ‘switzerland-latest.osm.pbf’ not modified on server. Omitting download.

[INFO ] 2021-02-18 15:18:37.186 [main] Execute - 56 processors available
[INFO ] 2021-02-18 15:18:37.205 [main] Execute - ../queries/osm_create_extensions.sql
[INFO ] 2021-02-18 15:18:37.302 [main] Execute - ../queries/osm_drop_views.sql
[INFO ] 2021-02-18 15:18:37.319 [main] Execute - ../queries/osm_drop_tables.sql
[INFO ] 2021-02-18 15:18:37.321 [main] Execute - ../queries/osm_create_tables.sql
[INFO ] 2021-02-18 15:18:40.281 [main] Import - 56 processors available
[INFO ] 2021-02-18 15:18:40.558 [main] ImportTask - Creating cache
[INFO ] 2021-02-18 15:18:45.616 [ForkJoinPool.commonPool-worker-60] ProgressLogger - 3.2%
[INFO ] 2021-02-18 15:18:50.679 [ForkJoinPool.commonPool-worker-52] ProgressLogger - 5.19%
(...)
[INFO ] 2021-02-18 15:22:46.344 [ForkJoinPool.commonPool-worker-63] ProgressLogger - 91.96%
[INFO ] 2021-02-18 15:22:54.597 [main] ImportTask - Importing data
[INFO ] 2021-02-18 15:23:03.576 [ForkJoinPool.commonPool-worker-27] ProgressLogger - 6.52%
[INFO ] 2021-02-18 15:23:10.033 [ForkJoinPool.commonPool-worker-43] ProgressLogger - 10.67%
(...)
[INFO ] 2021-02-18 15:25:31.238 [ForkJoinPool.commonPool-worker-12] ProgressLogger - 94.32%
[INFO ] 2021-02-18 15:25:39.018 [ForkJoinPool.commonPool-worker-38] ProgressLogger - 95.6%
[WARN ] 2021-02-18 15:25:47.889 [ForkJoinPool.commonPool-worker-48] GeometryHandler - Unable to build the geometry for relation 3362179
java.lang.NullPointerException: null
	at org.locationtech.jts.algorithm.locate.IndexedPointInAreaLocator.locate(IndexedPointInAreaLocator.java:88) ~[jts-core-1.18.0.jar:?]
	at org.locationtech.jts.operation.polygonize.EdgeRing.isInRing(EdgeRing.java:261) ~[jts-core-1.18.0.jar:?]
	at org.locationtech.jts.operation.polygonize.EdgeRing.findEdgeRingContaining(EdgeRing.java:87) ~[jts-core-1.18.0.jar:?]
	at org.locationtech.jts.operation.polygonize.HoleAssigner.findShellContaining(HoleAssigner.java:106) ~[jts-core-1.18.0.jar:?]
	at org.locationtech.jts.operation.polygonize.HoleAssigner.assignHoleToShell(HoleAssigner.java:78) ~[jts-core-1.18.0.jar:?]
	at org.locationtech.jts.operation.polygonize.HoleAssigner.assignHolesToShells(HoleAssigner.java:72) ~[jts-core-1.18.0.jar:?]
	at org.locationtech.jts.operation.polygonize.HoleAssigner.assignHolesToShells(HoleAssigner.java:40) ~[jts-core-1.18.0.jar:?]
	at org.locationtech.jts.operation.polygonize.Polygonizer.polygonize(Polygonizer.java:255) ~[jts-core-1.18.0.jar:?]
	at org.locationtech.jts.operation.polygonize.Polygonizer.getGeometry(Polygonizer.java:188) ~[jts-core-1.18.0.jar:?]
	at com.baremaps.osm.geometry.GeometryHandler.handle(GeometryHandler.java:104) ~[baremaps-osm-0.4.0.jar:?]
	at com.baremaps.osm.domain.Relation.accept(Relation.java:47) ~[baremaps-osm-0.4.0.jar:?]
	at com.baremaps.osm.handler.EntityHandler.accept(EntityHandler.java:20) ~[baremaps-osm-0.4.0.jar:?]
	at com.baremaps.osm.handler.EntityHandler.accept(EntityHandler.java:15) ~[baremaps-osm-0.4.0.jar:?]
	at java.util.function.Consumer.lambda$andThen$0(Consumer.java:65) ~[?:1.8.0_275]
	at java.util.ArrayList.forEach(ArrayList.java:1259) [?:1.8.0_275]
	at com.baremaps.osm.handler.BlockEntityHandler.handle(BlockEntityHandler.java:27) [baremaps-osm-0.4.0.jar:?]
	at com.baremaps.osm.domain.DataBlock.handle(DataBlock.java:39) [baremaps-osm-0.4.0.jar:?]
	at com.baremaps.osm.handler.BlockHandler.accept(BlockHandler.java:15) [baremaps-osm-0.4.0.jar:?]
	at com.baremaps.osm.handler.BlockHandler.accept(BlockHandler.java:9) [baremaps-osm-0.4.0.jar:?]
	at java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:439) [?:1.8.0_275]
	at java.util.ArrayList$Itr.forEachRemaining(ArrayList.java:901) [?:1.8.0_275]
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) [?:1.8.0_275]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) [?:1.8.0_275]
	at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) [?:1.8.0_275]
	at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) [?:1.8.0_275]
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [?:1.8.0_275]
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) [?:1.8.0_275]
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) [?:1.8.0_275]
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) [?:1.8.0_275]
(...)

Not quite sure where to search from here...

Typical examples are:
https://www.openstreetmap.org/relation/2585841
https://www.openstreetmap.org/relation/11312837
https://www.openstreetmap.org/relation/10032656
https://www.openstreetmap.org/relation/11485685

Improve the documentation of the export command

I'm not sure to fully understand the what the export command need as arguments to properly work:

# baremaps  export  \
--database 'jdbc:postgresql://'${HOST}':'${PORT}'/'${POSTGRES_DB}'?user='${POSTGRES_USER}'&password='${POSTGRES_PASSWORD} \
--config "${config_file}" \
--log-level=DEBUG \
--repository=./data/tiles

[INFO ] 2021-03-09 08:22:58.654 [main] Export - 56 processors available
[INFO ] 2021-03-09 08:22:58.681 [main] Export - Reading configuration
[DEBUG] 2021-03-09 08:22:58.795 [main] FileBlobStore - Read 
java.io.IOException: Is a directory
...

Here is the help:

Usage: baremaps export [--enable-s3] [--mbtiles] [--batch-array-index=READER]
                       [--batch-array-size=BATCH_ARRAY_SIZE] --config=YAML
                       --database=DATABASE [--log-level=LOG_LEVEL]
                       --repository=URL [--tiles=TILES]
Export vector tiles from the database.
      --batch-array-index=READER   The index of the batch in the array.
      --batch-array-size=BATCH_ARRAY_SIZE
                                   The size of the batch array.
      --config=YAML                The YAML source configuration file.
      --database=DATABASE          The JDBC url of the Postgres database.
      --enable-s3                  Enable Amazon S3 integration.
      --log-level=LOG_LEVEL        The log level.
      --mbtiles                    The repository is in the MBTiles format.
      --repository=URL             The tile repository URL.
      --tiles=TILES                The tiles to export.

Why is the --repository expecting a URL? Can't we just simply save the tiles to the file system? How do we achieve that then?
And what exactly are the --tiles and --mbtiles flags doing? Maybe I must use one of those instead, but the --repository flag seems mandatory and if I set a path to the -mbtiles one I got:

Invalid value for option '--mbtiles': './data/tiles' is not a boolean

which seems to suffer a tiny offset in what I understand from the documentation.

Maybe it could be nice to also print out the expected values for each args, as many bash command help pages are actually doing.

Thanks for having a look!

Improve the configuration file for producing tiles

The config/config.yaml file is used to generate vector tiles. It contains SQL queries for each zoom levels. As for now, the example provided only specifies queries for the zoom level 14. The idea would be to add queries for zoom levels 0 to 13, taking inspiration from what OpenStreetMap does for raster tiles.

https://www.openstreetmap.ch/
https://github.com/gravitystorm/openstreetmap-carto
https://wiki.openstreetmap.org/wiki/Features

Link from 'and supported by Camptocamp.' on https://www.baremaps.com leads to a 404

The link goes to https://www.camptocamp.com/en/solutions/geospatial-solutions/, but should (probably) go to https://www.camptocamp.com/geospatial_solutions

Add a command line parameter for the location of the LMDB cache

Right now, the command uses createTempDirectory. This is a good default value, but we shall be able to specify the location of the LMDB cache, especially if the host has several hard drives with different performance characteristics (HDD, SSD, see #97). Additionally, we should ensure that the cache is deleted when the execution terminates.

Add an executable for windows in the bin directory

The current distribution contains a bash script. The equivalent for windows systems should be created. Probably something like a baremaps.bat file containing:

@echo off
set dir=%~dp0%
set LIB="%dir%..\lib\*"
java -cp "$DIR/../lib/*" com.baremaps.cli.Baremaps %*

the code missing the package of com.baremaps.osm.binary

hi，sir， i download the latest code of the baremaps，but in the com.baremaps.osm.pbf.FileBlock,it looks like missing com.baremaps.osm.binary.*

Import data from IGN DB Topo in shapefiles format

Hi,
Thanks for this very good and interresting project.
I want import data in shapefile format from IGN DB Topo (https://geoservices.ign.fr/documentation/diffusion/telechargement-donnees-libres.html#bd-topo).
Since 1 January 2021, all public data (vector and ortho-images) from the National Institute of Geographic and Forest Information (IGN) are free and accessible free of charge (https://www.actuia.com/english/the-igns-public-data-relating-to-topography-relief-and-visualization-of-the-territory-are-free-and-open-to-the-public/)

Cdt
Youcef HILEM

Import StreamProgress over 100%

Hello,

I am trying to import planet.pbf (55GB) using the OSMVecto example. According to log output, the import is at over 1800%:

[INFO ] 2020-12-17 08:51:28.203 [main] StreamProgress - Progress: 1850.24%

I'm running out of disk space since the import seems to require a lot (currently ~1.3TB and growing) and it's difficult to estimate the disk requirements for the full import with >100% progress indicator.

Dead link in readme/website

The page https://www.camptocamp.com/en/solutions/geospatial-solutions/ is dead.

Referenced in README.md and https://www.baremaps.com/.

Improve the style of the map

The /config/style.json file is used to render the vector tiles in mapbox. It can easily be edited with maputnik. As for now, the style provided does not support all the zoom levels. The idea would be improve this style, taking inspiration from what OpenStreetMap does for raster tiles.

Set environment variable in yaml and sql files

The description of this issue comes from openstreetmap-vecto:
baremaps/openstreetmap-vecto#24

On the Docker Hub page, configure the link to the github repo hosting the Dockerfile

It feels quite empty on the Docker Hub, e.g. here https://hub.docker.com/r/baremaps/baremaps
compared to, e.g. https://hub.docker.com/r/osgeo/gdal
where one can spot the github page hosting the Dockerfile which has been used to build''n'push the image to the hub:

it's a "nice to have" feature, but an immensely useful one, especially for newcomers imho.

Handle empty tiles more properly

Empty tiles raises errors in the browser:

evented.js:109 Error: Error: http status 200 returned without content.

We should find a way to handle this case more properly

Add allow and block lists for tags and entities

As a data integrator
I want to import only the data that my style requires
Because I want to reduce the costs of storage and the import time

use case
I want to create a style for a basemap, that does not contain the buildings, but on a large dataset (planet OSM). I would like then to add a blacklist (in the config file?) where I list the tags related to buildings, so that in the end they are not imported in the PG database.

service URL should be more parametrable

To deploy baremaps-server in an environment with complex http routing, It would be probably beneficial to be able to parametrize the service route instead of having it hardcoded as /tiles.

See this point: https://github.com/baremaps/baremaps/blob/master/baremaps-server/src/main/java/com/baremaps/server/StyleBuilder.java#L54

java.nio.file.InvalidPathException when using the LocalFileSystem in Windows

I am working with Windows 10 and JDK 14. When using the LocalFileSystem in the application I run into the following exception when parsing a Windows File path passed from the CLI to LocalFileSystem:

java.nio.file.InvalidPathException: Illegal char <:> at index 2:

It happens when Path path = Paths.get(uri.getPath()); on Windows is called. I saw several similar bugs in the internet, and some fixed this by resolving the path like this: Paths.get(new File(uri).getAbsolutePath()), but I am unsure if it also works for Mac OS and Linux`.

That's why I am opening an issue instead of directly making a PR.

The LocalFileSystem then looks like this:

import com.baremaps.util.vfs.FileSystem;

import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URI;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;

public class LocalFileSystem extends FileSystem {

    @Override
    public boolean accept(URI uri) {
        return uri.getHost() == null && uri.getPath() != null;
    }

    @Override
    public InputStream read(URI uri) throws IOException {
        Path path = getPathFromUri(uri);

        return Files.newInputStream(path);
    }

    @Override
    public byte[] readByteArray(URI uri) throws IOException {
        Path path = getPathFromUri(uri);

        return Files.readAllBytes(path);
    }

    @Override
    public OutputStream write(URI uri) throws IOException {
        Path path = getPathFromUri(uri);

        if (!Files.exists(path.getParent())) {
            Files.createDirectories(path.getParent());
        }
        return Files.newOutputStream(path);
    }

    @Override
    public void writeByteArray(URI uri, byte[] bytes) throws IOException {

        Path path = getPathFromUri(uri);

        if (!Files.exists(path.getParent())) {
            Files.createDirectories(path.getParent());
        }
        Files.write(path, bytes);
    }

    @Override
    public void delete(URI uri) throws IOException {
        Path path = getPathFromUri(uri);

        Files.deleteIfExists(path);
    }

    private Path getPathFromUri(URI uri) {
        return Paths.get(new File(uri).getAbsolutePath());
    }
}

Add progress indications in the export command

Hello, my question is pretty simple :

Is there a way to know or estimate the export time ?

❗ I'm currently exporting since 7 hours now, for a DB based on the openstreemap-vecto project, for a ~120MB osm.pbf file on a 2vcore VPS.

❓ I know it's a light machine and that I must except a looong export time, but I'm curious about what influence the export time and if there is some known numbers (export benchmarks ?).

By the way, I'm doing all of that in a Docker container which mirror the tiles/ (export) folder onto the host machine, if that makes a difference (even if I think it's more of a CPU bottleneck than a disk one but anyway).

Thanks by advance for you help. 👍

Illegal reflective access by org.lmdbjava.ByteBufferProxy$AbstractByteBufferProxy (file:/opt/baremaps/lib/lmdbjava-0.8.1.jar) to field java.nio.Buffer.address

Hi there,

I'm following the OSM tutorial, and at this stage:

$ baremaps import \
>   --database 'jdbc:postgresql://localhost:5432/baremaps?&user=baremaps&password=baremaps' \
>   --file 'https://download.geofabrik.de/europe/liechtenstein-latest.osm.pbf'

I faced these warnings:

[INFO ] 2021-02-05 09:31:14.555 [main] Import - 56 processors available
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.lmdbjava.ByteBufferProxy$AbstractByteBufferProxy (file:/opt/baremaps/lib/lmdbjava-0.8.1.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of org.lmdbjava.ByteBufferProxy$AbstractByteBufferProxy
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[INFO ] 2021-02-05 09:31:14.777 [main] HttpBlobStore - Read https://download.geofabrik.de/europe/liechtenstein-latest.osm.pbf

Just wondering if it's normal?

Environment:

Ubuntu 18.04.5 LTS 5.4.0-65-generic x86_64 GNU/Linux

Release used: https://github.com/baremaps/baremaps/releases/tag/v0.3.4

Malformed SQL queries should be printed to stdout for debugging purpose

As a developer,
I'd like to be able to see the rendered SQL query on stdout,
so that I can copy-paste it in my pg admin tool to debug it.

For the moment, I'm unable to spot what's wrong from such error messages (printed to stdout on server's side) when browsing the app:

[WARN ] 2021-02-21 15:06:32.103 [armeria-common-worker-epoll-2-7] HttpResponseSubscriber - [id: 0x8de6f642, L:/172.28.0.3:9000 - R:/10.193.21.89:42688] Unexpected exception from a service or a response publisher: com.baremaps.server.TileService@258a2a21
java.lang.IllegalArgumentException: The SQL query malformed
	at com.baremaps.tile.postgres.PostgisQueryParser.parse(PostgisQueryParser.java:52) ~[baremaps-tile-postgres-0.4.0.jar:?]
	at com.baremaps.tile.postgres.PostgisTileStore.lambda$null$0(PostgisTileStore.java:137) ~[baremaps-tile-postgres-0.4.0.jar:?]

The SQL query is apparently still missing when --log-level=DEBUG is enabled:

[DEBUG] 2021-02-22 11:02:58.713 [pool-2-thread-10] FileBlobStore - Read config-rabbits.yml
[DEBUG] 2021-02-22 11:02:58.713 [pool-2-thread-6] FileBlobStore - Read config-rabbits.yml
[DEBUG] 2021-02-22 11:02:58.718 [pool-2-thread-6] FileBlobStore - Read config/rabbits/1_differential_rabbiHousing/layer.yml
[DEBUG] 2021-02-22 11:02:58.718 [pool-2-thread-10] FileBlobStore - Read config/rabbits/1_differential_rabbiHousing/layer.yml
[DEBUG] 2021-02-22 11:02:58.720 [pool-2-thread-6] FileBlobStore - Read config/rabbits/1_differential_rabbiHousing/stylesheet.yml
[DEBUG] 2021-02-22 11:02:58.720 [pool-2-thread-10] FileBlobStore - Read config/rabbits/1_differential_rabbiHousing/stylesheet.yml

Create a serve command for production use cases

I just made a few tests on my laptop to check how many tiles/s baremaps can handle in serve mode. I used the lichtenstein example "as is". In my tests baremaps cannot handle more than 10rps (requests per second). If used with more (say 12rps) the success rate drastically drops to 20-25 % and we get 75% of failed connections.

This behaviour is consistent with the cpu which was completely used, and probably it was no more possible to create new connections with the server.

up to 10 rps we get:

['baremap-10rps.log']
response time [ms] statistics for test : '10rps' 
mean : 1389.6041366666666
median : 388.16700000000003
min-max-range : 9150.050000000001
percentage-of-http-200 : 100.0
percentage-of-http-errors : 0.0
percentage-of-network-errors : 0.0
percentile90 : 4911.853500000011
percentile95 : 6935.2443
std : 2162.730614611565

and with 12rps:

['baremap-12rps.log']
response time [ms] statistics for test : '12rps' 
mean : 2418.637625
median : 2029.3229999999999
min-max-range : 9834.434
percentage-of-http-200 : 24.444444444444443
percentage-of-http-errors : 0.0
percentage-of-network-errors : 75.55555555555556
percentile90 : 5562.04
percentile95 : 6381.597750000001
std : 2206.189541154367

the test setup (not optimal) is my laptop pc:

ubuntu 18.04 64bit
CPU
postgresql 12 + postgis installed on local machine
baremaps v.0.3.4

to test I used yandex-tank with the input files provided as attachement yandex-config.zip. To run yandex, just do

docker run \
    -v $(pwd):/var/loadtest \
    --net host \
    -it \
    --entrypoint /bin/bash \
    direvius/yandex-tank

and in the container:

yandex-tank -c shoot-baremaps.ini baremaps.ammo

PostgisReferenceCache throws IllegalArgumentException

While I am experimenting with Windows, I have noticed, that the osm_relations table is always empty after the import. I think the problem is due to the PostgisReferenceCache implementation I am using.

In PostgisReferenceCache#get an IllegalArgumentException is thrown when the Reference does not exist:

https://github.com/baremaps/baremaps/blob/40409acd91d002f16297432527fd92f28d072957/baremaps-osm/src/main/java/com/baremaps/osm/cache/PostgisReferenceCache.java#L57

I think this will cause the whole stream to cancel when it is thrown in the RelationBuilder#build:

https://github.com/baremaps/baremaps/blob/40409acd91d002f16297432527fd92f28d072957/baremaps-osm/src/main/java/com/baremaps/osm/geometry/RelationBuilder.java#L69

Can't the Cache return null, when the Element is not available? Or should the call in the stream be wrapped with Try.of(...)?

Configure OSX and Windows builds in Github Action

This would ensure that baremaps works on every major operating systems (see #84).

Serve issue if no S3 configuration

serve process expects to have an AWS S3 configuration on the hosting machine, while it should run even though you don't have S3 configured.

Odd coordinates after importing maps

Hi!

I noticed a peculiar thing with coordinates when running baremaps to PostgreSQL database and thought to ask about the purpose or if there's something strange here.

The setup is:

Custom PostgreSQL.
Baremaps 0.32.
Windows Server 2019, 64-bit, fi-FI.

Not sure if a language issue. If I change language settings, Baremaps stops working (LMDB errors with code 112, the ava wrapper does not understand it, Baremaps stops). Maybe I check a bit later.

As for an example for https://download.geofabrik.de/europe/liechtenstein-latest.osm.pbf

SELECT * FROM public.osm_ways WHERE id = 294942376;
SELECT * FROM public.osm_nodes WHERE id IN (2985838396, 2985838357, 2985838346, 2985838390, 2985838517, 2985838501, 2985838396);
SELECT ST_AsText(geom) FROM public.osm_nodes WHERE id = 2985838396;
SELECT ST_SRID(geom) FROM public.osm_nodes WHERE id = 2985838396;
SELECT ST_AsText(geom) FROM public.osm_ways WHERE id = 294942376;

to a database that is generated from https://download.geofabrik.de/europe/liechtenstein-latest.osm.pbf. It's that the coordinates look very large. Here are the results from the SQL queries

id	version	uid	timestamp	changeset	tags	nodes	geom
294942376	1	0	2014-07-29T11:56:56	0	"building"=>"yes"	[2985838396, 2985838357, 2985838346, 2985838390, 2985838517, 2985838501, 2985838396]	0103000020110F000001000000070000001C92315647273041BF750E0517C45641EA7078054B27304113BCC09010C45641148CE30540273041FE3D2B2C10C456417DDC054F3E273041ACE9432E13C456411B446521382730418707BCF612C45641E414FC2836273041B0C2E46716C456411C92315647273041BF750E0517C45641

id	version	timestamp	tags	lon	lat	geom
2985838346	1	2014-07-29T11:56:50	NULL	9.509781400000001	47.1582123	0101000020110F0000148CE30540273041FE3D2B2C10C45641
2985838357	1	2014-07-29T11:56:50	NULL	9.509880200000001	47.1582219	0101000020110F0000EA7078054B27304113BCC09010C45641
2985838390	1	2014-07-29T11:56:51	NULL	9.509766	47.1582858	0101000020110F00007DDC054F3E273041ACE9432E13C45641
2985838396	1	2014-07-29T11:56:51	NULL	9.5098471	47.1583796	0101000020110F00001C92315647273041BF750E0517C45641
2985838501	1	2014-07-29T11:56:52	NULL	9.5096928	47.158364600000006	0101000020110F0000E414FC2836273041B0C2E46716C45641
2985838517	1	2014-07-29T11:56:53	NULL	9.5097105	47.158280500000004	0101000020110F00001B446521382730418707BCF612C45641

st_astext POINT(1058631.33669389 5967964.07900757)
st_srid 3857
POLYGON((1058631.33669389 5967964.07900757,1058635.02136903 5967938.26176359,1058624.02300334 5967936.69013929,1058622.30868319 5967948.72289507,1058616.13045145 5967947.8552264,1058614.16009646 5967961.62333743,1058631.33669389 5967964.07900757))

For instance, taking one point from there should be at https://www.openstreetmap.org/node/2985838396 and the geometry it produces probbly should be POLYGON((9.5098471 47.1583796,9.5098802 47.1582219,9.5097814 47.1582123,9.509766 47.1582858,9.5097105 47.1582805,9.5096928 47.1583646,9.5098471 47.1583796)) instead of that with the odd coorginates. At least this is how it looks to me when I plug the polygon, say, to http://arthur-e.github.io/Wicket/sandbox-gmaps3.html.

Issue importing data and serving tiles

I was able to eventually get through the steps outlined in https://www.baremaps.com/examples/openstreetmap/ to show a map of Liechtenstein with the Blueprint style.

I've been trying to get this to work with Switzerland and the Openstreetmap-vecto style. When I import the switzerland-latest.osm.pbf file, I encountered exceptions such as:

[WARN ] 2021-01-28 11:20:12.881 [ForkJoinPool.commonPool-worker-83] GeometryHandler - Unable to build the geometry for relation 1713904
java.lang.NullPointerException: null
at org.locationtech.jts.algorithm.locate.IndexedPointInAreaLocator.locate(IndexedPointInAreaLocator.java:88) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.EdgeRing.isInRing(EdgeRing.java:261) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.EdgeRing.findEdgeRingContaining(EdgeRing.java:87) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.HoleAssigner.findShellContaining(HoleAssigner.java:106) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.HoleAssigner.assignHoleToShell(HoleAssigner.java:78) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.HoleAssigner.assignHolesToShells(HoleAssigner.java:72) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.HoleAssigner.assignHolesToShells(HoleAssigner.java:40) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.Polygonizer.polygonize(Polygonizer.java:255) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.Polygonizer.getGeometry(Polygonizer.java:188) ~[jts-core-1.17.1.jar:?]
at com.baremaps.importer.geometry.GeometryHandler.handle(GeometryHandler.java:104) ~[baremaps-importer-0.3.4.jar:?]
at com.baremaps.osm.domain.Relation.accept(Relation.java:47) ~[baremaps-osm-0.3.4.jar:?]
at com.baremaps.osm.EntityHandler.accept(EntityHandler.java:20) ~[baremaps-osm-0.3.4.jar:?]
at com.baremaps.osm.EntityHandler.accept(EntityHandler.java:15) ~[baremaps-osm-0.3.4.jar:?]
at java.util.function.Consumer.lambda$andThen$0(Consumer.java:65) ~[?:?]
at java.util.ArrayList.forEach(ArrayList.java:1541) [?:?]
at com.baremaps.osm.BlockEntityHandler.handle(BlockEntityHandler.java:27) [baremaps-osm-0.3.4.jar:?]
at com.baremaps.osm.pbf.DataBlock.handle(DataBlock.java:42) [baremaps-osm-0.3.4.jar:?]
at com.baremaps.osm.BlockHandler.accept(BlockHandler.java:15) [baremaps-osm-0.3.4.jar:?]
at com.baremaps.osm.BlockHandler.accept(BlockHandler.java:9) [baremaps-osm-0.3.4.jar:?]

and then error messages such as:

[WARN ] 2021-01-28 11:20:32.414 [ForkJoinPool.commonPool-worker-123] GeometryHandler - Unable to build the geometry for relation 7521607
java.lang.NullPointerException: null
[WARN ] 2021-01-28 11:20:37.100 [ForkJoinPool.commonPool-worker-15] GeometryHandler - Unable to build the geometry for relation 5503310
java.lang.NullPointerException: null

It's unclear to me whether the import job finished despite the errors, and perhaps gave up after encountering the errors. Either way, when I try to run the server, I get exceptions such as:

[ERROR] 2021-01-28 11:47:05.277 [armeria-common-worker-epoll-2-6] TileService - org.postgresql.util.PSQLException: ERROR: relation "osm_ways_z12" does not exist
Position: 320
[ERROR] 2021-01-28 11:47:05.276 [armeria-common-worker-epoll-2-5] TileService - org.postgresql.util.PSQLException: ERROR: relation "osm_relations_z12" does not exist
Position: 320
[ERROR] 2021-01-28 11:47:05.276 [armeria-common-worker-epoll-2-4] TileService - org.postgresql.util.PSQLException: ERROR: relation "osm_water_polygons" does not exist
Position: 335

I believe something similar happened when I tried to import us-west. My larger goal is to import the OSM data for the world.

On a tangential note, I have used imposm3 with some success before; could Baremaps' server be made to work with imposm imports? (e.g. specifying an appropriate mapping file)

Thank you.

Create a search service (GRPC) based on Lucene

Make configuration readable and writable from the application

Publish SDKMAN release with github actions

Investigate the use of curl and of the maven plugin.

https://sdkman.io/vendors
https://github.com/sdkman/sdkman-vendor-maven-plugin

Execute integration tests with Github Action

(...and remove circleci builds)

Add batch index and batch number to the command line client

AWS Batch allows to distribute the computation of embarrassingly parallel problems. The creation of vector tiles is embarrassingly parallel. Adding command line arguments for the number of batches and the batch to be processed would allow baremaps to be deployed on AWS batch as a docker image.

LMDB error on Windows

[INFO ] 2020-12-07 16:10:20.906 [main] Import - 16 processors available
java : Exception in thread "main" java.lang.UnsatisfiedLinkError: unknown
At line:14 char:1
+ java -cp "C:\baremaps\lib\*" com.baremaps.cli.Baremaps import --datab ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Exception in th...kError: unknown:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
	at jnr.ffi.provider.jffi.NativeLibrary.loadNativeLibraries(NativeLibrary.java:93)
	at jnr.ffi.provider.jffi.NativeLibrary.getNativeLibraries(NativeLibrary.java:71)
	at jnr.ffi.provider.jffi.NativeLibrary.getSymbolAddress(NativeLibrary.java:50)
	at jnr.ffi.provider.jffi.NativeLibrary.findSymbolAddress(NativeLibrary.java:60)
	at jnr.ffi.provider.jffi.AsmLibraryLoader.generateInterfaceImpl(AsmLibraryLoader.java:138)
	at jnr.ffi.provider.jffi.AsmLibraryLoader.loadLibrary(AsmLibraryLoader.java:86)
	at jnr.ffi.provider.jffi.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:44)
	at jnr.ffi.LibraryLoader.load(LibraryLoader.java:392)
	at jnr.ffi.LibraryLoader.load(LibraryLoader.java:371)
	at org.lmdbjava.Library.<clinit>(Library.java:125)
	at org.lmdbjava.Env$Builder.open(Env.java:486)
	at org.lmdbjava.Env$Builder.open(Env.java:512)
	at com.baremaps.cli.Import.call(Import.java:116)
	at com.baremaps.cli.Import.call(Import.java:45)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1933)
	at picocli.CommandLine.access$1200(CommandLine.java:145)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2332)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2326)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2291)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2159)
	at picocli.CommandLine.execute(CommandLine.java:2058)
	at com.baremaps.cli.Baremaps.main(Baremaps.java:43)'

It appears I miss some native library, but I do do not know which one. I dropped https://jdbc.postgresql.org/download.html to the lib directory, just in case (didn't help).

Export in the mbtiles format

MBTiles is a specification for storing arbitrary tiled map data in SQLite databases for immediate usage and for efficient transfer.

https://github.com/mapbox/mbtiles-spec

Clean and update the docker directory

Clean the docker directory form unecessary images (e.g. postgis24, java8)
Use java 11 instead of java 8in the baremaps image
Update postgis and ensure that postgres.conf file is taken into account
Rebuild and push the docker images with github action upon the publication of a new release tag (e.g. v1.0.0)

@nicolas-heigvd do not hesitate to list here all the ideas you may have regarding docker's best practices. For instance, a possibility may be to use the kubernetes-maven-plugin to create docker images and k8s configurations.

baremaps/openstreetmap-vecto#9

Don't create a parallel Stream if parallel==false

https://github.com/baremaps/baremaps/blob/860396be5648d060712e5dedea0e5feb5f25a21a/baremaps-osm/src/main/java/com/baremaps/osm/reader/pbf/FileBlockStreamer.java#L24

Create a routing service (GRPC) based on Graphhopper

Create a location service (GRPC) based on GeoIP

Display version number (--version flag)

The CLI should be able to display the version number set in the pom.xml file.

Encoding issue in blueprint.html

If you put accent in blueprint.html, the the encoding is not correct.

Make data import transactional

The current approach for importing data drops the tables, creates a new one, fills them, creates the indexes, etc. This means that the database is in an unusable state for some time. It would be a good thing to perform these operations within a transaction to ensure that the database remains in a usable state.

Check if the query parser could be replaced by the JOOQ or JSqlParser

https://www.jooq.org/doc/latest/manual/sql-building/sql-parser/

https://www.jooq.org/doc/latest/manual/sql-building/dsl-context/custom-settings/settings-parser/

Set Settings.parseUnknownFunctions to IGNORE

Publish the docker images with github action

The docker images are created and published manually. This may be automated with the (jib-maven-plugin)[https://github.com/GoogleContainerTools/jib/tree/master/jib-maven-plugin#quickstart] and with github actions.