Giter VIP home page Giter VIP logo

openstreetmap-tile-server's Introduction

openstreetmap-tile-server

Build Status Docker Image Version (latest semver)

This container allows you to easily set up an OpenStreetMap PNG tile server given a .osm.pbf file. It is based on the latest Ubuntu 18.04 LTS guide from switch2osm.org and therefore uses the default OpenStreetMap style.

Setting up the server

First create a Docker volume to hold the PostgreSQL database that will contain the OpenStreetMap data:

docker volume create osm-data

Next, download an .osm.pbf extract from geofabrik.de for the region that you're interested in. You can then start importing it into PostgreSQL by running a container and mounting the file as /data/region.osm.pbf. For example:

docker run \
    -v /absolute/path/to/luxembourg.osm.pbf:/data/region.osm.pbf \
    -v osm-data:/data/database/ \
    overv/openstreetmap-tile-server \
    import

If the container exits without errors, then your data has been successfully imported and you are now ready to run the tile server.

Note that the import process requires an internet connection. The run process does not require an internet connection. If you want to run the openstreetmap-tile server on a computer that is isolated, you must first import on an internet connected computer, export the osm-data volume as a tarfile, and then restore the data volume on the target computer system.

Also when running on an isolated system, the default index.html from the container will not work, as it requires access to the web for the leaflet packages.

Automatic updates (optional)

If your import is an extract of the planet and has polygonal bounds associated with it, like those from geofabrik.de, then it is possible to set your server up for automatic updates. Make sure to reference both the OSM file and the polygon file during the import process to facilitate this, and also include the UPDATES=enabled variable:

docker run \
    -e UPDATES=enabled \
    -v /absolute/path/to/luxembourg.osm.pbf:/data/region.osm.pbf \
    -v /absolute/path/to/luxembourg.poly:/data/region.poly \
    -v osm-data:/data/database/ \
    overv/openstreetmap-tile-server \
    import

Refer to the section Automatic updating and tile expiry to actually enable the updates while running the tile server.

Please note: If you're not importing the whole planet, then the .poly file is necessary to limit automatic updates to the relevant region. Therefore, when you only have a .osm.pbf file but not a .poly file, you should not enable automatic updates.

Letting the container download the file

It is also possible to let the container download files for you rather than mounting them in advance by using the DOWNLOAD_PBF and DOWNLOAD_POLY parameters:

docker run \
    -e DOWNLOAD_PBF=https://download.geofabrik.de/europe/luxembourg-latest.osm.pbf \
    -e DOWNLOAD_POLY=https://download.geofabrik.de/europe/luxembourg.poly \
    -v osm-data:/data/database/ \
    overv/openstreetmap-tile-server \
    import

Using an alternate style

By default the container will use openstreetmap-carto if it is not specified. However, you can modify the style at run-time. Be aware you need the style mounted at run AND import as the Lua script needs to be run:

docker run \
    -e DOWNLOAD_PBF=https://download.geofabrik.de/europe/luxembourg-latest.osm.pbf \
    -e DOWNLOAD_POLY=https://download.geofabrik.de/europe/luxembourg.poly \
    -e NAME_LUA=sample.lua \
    -e NAME_STYLE=test.style \
    -e NAME_MML=project.mml \
    -e NAME_SQL=test.sql \
    -v /home/user/openstreetmap-carto-modified:/data/style/ \
    -v osm-data:/data/database/ \
    overv/openstreetmap-tile-server \
    import

If you do not define the "NAME_*" variables, the script will default to those found in the openstreetmap-carto style.

Be sure to mount the volume during run with the same -v /home/user/openstreetmap-carto-modified:/data/style/

If you do not see the expected style upon run double check your paths as the style may not have been found at the directory specified. By default, openstreetmap-carto will be used if a style cannot be found

Only openstreetmap-carto and styles like it, eg, ones with one lua script, one style, one mml, one SQL can be used

Running the server

Run the server like this:

docker run \
    -p 8080:80 \
    -v osm-data:/data/database/ \
    -d overv/openstreetmap-tile-server \
    run

Your tiles will now be available at http://localhost:8080/tile/{z}/{x}/{y}.png. The demo map in leaflet-demo.html will then be available on http://localhost:8080. Note that it will initially take quite a bit of time to render the larger tiles for the first time.

Using Docker Compose

The docker-compose.yml file included with this repository shows how the aforementioned command can be used with Docker Compose to run your server.

Preserving rendered tiles

Tiles that have already been rendered will be stored in /data/tiles/. To make sure that this data survives container restarts, you should create another volume for it:

docker volume create osm-tiles
docker run \
    -p 8080:80 \
    -v osm-data:/data/database/ \
    -v osm-tiles:/data/tiles/ \
    -d overv/openstreetmap-tile-server \
    run

If you do this, then make sure to also run the import with the osm-tiles volume to make sure that caching works properly across updates!

Enabling automatic updating (optional)

Given that you've set up your import as described in the Automatic updates section during server setup, you can enable the updating process by setting the UPDATES variable while running your server as well:

docker run \
    -p 8080:80 \
    -e REPLICATION_URL=https://planet.openstreetmap.org/replication/minute/ \
    -e MAX_INTERVAL_SECONDS=60 \
    -e UPDATES=enabled \
    -v osm-data:/data/database/ \
    -v osm-tiles:/data/tiles/ \
    -d overv/openstreetmap-tile-server \
    run

This will enable a background process that automatically downloads changes from the OpenStreetMap server, filters them for the relevant region polygon you specified, updates the database and finally marks the affected tiles for rerendering.

Tile expiration (optional)

Specify custom tile expiration settings to control which zoom level tiles are marked as expired when an update is performed. Tiles can be marked as expired in the cache (TOUCHFROM), but will still be served until a new tile has been rendered, or deleted from the cache (DELETEFROM), so nothing will be served until a new tile has been rendered.

The example tile expiration values below are the default values.

docker run \
    -p 8080:80 \
    -e REPLICATION_URL=https://planet.openstreetmap.org/replication/minute/ \
    -e MAX_INTERVAL_SECONDS=60 \
    -e UPDATES=enabled \
    -e EXPIRY_MINZOOM=13 \
    -e EXPIRY_TOUCHFROM=13 \
    -e EXPIRY_DELETEFROM=19 \
    -e EXPIRY_MAXZOOM=20 \
    -v osm-data:/data/database/ \
    -v osm-tiles:/data/tiles/ \
    -d overv/openstreetmap-tile-server \
    run

Cross-origin resource sharing

To enable the Access-Control-Allow-Origin header to be able to retrieve tiles from other domains, simply set the ALLOW_CORS variable to enabled:

docker run \
    -p 8080:80 \
    -v osm-data:/data/database/ \
    -e ALLOW_CORS=enabled \
    -d overv/openstreetmap-tile-server \
    run

Connecting to Postgres

To connect to the PostgreSQL database inside the container, make sure to expose port 5432:

docker run \
    -p 8080:80 \
    -p 5432:5432 \
    -v osm-data:/data/database/ \
    -d overv/openstreetmap-tile-server \
    run

Use the user renderer and the database gis to connect.

psql -h localhost -U renderer gis

The default password is renderer, but it can be changed using the PGPASSWORD environment variable:

docker run \
    -p 8080:80 \
    -p 5432:5432 \
    -e PGPASSWORD=secret \
    -v osm-data:/data/database/ \
    -d overv/openstreetmap-tile-server \
    run

Performance tuning and tweaking

Details for update procedure and invoked scripts can be found here link.

THREADS

The import and tile serving processes use 4 threads by default, but this number can be changed by setting the THREADS environment variable. For example:

docker run \
    -p 8080:80 \
    -e THREADS=24 \
    -v osm-data:/data/database/ \
    -d overv/openstreetmap-tile-server \
    run

CACHE

The import and tile serving processes use 800 MB RAM cache by default, but this number can be changed by option -C. For example:

docker run \
    -p 8080:80 \
    -e "OSM2PGSQL_EXTRA_ARGS=-C 4096" \
    -v osm-data:/data/database/ \
    -d overv/openstreetmap-tile-server \
    run

AUTOVACUUM

The database use the autovacuum feature by default. This behavior can be changed with AUTOVACUUM environment variable. For example:

docker run \
    -p 8080:80 \
    -e AUTOVACUUM=off \
    -v osm-data:/data/database/ \
    -d overv/openstreetmap-tile-server \
    run

FLAT_NODES

If you are planning to import the entire planet or you are running into memory errors then you may want to enable the --flat-nodes option for osm2pgsql. You can then use it during the import process as follows:

docker run \
    -v /absolute/path/to/luxembourg.osm.pbf:/data/region.osm.pbf \
    -v osm-data:/data/database/ \
    -e "FLAT_NODES=enabled" \
    overv/openstreetmap-tile-server \
    import

Warning: enabling FLAT_NOTES together with UPDATES only works for entire planet imports (without a .poly file). Otherwise this will break the automatic update script. This is because trimming the differential updates to the specific regions currently isn't supported when using flat nodes.

Benchmarks

You can find an example of the import performance to expect with this image on the OpenStreetMap wiki.

Troubleshooting

ERROR: could not resize shared memory segment / No space left on device

If you encounter such entries in the log, it will mean that the default shared memory limit (64 MB) is too low for the container and it should be raised:

renderd[121]: ERROR: failed to render TILE default 2 0-3 0-3
renderd[121]: reason: Postgis Plugin: ERROR: could not resize shared memory segment "/PostgreSQL.790133961" to 12615680 bytes: ### No space left on device

To raise it use --shm-size parameter. For example:

docker run \
    -p 8080:80 \
    -v osm-data:/data/database/ \
    --shm-size="192m" \
    -d overv/openstreetmap-tile-server \
    run

For too high values you may notice excessive CPU load and memory usage. It might be that you will have to experimentally find the best values for yourself.

The import process unexpectedly exits

You may be running into problems with memory usage during the import. Have a look at the "Flat nodes" section in this README.

License

Copyright 2019 Alexander Overvoorde

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

openstreetmap-tile-server's People

Contributors

always-prog avatar atychang avatar balchen avatar buddavis avatar curtis18 avatar gakowalski avatar galewis2 avatar gjacquenot avatar hhansen06 avatar istador avatar jdmonin avatar jmontleon avatar kiddikai avatar leakim avatar mhajder avatar mou97 avatar overv avatar pandel avatar ph1ll avatar prih avatar ruhepuls avatar schteph avatar sohalt avatar stevo01 avatar vpithart avatar y0ngg4n avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

openstreetmap-tile-server's Issues

Can I import from multiple .osm.pbf files ?

Hi I am trying to import whole United states, but while it requires much more memory then i have. I thought why not import it in sequence ? Is there any way I can import it one by one ? currently it is replacing old map data.

How to use Https ?

Hi,
I used default setting is OK for http
but I want to use https for this container, where can I modify config?

I already used

docker run -p 80:80 -p 443:443 -e THREADS=24 -v openstreetmap-data:/var/lib/postgresql/10/main -v openstreetmap-rendered-tiles:/var/lib/mod_tile -d overv/openstreetmap-tile-server run

to open 443 port ,but it's seen not work....

is there other way to set that ?

Thanks.

import europe - error

Hi, do you have any idea, why?: (i want to import europe-latest.osm.pbf)

Reading in file: /data.osm.pbf
Using PBF parser.
Processing: Node(1720270k 335.7k/s) Way(0k 0.00k/s) Relation(0 0.00/s)/run.sh: line 44: 101 Killed sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /home/renderer/src/openstreetmap-carto/openstreetmap-carto.lua -C 2048 --number-processes ${THREADS:-4} -S /home/renderer/src/openstreetmap-carto/openstreetmap-carto.style /data.osm.pbf

  • service postgresql stop
  • Stopping PostgreSQL 10 database server
    ...done.
  • exit 0

How could I set the style?

Hi,

Is there any way for setting another map style? Not the default OSM. Mapnik for example?

Thanks in advance!

importing more then one pbf file

Seems I cannot import more then one pbf file?

Say I want to loaf Netherlands first and America second, if I run import more then once the data in the second file is processed but never shown.

I have tried in two ways to do thiss.

First just run the import option twice once with pbf file a and second time with pbf file B,
The map only shows data from file A.

Second I immport file A (the import statement als creates the database for the first time.
Then run the import command from the running tileserver, also no luck.

Both times there are no errors and the script finishes without error.

Is it not possible to load more then one pbf file using your script?

Introduce versioning

Please add version number tags to docker images pushed to docker hub, so that it's possible to use older versions by referring to a specific version of the image, instead of always using latest.

Building upon latest is problematic for users of the image, because it isn't reproducible over time. Newer versions might introduce breaking changes or changes to the performance that might favor one setup over another. Depending on when the image was pulled, latest refers to different images on different machines.

Users inexperienced with docker might forget to docker pull when they want to use the newest latest version. Advocating the use of specific version numbers allows them to easily recognize that they are using an outdated version and that they might want to upgrade it when facing errors.

https://blogs.msdn.microsoft.com/stevelasker/2018/03/01/docker-tagging-best-practices-for-tagging-and-versioning-docker-images/


The same version numbers should also be used in this repository (e.g. as tags added to the commits), so that it's traceable from which commit the image was build from.

Downside of using git tags: Github automatically interprets them as "releases".


Optionally, this could even be done retroactively for old versions of this repository, but would require building and pushing those old images again.

Mismatch config owner and data owner

When I run the container like it is described in the readme I get the following error:

Error: Config owner (postgres:102) and data owner (renderer:1000) do not match, and config owner is not root
   ...fail!

How can this be resolved?

Challenges and questions from @stevo01

Biggest challenges that I observed:

  • Import of the whole continent ( or planet ) needs long time. Not clear how to "tune" import process for particular machine.
  • Find a strategy to keep database up to date after initial import.
  • Find a way how to reproduce the setup of tile server on different machine (docker should be the solution here right?)
  • Diagnostics: How to recognise that something is not working, indicate problems to server owner and how to find the reason.

My questions:

  • do you plan to setup and maintain own tile server or do you know someone who runs a tile server for a long period of time?
  • can we share results of our investigations and try to find "best practice" method.

Importing Europe REALLY slow

Hi Guys,
So I'm currently importing Europe (or trying to at least)

So far its been running for 2 days:

Processing: Node(2213933k 119.5k/s) Way(9196k 0.07k/s) Relation(0 0.00/s)

Is my "way" part of this running rather slow? how would i speed this up

Its running on a VM with 12 cores 16gb ram and 2tb of space

Access from browser

Just want to ask if I can access the service from browser since if want to check if the service is running. The issue I encountered is that it shows Welcome to nginx!
on localhost:80

Is it possible to add WMTS capabilities to this server

I have just set up my server and it is working great. Now I'm wondering if it is possible to extend this server to be used as WMTS? Can you give suggestions or a guide that describes the process?

I see someone here has something like that done, but I'm not sure how to configure that file to work with this server

Missing documentation

I've just tried it on an Ubuntu 18.04 server, and, first things first, something's needed to get "docker" installed. There are a number of different ways to do that and it'd be useful to know what you've previous done to get to the top of the README".

I did a "sudo apt install docker-compose" but "docker volume create openstreetmap-data" fails with permissions whereas "sudo docker volume create openstreetmap-data" does not. What system did you use when writing the README (i.e. on which systems could you follow the README with no changes and have everything work?)

Subsequently after

wget http://download.geofabrik.de/europe/luxembourg-latest.osm.pbf

I did

sudo docker run -v /home/blah/src/openstreetmap-tile-server/luxembourg.osm.pbf:/data.osm.pbf -v openstreetmap-data:/var/lib/postgresql/10/main overv/openstreetmap-tile-server import

which failed with

Reading in file: /data.osm.pbf
Using PBF parser.
node cache: stored: 0(-nan%), storage efficiency: -nan% (dense blocks: 0, sparse nodes: 0), hit rate: -nan%
Osm2pgsql failed due to ERROR: Read failed: Is a directory

Country names are hidden

Hi, I run docker and started render_list to prerender tiles

But when check the result, there is no countries name label on the map

I had manually installed osm-tile server(without container), it displayed another style of map(with country names) than your docker setup

What I did wrong, there is no errors, mapnik.xml is generated well

screenshot-195 65 227 73-2019 07 30-17-34-37

request for flatnode support

The osm2psql application allows usage of flatnodes. Extract from manual:

-F|--flat-nodes  Specifies the flat file to use to persistently store node information in slim mode
instead of in PostgreSQL. This file is a single > 40Gb large file. Only recommended for full planet
imports. Default is disabled.

The usage of this flag is a candidate for performance improvement. Additional its seems to be a workaround for the out of memory problems reportet here Issue 931 and here Issue31.

Environment variables support

Thanks a lot for sharing! Do you intend to maintain this repo and make it an easy to deploy solution listed on switch2osm, or is it just an example?

I was looking for solutions to render and host tiles on a dockerized platform, alongside other containers (CMS, db, etc.). I see that some options are hard-coded (for example luxembourg region, geofabrik.de server...), couldn't they be set from environment variables, that could be easily set in Docker compose?

tuning & monitoring

hi there, I have set up a tileserver for our wildlife protection organisation Sensing Clues (see here ).
I have set up a docker host on a 4 core 16 GB RAM server.
The tileserver is setup and I imported 7 countries ( NL, Tanzanie, Kenia, South Africa, Namibia, Zambia, Zimbabwe) I expect to load more countries soon.

My question/issue is how do monitor what is going on on the tile server?

I have pre-renderd tiles like so:

render_list -n 8 -a  -f  -m ajt  -z 9  -Z 15

And tested with some zoom levels from 0 to x but if I look at the volumes in docker with

docker system df -v

I see 57 GB of postgresql data but only see 350 MB of renderd tiles?

VOLUME NAME LINKS SIZE
9bb722d5661210d1cc218bfa29f951453f01b6344f2999fdd92661bb890bb919 1 424B
openstreetmap-data 1 57.25GB
openstreetmap-rendered-tiles 1 334.9MB

No matter what I do with render_list seems it does not pre-render any more tiles?

Memory usage:

CONTAINER ID        NAME                                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
acb8612ddd66        tileserver                          395.06%             3.697GiB / 15.66GiB   23.60%              10.9MB / 128MB      30GB / 262MB        235
69f87448cb4a        registry                            0.00%               4.355MiB / 15.66GiB   0.03%               9.73MB / 99.2MB     2.15GB / 0B         13
872f5d1e6647        portainer                           0.00%               8.332MiB / 15.66GiB   0.05%               9.91MB / 59.3MB     7.73GB / 93.3MB     14
9a5bc06c2d41        hello                               0.00%               2.086MiB / 15.66GiB   0.01%               31.9kB / 0B         4.33MB / 0B         2
84ffec2790ff        letsencrypt-nginx-proxy-companion   0.12%               11.08MiB / 15.66GiB   0.07%               130kB / 70.5kB      3.68GB / 73.7kB     18
f37924d45aa6        nginx-proxy                         0.12%               14.43MiB / 15.66GiB   0.09%               335MB / 350MB       3.69GB / 377kB      33

If i look at htop postgresql seems busy, also with some jobs over more then 3 hours ????

 renderer gis [local] select

I in a previous post i played around with THREADS and shm_size the docker-compose yml

Assumptions / Questions:

  1. As I have only 4 cores THREADS > 8 seems not usefull? I do not see any impact
  2. As I have 16 GB of RAM and this is a more or less dedicated tileserver I hardly see the memory usages exceed 25% . How can I make use of more memory?
  3. I am on postgres 10 is the latest image on 11 available? I have seen moderate speed increas but especially on the import part, does it also speed up rendering itself?
  4. Are there any ways to monitor and understand better if and how the renderd process actually has renderd tiles? How many tiles are actually prerenderd etc etc. With 335 MB in the renderd_tiles folder something seems not right?
  5. I have seen posts that pre_rendering takes a lot of time, but that is in hours not in days my tileserver is now up for quite a while (week) I have uses render_list from within docker via exec but again I do not know when it is finished what result to expect and how to optimize pre-rendering.

For completeness my current docker-compose file:

version: '3'
services:
  tileserver:
    container_name: tileserver
    image: overv/openstreetmap-tile-server
    restart: always
    ports:
        - "8099:80"
    volumes:
        - 'openstreetmap-data:/var/lib/postgresql/10/main'
        - 'openstreetmap-rendered-tiles:/var/lib/mod_tile'
        - '/data/docker/tileserver:/data'
    environment:
        THREADS: 8
    shm_size: 256M
    command: run

volumes:
  openstreetmap-data:
    external: true
  openstreetmap-rendered-tiles:
    external: true

Loading http://localhost:80/tile/{z}/{x}/{y}.png or leaflet-demo.html

Don't know if I'm doing something wrong. Every step seems OK with no errors, until I try to load a map. I can't load http://localhost:80/tile/{z}/{x}/{y}.png and neither leaflet-demo.html.

I downloaded canary-islands-latest.osm.pbf.

Should z x y been replaced with a value? The leaflet-demo.html should be loaded as the following URL: http://localhost:80/tile/leaflet-demo.html or http://localhost:80/leaflet-demo.html (neither one ore the other loads)?

Thank you in advance.

Backup Strategie

After some investigations and study of docker documentation I wrote following script to create backup of the osm database:

#!/bin/bash
# backup_planet.sh

AREA_NAME=planet-latest
VOLUME_PSQL="openstreetmap-data-$AREA_NAME"
BACKUP_PATH="/media/NVMe/osm_backup"

BACKUP_CMD="docker run -it -v $VOLUME_PSQL:/var/lib/postgresql/10/main \
                -v $BACKUP_PATH:/backup_path \
               alpine tar cfz /backup_path/$AREA_NAME.database.tar.gz -C /var/lib/postgresql/10/main ./"

$BACKUP_CMD

exit 0

The size of docker volume with psql database is arount 1 TBytes after import of the planet. The size of backup is around 240GBytes.

Constrains of the solution:

  • I need to stop the tile server before I can start the backup procedure.
  • The backup procedure needs a lot of time. The compression seems to run in single thread and is using just one core of CPU.

Questions:

  • Any idea how to improve the backup? Is there an "in build" psql technology available to perform backups?

Pre Render question

Hi,

I seen some way about Pre-Render from issue#15

I used render_list , render_list_geo.pl ,but it's just show

render_list

root@911a7c3b8f22:/home/renderer# render_list -a -f -z 10 -Z 10 x 106 X 107 y 54 Y 56
debug: init_storage_backend: initialising file storage backend at: /var/lib/mod_tile
Rendering client
Starting 1 rendering threads
Rendering all tiles from zoom 10 to zoom 10
Rendering all tiles for zoom 10 from (0, 0) to (1023, 1023)

or

render_list_geo.pl

Rendering started at: Wed Feb 13 01:52:53 UTC 2019

Use of uninitialized value in addition (+) at ./render_list_geo.pl line 41.
Use of uninitialized value in addition (+) at ./render_list_geo.pl line 42.
Use of uninitialized value in multiplication (*) at ./render_list_geo.pl line 43.
Use of uninitialized value in multiplication (*) at ./render_list_geo.pl line 43.
Use of uninitialized value in multiplication (*) at ./render_list_geo.pl line 44.
Use of uninitialized value in multiplication (*) at ./render_list_geo.pl line 44.
render_list -a -z 10 -Z 10 -x 512 -X 519 -y 512 -Y 519 -n 2
debug: init_storage_backend: initialising file storage backend at: /var/lib/mod_tile
Rendering client
Starting 2 rendering threads
Rendering all tiles from zoom 10 to zoom 10
Rendering all tiles for zoom 10 from (512, 512) to (519, 519)
Waiting for rendering threads to finish

There nothing happend in my server and /var/lib/mod_tile/ajt folder.

So my question is that only way just like you said "used script loop to force rendered tiles"?

or something I miss config for render_list/render_list_geo.pl?

Thank's for your reading. :)

Deploy application to kubernetes

Is there an easy and straightforward way to deploy the application to kubernetes? I've been playing around with kompose and kompose convert but dont have any success. Can anyone help or has tips?
Would be nice to have a helm chart in the future to make deploying as easy as possible and I would like to help with that, I'm still relatively new to Kubernetes/Docker and still learning a lot but would like to contribute.

Any help is appreciated :)

No Rendered Tiles

Setup
Cores: 8
RAM: 32gb
SWAP: 120Ggb
Disk Space: ~7tb

Import
europe.osm.pbf imported by running the following command:

docker run \
    -v /home/pbf/europe-latest.osm.pbf:/data.osm.pbf \
    -v openstreetmap-data:/var/lib/postgresql/10/main \
    overv/openstreetmap-tile-server \
    import

This exited after 27 days saying it had managed to import.

Running the tile server
I've ran the following command:

docker run \
    -p 80:80 \
    -e ALLOW_CORS=1 \
    -e AUTOVACUUM=off \
    -e THREADS=6 \
    -v openstreetmap-data:/var/lib/postgresql/10/main \
    -v openstreetmap-rendered-tiles:/var/lib/mod_tile \
    -d overv/openstreetmap-tile-server \
    run

However, when i visit http://145.239.252.99/tile/0/0/0.png it shows a 404 - I'm kinda lost as to where to check next, any suggestions?

Cannot import external PBF file

Bind mapping of the external PBF file does not work as expected.

The initial log for

docker run -v "/f/temp/myregion.osm.pbf:/data.osm.pbf" -v openstreetmap-data:/var/lib/postgresql/10/main overv/openstreetmap-tile-server import

is:

 * Starting PostgreSQL 10 database server
   ...done.
CREATE EXTENSION
CREATE EXTENSION
ALTER TABLE
ALTER TABLE
WARNING: No import file at /data.osm.pbf, so importing Luxembourg as example...
/data.osm.pbf: Is a directory
osm2pgsql version 0.96.0 (64 bit id space)

Using lua based tag processing pipeline with script /home/renderer/src/openstreetmap-carto/openstreetmap-carto.lua
Using projection SRS 3857 (Spherical Mercator)

The run continues and sucessfully imports the Luxembourg data.

According to the docker documentation

If you use -v or --volume to bind-mount a file or directory that does not yet exist on the Docker host, -v creates the endpoint for you. It is always created as a directory.

This could explain the log line

/data.osm.pbf: Is a directory

Luxemburg only

Very nice project... thanks for that. I was able to setup the tileserver, and I used europe-latest.osm.pbf instead of Luxemburg. Despite that, I only see detailed tiles for Luxembourg. Am I missing something?

Note: please provide a donation link. This stuff is really awesome if you want to setup your own tileserver.

Pre-Render?

Is there a way to pre-render the tiles so they are not rendered on the fly?

rendering issues on full planet import

Hi,

I have some rendering issues on full planet import:
The import completes without errors, but some details are missing from rendered tiles. The tiles from my server lack at least lakes, rivers and building footprints, please take a look at the screenshot attached.

image

If I import the data for a smaller area, I don't get any render issues. Any pointers are really appreciated.

Thanks,
Chris

tile-server container doesn't lose mem usage.

i had docker import planet.osm.pdf. (during for 7 days)
finished pre-render zoomlevel 7~14.
but docker stats is
tile-server MEM USAGE always 99.9% ( setted --memory=30G)
how do lose mem usage?
it is normal mem-usage 99%?

How to render the tiles?

I made the "Setting up the server" command and all was running for more than 500seconds and it was finishing successful.

Now I'm only getting empty tiles (I only used the data from Georgia)
http://188.68.46.145/tile/12/2563/1518.png -> empty tile
http://188.68.46.145/tile/0/0/0.png -> the world tile (working)

I downloaded the .osm.pbf file from georgia and run:
docker run -v /absolute/path/to/downloads/georgia.osm.pbf:/data.osm.pbf ...

But there was a strange behavior: it started to download some GB of data. For me it looked like the container didn't find the downloaded file. Did I something wrong?

Invalid volume specification

Hello,

I am trying to execute the docker command line in PowerShell in Windows Server 2019 (using Hyper-V and PowerShell package Docker as DockerProvider), but I cannot get the openstreetmap-data volume to mount. Here is the output when I try:

`PS D:> docker volume ls
DRIVER VOLUME NAME
local openstreetmap-data

PS D:> docker run -v openstreetmap-data:/var/lib/postgresql/10/main overv/openstreetmap-tile-server
C:\Program Files\Docker\docker.exe: Error response from daemon: invalid bind mount spec "openstreetmap-data:/var/lib/postgresql/10/main": invalid volume specification: 'openstreetmap-data:\var\lib\postgresql\10\main'.
See 'C:\Program Files\Docker\docker.exe run --help'.
`

no space left on device

I have used osmconvert to merged 6 countries to one pbf, imported this files into my test setup, all fine and running ok.

Now I replicate this on the production server, exact same setup, exact same procedure and I bump into this:

renderd[121]: ERROR: failed to render TILE ajt 2 0-3 0-3
renderd[121]:    reason: Postgis Plugin: ERROR:  could not resize shared memory segment "/PostgreSQL.790133961" to 12615680 bytes: No space left on device
in executeQuery Full sql was: 'SELECT ST_AsBinary("way") AS geom,"feature","name","way_pixels" FROM (SELECT
    way,
    way_area/NULLIF(39135.8::real*39135.8::real,0) AS way_pixels,
    COALESCE(
      'landuse_' || CASE WHEN landuse IN ('forest', 'military', 'farmland') THEN landuse ELSE NULL END,
      'natural_' || CASE WHEN "natural" IN ('wood', 'glacier', 'sand', 'scree', 'shingle', 'bare_rock',
                                            'water', 'bay') THEN "natural" ELSE NULL END,
      'place_' || CASE WHEN place IN ('island') THEN place ELSE NULL END,
      'boundary_' || CASE WHEN boundary IN ('national_park') THEN boundary ELSE NULL END,
      'leisure_' || CASE WHEN leisure IN ('nature_reserve') THEN leisure ELSE NULL END
    ) AS feature,
�
,
    CASE WHEN building = 'no' OR building IS NULL THEN 'no' ELSE 'yes' END AS is_building -- always no with the where conditions
  FROM planet_osm_polygon
  WHERE (landuse IN ('forest', 'military', 'farmland')
      OR "natural" IN ('wood', 'glacier', 'sand', 'scree', 'shingle', 'bare_rock', 'water', 'bay')
      OR "place" IN ('island')
      OR boundary IN ('national_park')
      OR leisure IN ('nature_reserve'))
    AND building IS NULL
    AND name IS NOT NULL
  ORDER BY way_area DESC
) AS text_poly_low_zoom WHERE "way" && ST_SetSRID('BOX3D(-20037508 -20037508,20037508 20037508)'::box3d, 3857)'
renderd[121]: DEBUG: DONE TILE ajt 2 0-3 0-3 in 135.980 seconds

I copied the merged file over to the production server, checked de md5sum to make sure the file is not corrupted.

Import ran without issues, container stopped when done, same as on test setup.
I checked disk usage on the server space enough according to to OS:

df -h
Filesystem                            Size  Used Avail Use% Mounted on
udev                                  3.9G     0  3.9G   0% /dev
tmpfs                                 798M  1.4M  797M   1% /run
/dev/mapper/37--97--158--90--vg-root  288G  103G  170G  38% /
tmpfs                                 3.9G  3.1M  3.9G   1% /dev/shm
tmpfs                                 5.0M     0  5.0M   0% /run/lock
tmpfs                                 3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/vda1                             236M  127M   98M  57% /boot

What could be the case?

update strategy

If I'm not mistaken, the code and explanations only deal with initial import and rendering of data.

Do you intend to add in solutions to manage osm data update (diff imports or full update and re-rendering?) through this container (cron with environment variables to define update options?) or a docker-compose file example?

Docker hub latest tag has no corresponding non-latest tag

Hello,

This image is wonderful. THANK YOU.

I'm building a docker image off of this one (I want to add render_list_geo.pl and meta2tile for pre-rendering a set of tiles based on lat/long range).

https://hub.docker.com/r/overv/openstreetmap-tile-server/tags shows latest was updated 4 days ago, but no other tag matches that modification date. I'd like to pin my Dockerfile to a specific version rather than latest.

Would you add a specific version tag in the hub for the new changes?

Thanks,
Matt

https://github.com/abc3660170/startosm

Hello, I have referenced your project and also made a simple openstreetmap docker demo. In order to facilitate development and debugging, I split 9 packages. Now this project is still under development. I hope you can give suggestions.

Automatic updates not carried out

Import
docker run -v /home/florian/osmdata/austria-latest.osm.pbf:/data.osm.pbf -v openstreetmap-data:/var/lib/postgresql/10/main -v /home/florian/osmdata/austria.poly:/data.poly overv/openstreetmap-tile-server import

Run
docker run -p 8080:80 -e UPDATES=enabled -v openstreetmap-data:/var/lib/postgresql/10/main -v openstreetmap-rendered-tiles:/var/lib/mod_tile -d overv/openstreetmap-tile-server run

crontab
image

run.log
[2019-09-13 12:40:01] 442 start import from seq-nr , replag is second(s)
[2019-09-13 12:41:01] 515 start import from seq-nr , replag is second(s)
[2019-09-13 12:42:01] 539 start import from seq-nr , replag is second(s)
[2019-09-13 12:43:01] 572 start import from seq-nr , replag is second(s)
[2019-09-13 12:44:01] 596 start import from seq-nr , replag is second(s)
[2019-09-13 12:45:01] 640 start import from seq-nr , replag is second(s)
[2019-09-13 12:46:01] 664 start import from seq-nr , replag is second(s)
[2019-09-13 12:47:01] 693 start import from seq-nr , replag is second(s)
[2019-09-13 12:48:01] 717 start import from seq-nr , replag is second(s)
[2019-09-13 12:49:01] 743 start import from seq-nr , replag is second(s)
[2019-09-13 12:50:01] 767 start import from seq-nr , replag is second(s)
[2019-09-13 12:51:01] 794 start import from seq-nr , replag is second(s)
[2019-09-13 12:52:01] 828 start import from seq-nr , replag is second(s)
[2019-09-13 12:53:01] 852 start import from seq-nr , replag is second(s)
[2019-09-13 12:54:02] 876 start import from seq-nr , replag is second(s)
[2019-09-13 12:55:01] 903 start import from seq-nr , replag is second(s)
[2019-09-13 12:56:01] 927 start import from seq-nr , replag is second(s)
[2019-09-13 12:57:01] 956 start import from seq-nr , replag is second(s)

The osmosis working directory in /var/lib/mod_tile/.osmosis has not been created.

It seems to me the first call of sudo -u renderer openstreetmap-tiles-update-expire $REPLICATION_TIMESTAMP within import was not carried out.

After calling sudo -u renderer openstreetmap-tiles-update-expire 2019-09-02 manually within the container the cron job triggered the updates as expected.


Recognized the tile volume (openstreetmap-rendered-tiles) was not attached during import
procedure where the first call to openstreetmap-tiles-update-expire is being initiated. After attaching the tiles volume during import as well everything works as expected.

Import multiple PBF files

Is that possible?

When I use tool like osmium to merge multiple PBF I get error during import.

Best option would be that I could import many PBFs one after another.

Updating error trim_osc 'list' object has no attribute '__array_interface__' and infinite downloading

I have built the Docker container on my own yesterday.

During updating I get an error on osm_trim

renderer@bffdaac26a5f:~/src/openstreetmap-carto$ /home/renderer/src/regional/trim_osc.py -d gis -p /var/lib/mod_tile/data.poly -z /var/lib/mod_tile/changes.osc.gz /var/lib/mod_tile/changes2.osc.gz
Traceback (most recent call last):
  File "shapely/speedups/_speedups.pyx", line 234, in shapely.speedups._speedups.geos_linearring_from_py
AttributeError: 'list' object has no attribute '__array_interface__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/renderer/src/regional/trim_osc.py", line 82, in <module>
    tpoly = poly_parse(options.poly)
  File "/home/renderer/src/regional/trim_osc.py", line 42, in poly_parse
    result = Polygon(poly)
  File "/usr/lib/python3/dist-packages/shapely/geometry/polygon.py", line 240, in __init__
    ret = geos_polygon_from_py(shell, holes)
  File "/usr/lib/python3/dist-packages/shapely/geometry/polygon.py", line 494, in geos_polygon_from_py
    ret = geos_linearring_from_py(shell)
  File "shapely/speedups/_speedups.pyx", line 319, in shapely.speedups._speedups.geos_linearring_from_py
TypeError: object of type 'map' has no len()

My current workaround is to apt install python-psycopg2 python-shapely python-lxml and run osm_trim with python (which is python 2)

There is already an report in the trim_osc repo: Zverik/regional#5

But as this Docker container does not seem to work, we might think about switching to python 2 here?

This is especially problematic because the cron seems to pick up the full diff every time because of this error and creates an unnecessary load on the diff servers

Recreate tiles with new mapnik.xml file

Hey,

I am trying to create tiles with my own style. I've created mapnik.xml file and replaced the old one in /home/renderer/src/openstreetmap-carto/mapnik.xml. Next, I've restart container and enter to http:/[host]/0/0/0.png website and... nothing happened.

So I've tried to run renderd script (from /home/renderer/src/mod_tile) and run.sh file, but with no results.

Could you tell tell how can I reproduce tiles with new mapnik.xml file?

Thanks in advance
Michal

Clearly update method

Dear contributors,

How the update works?
Every time if I update the pbf and poly file I need to stop and delete everyting expect of the generated tiles and run the „import” then the „run” task with the UPDATES=enabled env variable?

I now use thees commands to run.

docker run --rm -v /opt/openstreetmap/hungary-latest.osm.pbf:/data.osm.pbf -v /opt/openstreetmap/hungary.poly:/data.poly -v openstreetmap-data:/var/lib/postgresql/10/main overv/openstreetmap-tile-server import

docker run -p 80:80 -e UPDATES=enabled -v openstreetmap-data:/var/lib/postgresql/10/main -v openstreetmap-rendered-tiles:/var/lib/mod_tile -d overv/openstreetmap-tile-server run

Error response from daemon: invalid mode: /data.osm.pbf.

Hello,

When I am trying to create the server in this way:

docker run -v https://download.geofabrik.de/asia/uzbekistan-latest.osm.pbf:/data.osm.pbf -v openstreetmap-data:/var/lib/postgresql/10/main overv/openstreetmap-tile-server import

Getting the following error:
docker: Error response from daemon: invalid mode: /data.osm.pbf.

HTTP 503 after install and import

As the title says, I've ran through the docker setup, imported a osm.pbf.

issued the run command and when i go to my IP address im getting a HTTP 503

Any ideas?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.