tuning & monitoring about openstreetmap-tile-server HOT 8 CLOSED

ok seems my renderd remains very busy all the time:

renderd[121]: DEBUG: START TILE ajt 9 256-263 160-167, age 3.17 days
renderd[121]: Rendering projected coordinates 9 256 160 -> 0.000000|6887893.492837 626172.135713|7514065.628550 to a 8 x 8 tile
renderd[121]: DEBUG: DONE TILE ajt 13 4216-4223 2712-2719 in 661.364 seconds
debug: Creating and writing a metatile to /var/lib/mod_tile/ajt/13/0/16/10/121/136.meta

renderd[121]: DEBUG: START TILE ajt 9 256-263 168-175, age 3.17 days
renderd[121]: Rendering projected coordinates 9 256 168 -> 0.000000|6261721.357125 626172.135713|6887893.492837 to a 8 x 8 tile
renderd[121]: DEBUG: DONE TILE ajt 14 8432-8439 5416-5423 in 758.299 seconds
debug: Creating and writing a metatile to /var/lib/mod_tile/ajt/14/0/33/5/242/8.meta

renderd[121]: DEBUG: START TILE ajt 16 33760-33767 21704-21711, age 4.61 days
renderd[121]: Rendering projected coordinates 16 33760 21704 -> 606604.256471|6760702.277771 611496.226282|6765594.247581 to a 8 x 8 tile
renderd[121]: DEBUG: DONE TILE ajt 16 33760-33767 21696-21703 in 998.452 seconds
debug: Creating and writing a metatile to /var/lib/mod_tile/ajt/16/0/133/52/236/0.meta

In this post it seems that tiles are re-renderd every 3 days by default? That is not necessary for my application so I need to make sure the rerendering does not take place. My 4 core server seems now very bussy keeping up rerendering all tiles untill zoom level 15. It seems it cannot keep up ... does that make sense to you?

from openstreetmap-tile-server.

3 days is default for declaring tiles old if not configured otherwise.
Easy solution: in /var/lib/mod_tile and/or /var/lib/mod_tile/ajt create a file called planet-import-complete with the creation date of your last postgres db import.
(e.g.: touch -d "2 days ago" planet-import-complete )
With this file present, mod_tile won't rerender tiles if they are younger than the planet-import-complete filetime. (Which means you'll have to use other techniques to update tiles).

(btw you could have read about this solution in the help thread you linked to yourself).

from openstreetmap-tile-server.

seems I am doing something wrong while trying to create the log rotation

ERROR: for tileserver  Cannot create container for service tileserver: unknown log opt 'max-files' for json-file log driver

?

[...]

any suggestions?

My bad, it's max-file and not max-files.
https://docs.docker.com/compose/compose-file/#logging

from openstreetmap-tile-server.

4. Are there any ways to monitor and understand better if and how the renderd process actually has renderd tiles? How many tiles are actually prerenderd etc etc. With 335 MB in the renderd_tiles folder something seems not right?

I use (among other tools) munin to monitor a few servers. Luckily mod_tile does come with munin plugin scripts to monitor the rendering process. So they should already be present inside your container, you might just need to execute them properly.

apache2 and postgresql, that run inside the container, can also be monitored by munin, but for those two there are also other tool(s) around to monitor them.

To inspect what currently is being worked on, there is output of the renderd process that is collected by docker and can be watched with it:

$ docker-compose logs --follow tileserver

BTW: docker does save but not rotate these logs by default, so it (slowly) grows and grows and grows over time until the harddisk (of /var/lib/docker/) is full. It might make sense to configure a log rotation for it, e.g. like this:

services:
  tileserver:
    # ...
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "10"

As for the apache2 and postresql services running inside of the container: they are started with service so their runtime output should by accessible via journalctl, but both should also have logfiles inside /var/log/ by default.

from openstreetmap-tile-server.

ok so we have " To allow re-rendering but not have the client wait for such tiles, set your ModTileRequestTimeout to 0 instead of 3".

In the default acpache.conf we have ModTileRequestTimeout = 0 right?

from openstreetmap-tile-server.

1. Are there any ways to monitor and understand better if and how the renderd process actually has renderd tiles? How many tiles are actually prerenderd etc etc. With 335 MB in the renderd_tiles folder something seems not right?

I use (among other tools) munin to monitor a few servers. Luckily mod_tile does come with munin plugin scripts to monitor the rendering process. So they should already be present inside your container, you might just need to execute them properly.

apache2 and postgresql, that run inside the container, can also be monitored by munin, but for those two there are also other tool(s) around to monitor them.

To inspect what currently is being worked on, there is output of the renderd process that is collected by docker and can be watched with it:

$ docker-compose logs --follow tileserver

BTW: docker does save but not rotate these logs by default, so it (slowly) grows and grows and grows over time until the harddisk (of /var/lib/docker/) is full. It might make sense to configure a log rotation for it, e.g. like this:

services:
  tileserver:
    # ...
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-files: "10"

As for the apache2 and postresql services running inside of the container: they are started with service so their runtime output should by accessible via journalctl, but both should also have logfiles inside /var/log/ by default.

seems I am doing something wrong while trying to create the log rotation

ERROR: for tileserver  Cannot create container for service tileserver: unknown log opt 'max-files' for json-file log driver

?

my complete docker-compose:

version: '3'
services:
  tileserver:
    container_name: tileserver
    image: overv/openstreetmap-tile-server
    restart: always
    ports:
        - "8099:80"
    volumes:
        - 'openstreetmap-data:/var/lib/postgresql/10/main'
        - 'openstreetmap-rendered-tiles:/var/lib/mod_tile'
        - '/data/docker/tileserver:/data'
        # get localtime from host
        - '/etc/timezone:/etc/timezone:ro'
        - '/etc/localtime:/etc/localtime:ro'

    environment:
        VIRTUAL_HOST: 'tiles.x.org'
        LETSENCRYPT_HOST: 'tiles.x.org'
        LETSENCRYPT_EMAIL: [email protected]
        THREADS: 8
        AUTOVACUUM: 'off'
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-files: "10"
# test certs
#        LETSENCRYPT_TEST: 'true'
#        ACME_CA_URI: 'https://acme-staging.api.letsencrypt.org/directory'
    shm_size: 256M
    command: run

volumes:
  openstreetmap-data:
    external: true
  openstreetmap-rendered-tiles:
    external: true

any suggestions?

from openstreetmap-tile-server.

Hi @Istador thx for your help. I am still strugling with a proper setup, especially now we keep adding countries to the tileserver as new customers come in .

I now have Netherlands, 7 African countries en recently added India Portugal Spain and Italy.

Now I upgraded my machine to be 8 cores 32 GB 200GB disk so still rather small (but sponsored so not 100% in my control). We are serving Natureparks in these countries, sometimes also interested in corridors between Narute parks.

Last night I bumped into this:

Osm2pgsql failed due to ERROR: Connection to database failed: FATAL:  remaining connection slots are reserved for non-replication superuser connections

So apparently 16 THREADS is to much for the default max_connections of 250 if I understand correctly?

In your reply to issue 5 you specked out some details on your setup, did you already get consent from your employer to share your setup detail ;-) ?

I would be most interested in a postgresql conf file and docker file that work smoothly on a machine more or less my size...

I think I would like to create my own image using postgresql 11 but overwhelmed by all the options and settings in the conf files to make it work smoothly.

from openstreetmap-tile-server.

Last night I bumped into this:

Osm2pgsql failed due to ERROR: Connection to database failed: FATAL:  remaining connection slots are reserved for non-replication superuser connections

So apparently 16 THREADS is to much for the default max_connections of 250 if I understand correctly?

Well, the default value for max_connections is 100. If you're running the newest version of this docker image (from 2019-04-23), then it was increased to 250 by introducing a custom postgresql.conf.

I don't know out of my head which value I used in my setup, that was in July 2018 (when Google Maps drastically changed their pricing model).

According to osm2pgsql-dev/osm2pgsql#885 the amount of connections depends on the amount of threads running and tables in the database (7 or more). Because you faced this error with osm2pgsql, please note that if you're serving tiles with renderd (16 threads) and at the same time append to the same database with osm2pgsql (16 threads), then your total thread count is at least 32 (or more, depending on what else is accessing the database (background tasks, monitoring, etc.)). Theoretically with 7 x 32 = 224 you should be fine with a setting of 250 for max_connections, but apparently you're not.

(Please verify if you're running the latest version of this docker image or not. Working with an existing system that is/was running the old version, requires an docker-compose pull to download the latest image version and docker-compose up -d to recreate and restart the container.)

• Number Of Database Connections - PostgreSQL wiki
• Tuning Your PostgreSQL Server - PostgreSQL wiki

In your reply to issue 5 you specked out some details on your setup, did you already get consent from your employer to share your setup detail ;-) ?

Nope, nothing new on that. I asked my boss two or three times about it (last time in March), but only got the reply that the request will be forwarded to our CEO.

The setup details that I shared already aren't that much of a problem, because I don't disclose anywhere who my employer is and even then this information wouldn't be that sensitive, but I have to operate based on my memory in my free time and can not dedicate paid working time on it.

The only problem I have is, that I can't contribute any commits/changes that I did for the setup (e.g. new environment variables for memory and flatnodes, changes to run.sh to update/extend the database and expire/re-render tiles) to this project or an public fork without approval (because my work contract grants my employer all usage rights on the work products I develop for him).

from openstreetmap-tile-server.