stadtnavi / digitransit-ansible Goto Github PK

• Add bot user for dockerhub
• For every docker build, push to stadtnavi as well to allow smooth transition
• Move digitansit-ansible to new Github org
• In sensible scripts, switch docker repo to stadtnavi (except for otp)
• Move digitransit-ui to new Github org
• Activate Travis for new repo and check if PR notifications work
• Create docker hub repo description pages

We want to experiment with manually modifying the ranking of places returned by Photon. In our concrete example the search for "Ahrensburg" returns the city of Ahrensburg and then the two subway stations, but not the main train station - at least not ranked high enough. This is because the train station doesn't have a wikidata or wikipedia tag.

Therefore we want to experiment with manually changing the importance.

Connecting to the Nominatim DB

If you're ssh'ed into the host infrastracture.stadtnavi.eu you can use the following command line to connect to the nominatim DB:

psql -h localhost -p 5432 -U nominatim

The password required is in the ansible vault but is visiable in decrypted form in the file /etc/systemd/system/nominatim.service.

IntelliJ

If you have the professional version of IntelliJ you can also use that for inspecting the database. For this you need to set up SSH tunneling in the UI.

Queries

If you want to find your way around a little, you could use the following queries:

-- Ahrensburg West (U-Bahn)
select * from placex where osm_id = 6233764950;

-- Ahrensburg Ost (U-Bahn)
select * from placex where osm_id = 5196118052;

-- Ahrensburg (Deutsche Bahn)
select * from placex where osm_id = 3205295000;

Setting the importance

In the queries and in the Photon debug output you can see that the two subway station have a score of ~0.31. We want to set the importance to something above that.

(Hamburg main station has an importance of ~0.46, by the way.)

The following query will set the wikipedia tag to the one of the city of Ahrensburg. and set the indexing status to 2, meaning Photon will know that it needs to import the row:

update placex set extratags = extratags || hstore('wikipedia', 'en:Ahrensburg'), indexed_status = 2 where osm_id = 3205295000;

During Photon's import it also sets the indexed_status and this will retrigger a computation of the importance.

Updating Photon

Now that we have updated the DB we can then trigger a Photon reimport with:

sudo systemctl start photon-update.service

You this will return immediately as it only triggers the HTTP call to start the import. You can view the progress in the log with

journalctl -u photon -f

If you have only a few rows to import then it should complete within seconds.

Result

Searching for Ahrensburg will lead to the train station being ranked highest.

Right now Photon runs at https://photon-temp.leonard.io/, however we probably want a proper DNS entry for this.

@hbruch Please decide which one it is and add the entries. If you like the URL to be something like https://api.mfdz.de/photon/ then let me know and I will configure it.

Previously I removed the names of water bodies as they were way too prominent (#17) but upstream has some interesting fixes: they display the names based on the size of the lake/river.

This means that not every puddle shows up on the map which is what we want.

Acceptance criteria

• Get the latest process-openmaptiles.lua file from upstream and merge their changes. Be careful not to overwrite 3737109
• Check the changes in config-openmaptiles.json. This file also configures zoom levels but to be honest I haven't fully understood how it interacts with the lua file. We might need the changes, might not.

Tips:

• make tileserver-kreis-boeblingen (https://www.openstreetmap.org/relation/62721) will give you a larger area to test. Of course computing this will take a while.
• The changes regarding the size of features are quite new. If you are having problems, make sure that you've pulled the latest version of the docker image lehrenfried/tilemaker.

On 2020-02-21 @Beck-berry reported a problem with the Matomo installation on track.mfdz.de. The reported problem was a missing Matomo configuration file.

@leonardehrenfried analysed the the problem and found a problem with the systemd unit responsible for starting up and cleaning shutting down the docker-compose network for Matamo: the command docker-compose down -v was copied from the digitransit network. This however means that when shutting down or restarting Matomo its volumes deleted, as indicated by the -v option. This is the desired behaviour for digitransit as it has no persistent state and its hsl-map-server even needs the volumes to be deleted.

In 10b2145 @leonardehrenfried changed the shutdown procedure to issue docker-compose stop instead which shut be more gentle to the persistent containers' data.

On 2020-02-24 he tested the fix and Matomo indeed survives a restart of the systemd unit.

/cc @hbruch @meromisi

The current map gives too much prominence to the names of train tracks. I have circled the problematic ones in the screenshot above.

Please investigate how to drastically reduce the amount of times it's displayed or completely remove them.

Note: these are not the names of lines but the name of the actual train tracks. I lived in village near Herrenberg until I was 20 and didn't know that these tracks even have a name, so they don't really add useful information.

When you switch to the satellite map mode you can see that construction sites, tunnels and probably a few other types of roads, are too prominent.

This is not a OSM extraction issue but a problem with the map style in the file roles/tileserver-gl/templates/satellite-overlay.json.

Please check the tunnel and construction site style elements and adjust the opacity to match the rest of the roads.

The parkapi script that downloads data from thingsboard taking several minutes on the the staging server, even though on my laptop it runs in a few seconds.

I think I debugged the issue as blocked IPv6 connections. The Thingsboard host advertises an IPv6 address:

but connecting to it with IPV6 fails from the staging host:

curl https://<thingsboard-url>/api -6

With IPv4 this call is instant.

My theory: the python script tries for every request to use IPv6 but times out and then uses ipv4. Since there are a few calls to this host, the whole script takes several minutes.

Note: Connecting to the host ssh with IPv6 also times out.

@hbruch Can you please take care of this.

Edit: It's not down to blocked IPv6 connections, it's simply that the advertised AAAA record of thningsboard-server cannot be routed to. The AAAA record needs to be corrected or removed.

Currently updating the charging-stations-ocpi.json template requires updating the json-thingsboard-enhancer commit and an ansible redeployment.

We should download the template every n (e.g. 10 minutes) from a remote url and use that for rendering instead.

Question is what to use as the download URL of the GTFS feed that includes the URLs, which are sensitive data.

One solution would be to put it into a public bucket (perhaps Azure because the elevation data is also hosted there) and use an un-guessable URL which is encrypted in ansible vault.

We might want to add password authentication but in that case I would have to change the data-container repo as it assumes everything is freely downloadable.

Reminder: Also review current crawling/leak protection measures.

Stadtgrabenstaffel (https://www.openstreetmap.org/way/129886256) is not rendered on the map.

Most likely because it is tagged as highway=steps. To me it seems strange to tag this as a highway, but apparently it's the correct scheme: https://wiki.openstreetmap.org/wiki/Tag:highway=steps?uselang=en

I got this VM with 512 MB RAM

and not the more appropriate setup for libvirt as in the Vagrantfile

In fact I'd prefer using libvirt instead of VirtualBox but I could not find out how.

Holger has reported that tunnels near his house are not shown transparently:

• https://www.openstreetmap.org/way/13876085
• https://www.openstreetmap.org/way/15242084
• https://www.openstreetmap.org/way/14502948

Notes:

• You can render the map for this part of Stuttgart with make tilemaker-vaihingen. (https://github.com/leonardehrenfried/polygons/blob/master/vaihingen.geojson)
• In the tile server overview (localhost:8080/) you can also select the "data view" which will show you the detailed data about a map feature, ie. which class they are.

My feeling is that it's a different class of bridge/tunnel.

It should be 2a01:4f8:151:10a8::2.

That's the reason IPv6 traffic doesn't work to this host.

Perhaps we ought to use the hostname staging.stadtnavi.eu.

Self-hosted:

Maptiler:

https://mih.mitfahren-bw.de/reiseplan/Vikarweg%2C%2070567%20Stuttgart%3A%3A48.7246315%2C9.1432803/Wochenmarkt%20M%C3%B6hringen%3A%3A48.7260499669869%2C9.14494784348986?optimize=TRIANGLE&safetyFactor=0.4&selfHostedTiles&slopeFactor=0.3&time=1589466202&timeFactor=0.3

This is of course debatable but I would show them at one zoom level higher.

• document systemd timers in README
• hbg.merged.gtfs.zip contains 4 merged feeds

The new project stadtnavi/cifs-transformer generates a cifs.json which contains the BW road works as well as those published by the city of Herrenberg via CSV.

Currently, this file is generated periodically at https://data.mfdz.de/hbg/roadworks/cifs.json.

In a first step, this file should be used in the tilelive map layer.

In the second step, a systemd job should be added to digitransit-ansible which periodically (e.g. every 10min) publishes the cifs via api.*

user/certbot_admin_email will be deployment specific vars. They should be extracted from group_vars/all.yml

Whilst rolling out to production, a had to make changes to the ansible roles to make them compatible with Ubuntu 18.04, as the staging server is running Debian 10.

Whilst the changes are easy to do, it really defeats the point of a staging system, if they run different operation systems. We really should not find out on the production server, if a change is incompatible.

Acceptance criteria:

• Change the production system to use Debian 10

For different setups (or to use preprocessed osm.pbf), the pbf url should be configurable

Right now i'm using track-temp.leonard.io but that is of course a temporary solution.

Right now, the installation on a completely fresh host will fail as the digitransit docker-compose requires the image mfdz/opentripplanner-data-container-hb:local.

You might ask why it doesn't use mfdz/opentripplanner-data-container-hb:latest. If you do that and do a docker-compose pull the remote tag will overwrite the local tag (the freshly build data container), even though it is way older.

Then why don't you push the freshly built data container to docker hub? Because it contains sensitive data about carpooling routes that we don't want to publish.

Possible solutions

• When you first install digitransit on a host, download mfdz/opentripplanner-data-container-hb:latest and tag it locally as mfdz/opentripplanner-data-container-hb:local. You need to make sure though that this only happens at the very first time as otherwise you again overwrite the tag.

I've asked the author of tilemaker about railway tracks and if he would welcome a PR that removes them. He said that he doesn't want to remove them but only display them starting at zoom level 14. Therefore we need to build upon #29.

Acceptance criteria

• Bring back names of railway tracks.
• Figure out how to set the zoom level for these names starting from level 14. The documentation might give you an overview. This commit might make it easier to do it in the lua file. But perhaps it's also possible to just use the process-openmaptiles.json.
• Upstream the changes with before-and-after pictures in the PR.

Beki has sucessfully added the construction works to our map tiles. We would like the changes to process-openmaptiles.lua to be upstreamed to https://github.com/systemed/tilemaker/blob/master/resources/process-openmaptiles.lua

Note that since I took a copy of this file it has changed a bit so you cannot just copy our file back.

Acceptance criteria

• Do a bit more QA by checking other construction sites in BaWü (https://github.com/systemed/tilemaker/blob/master/resources/process-openmaptiles.lua) and make sure they are rendered on our map correctly
• Take the construction works and copy them upstream
• If it becomes too difficult to just upstream, you can copy their file into our repo, add the construction back in and the upstream that.

Should be easy to switch but beforehand it needs to be deployed to dockerhub first. I don't have the permissions to do that.

We need to get started with the nominatim and photon upgrades. I'm going to summarise the installation process here since there are quite a few steps required.

Nominatim

• Use standard nominatim 3.7
• Configure the the extra-tags import style with the following addition

{   "keys" : ["public_transport"],
    "values" : {
       "platform" : "main,fallback,with_name",
       "stop_position" : "main,fallback,with_name",
       "station" : "main,fallback,with_name"
    }
},

• Initial nominatim import
• Download https://data.mfdz.de/mfdz/nvbw_osm_matches.csv.gz
• Import CSV stop data with ./nominatim_add_ifopt.py -i osm-nvbw-stop-matches.csv (with invalidate)
• Nightly import run ./nominatim_add_ifopt.py osm-nvbw-stop-matches.csv (no invalidate!)

Photon

• use current komoot/master with two PRs merged in:
  • komoot/photon#581
  • komoot/photon#580
• start up Photon with the flag -extra-tags ref:IFOPT
• run regular import
• start Photon with -synonym-file

Did I miss something?

@lonvia @hbruch

@frankgerhardt has requested that Matomo be added to the ansible installation.

According to him it's a docker compose file with 3 containers in them:

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
2faa4d965b23        matomo:fpm-alpine   "/entrypoint.sh php-…"   5 days ago          Up 5 days           9000/tcp               digitransit-matomo_app_1
16588c5cbf68        nginx:alpine        "nginx -g 'daemon of…"   5 days ago          Up 5 days           0.0.0.0:8080->80/tcp   digitransit-matomo_web_1
f727b18adc40        mariadb             "docker-entrypoint.s…"   5 days ago          Up 5 days           3306/tcp               digitransit-matomo_db_1

This is the street in OSM: https://www.openstreetmap.org/way/145661403