osm-tonie / ptna Goto Github PK

See output:

|- bgcolor="#dfdfdf"
|| '''{{Sketch Line|256|VBN|wuppertal}}''' || colspan="5" align="right"| Kommentar: OL Nord; Von: Delmenhorst; Nach: Oldenburg; Betreiber: Weser-Ems-Bus
|-
| De_VBN_256_1 || route || {{Relation|77044}} || align="center" | ? || Multiple Routes but no Route-Master || 'network' is short form
'public_transport:version' is not set
|-
| De_VBN_256_2 || route || {{Relation|77563}} || align="center" | ? || Multiple Routes but no Route-Master || 'network' is short form
'public_transport:version' is not set
|-
| De_VBN_256_1 || route || {{Relation|77044}} || align="center" | ? || Multiple Routes but no Route-Master
Multiple Routes but no Route-Master || 'network' is short form
'public_transport:version' is not set
'network' is short form
'public_transport:version' is not set
|-
| De_VBN_256_2 || route || {{Relation|77563}} || align="center" | ? || Multiple Routes but no Route-Master
Multiple Routes but no Route-Master || 'network' is short form
'public_transport:version' is not set
'network' is short form
'public_transport:version' is not set

In "sub analyze_route_master_environment" do a similar check as wirh 'ref' is set but 'network' differs.

see at: # 'ref' is the same, check for other problems

Currently, those are skipped

This shall include

• route=coach
  • might be a valid contents of Overpass query output, but is not a widely accepted route-type)
• route=bicycle or hiking or ...
  • this shouldn't be in the Overpass query output

if the route uses a 'motorway_link' then check whether a 'motorway'

• will be used directly later on
• has been used directly before that

Consider also

• 'trunk' in addition to 'motorway' (Autobahnkreuz Bonn/Ramersdorf, B 42 comming from the south)
• multiple 'motorway_link' segments

Errors may indicate

• wrong tagging of a way, which isn't actually a 'motorway_link'
• wrong routing of 'bus', ... entering a 'motorway_link', make a u-turn leaving via another 'motorway_link', heading back to original way

In some cases, the sorting of the relations with same 'ref' can not be predicted.
This is the case, when sort_name or ref_trips or name are identical for 2 or more relations.

Apppend the relation-ID to the sorting criteria to ensure unique sorting strings.

When analyzing a Route-Relation, check for existence of Route-Master(s) where this Route-Relation is a member of:

• there should be only one Route-Master for PTv2 Route-Relations
  report an issue if there is more than one Route-Master

See:
https://wiki.openstreetmap.org/w/index.php?title=Proposed_features/Public_Transport&oldid=625726#Compatibility_with_well_known_tags
and consider these combinations as valid
Maybe add option "--strict-ptv2" or "--allow-compatibility"

currently, the upload/download to/from OSM Wiki is done manually.
With the increasing number of Networks to be supported, this becomes tedious.
Create a script for upload and download of Wiki Data (i.e. at least for the results of the analysis) and use that script in the analyze-network.sh scripts

@well_known_network_types = ( 'international', 'national', 'regional', 'local', 'icn', 'ncn', 'rcn', 'lcn', 'iwn', 'nwn', 'rwn', 'lwn', 'road' );

Print issues for missing stop_positions on first or last node of way only if there is at least one stop_position in the route.

• stop_positions nodes are optional. But if there is at least one stop node, the mapper should add also the others

When printing errors for nodes and ways, not only show ID and the links to iD adn JOSM, ... but also the name of the street, bus-stop, platform - if 'name' is set and makes sense for the particular type of error.

$text =~ s/[[[^:]+:[^\]]+]]//g; was supposed to delete only [[Category:....]] but deletes also other stuff

This way is a highway=primary and does not have a 'role', so why complain?

add parameter "--section=" with valid values

• pull: 0, 1, 2, 3, 4, ...
• push: 0,1, 2, 3, 4, ..., new

Use case: automated changes on section=2 of https://wiki.openstreetmap.org/wiki/User:ToniE/analyze-routes
which is
https://wiki.openstreetmap.org/wiki/User:ToniE/analyze-routes#Analysierte_Verbunde
to change the contents of columns "4" labled as "Änderungen zur vorangegangenen Auswertung (experimentell, siehe dort: Naivgation rechts unten) " with the date and time of the date of OSM data (upload of HTML result of the analysis)

Check whether the value of the 'colour' tag of a route is the same as the one for the corresponding route_master

Tool does not at all or reports only once or even twice for a specific route_master relation when there is more than one route_master for this "ref".

See busses of DE-HB-VBN 304, 306, 307, 308, 310, 311, 314, 315, 317, 318, 321, 322, 323, 324 as of 20:21, 25. Feb. 2018‎

| Bus 304 || route_master || {{Relation|3767311}} || align="center" | ? || There is more than one Route-Master for this line
Route-Masters have more Routes than actually exist (4 versus 2) in the given data set
'public_transport:version' is not set || 'network' is short form
|-
| Bus 304 || route_master || {{Relation|5488560}} || align="center" | ? || 'network' is not set
'public_transport:version' is not set ||
|-

| Bus 306 || route_master || {{Relation|447879}} || align="center" | ? || 'public_transport:version' is not set || 'network' is short form
|-
| Bus 306 || route_master || {{Relation|5488561}} || align="center" | ? || There is more than one Route-Master for this line
Route-Masters have more Routes than actually exist (4 versus 2) in the given data set
There is more than one Route-Master for this line
Route-Masters have more Routes than actually exist (4 versus 2) in the given data set
'network' is not set
'public_transport:version' is not set ||
|-

| Bus 307 || route_master || {{Relation|3768321}} || align="center" | ? || 'public_transport:version' is not set || 'network' is short form
|-
| Bus 307 || route_master || {{Relation|5488562}} || align="center" | ? || There is more than one Route-Master for this line
Route-Masters have more Routes than actually exist (4 versus 2) in the given data set
There is more than one Route-Master for this line
Route-Masters have more Routes than actually exist (4 versus 2) in the given data set
'network' is not set
'public_transport:version' is not set ||
|-

| Bus 308 || route_master || {{Relation|3770400}} || align="center" | ? || There is more than one Route-Master for this line
Route-Masters have more Routes than actually exist (8 versus 4) in the given data set
'public_transport:version' is not set || 'network' is short form
|-
| Bus 308 || route_master || {{Relation|5488563}} || align="center" | ? || 'network' is not set
'public_transport:version' is not set ||
|-

| Bus 310 || route_master || {{Relation|3749333}} || align="center" | ? || There is more than one Route-Master for this line
Route-Masters have more Routes than actually exist (18 versus 9) in the given data set
'public_transport:version' is not set || 'network' is short form
|-
| Bus 310 || route_master || {{Relation|5488564}} || align="center" | ? || 'network' is not set
'public_transport:version' is not set ||
|-

| Bus 317 || route_master || {{Relation|3785080}} || align="center" | ? || 'public_transport:version' is not set || 'network' is short form
|-
| Bus 317 || route_master || {{Relation|5488568}} || align="center" | ? || There is more than one Route-Master for this line
Route-Masters have more Routes than actually exist (4 versus 2) in the given data set
There is more than one Route-Master for this line
Route-Masters have more Routes than actually exist (4 versus 2) in the given data set
'network' is not set
'public_transport:version' is not set ||
|-

...

an new analysis report is compared with an old one (*.html.save).
The size of the diff is used to decide whether to upload the new report or not, whether someting has changed or not.
OSM Data time stamp and the AREAs time stamp always differ, so the size is always > 0.

Do not take the size of the diff (104, 200, 204, ...) to decide this.
Do take the number of lines in the diff file (4 or 6).

• 4 if the overpass query is based on poly(...) - no AREA time stamp information
• 6 if the overpass query is based on boundary information - including AREA time stamp information
  A difference can be assumed if the number of lines is larger than 4 or 6

Example including boundary information
100,101c100,101

OSM-Base Time : 2018-08-30 23:11:02 UTC
Areas Time : 2018-08-30 22:28:02 UTC

< OSM-Base Time : 2018-08-29 23:11:02 UTC
< Areas Time : 2018-08-29 22:28:02 UTC

If the first way is a oneway way and used in wrong direction, the tool simply reports a gap with no further hint.
It is hard to detect the issue in the relation with JOSM, because JOSM does not detect that at all.
A text like the existing "PTv2 route: using oneway way(s) in wrong direction: %s" should appear also here.
Test with Test/RBO-6209-006-issue-17.osm

should read:
Route-Masters have more Routes than actually match (%d versus %d) in the given data set
Route-Masters have less Routes than actually match (%d versus %d) in the given data set
and
Route-Master has more Routes than actually match (%d versus %d) in the given data set
Route-Master has less Routes than actually match (%d versus %d) in the given data set

see also issue #37

@well_known_other_route_types = ( 'bicycle', 'mtb', 'hiking', 'road', 'foot', 'inline_skates', 'canoe', 'detour', 'fitness_trail', 'horse', 'motorboat', 'nordic_walking', 'pipeline', 'piste', 'power', 'running', 'ski', 'snowmobile', 'cycling' , 'historic', 'motorcycle', 'riding' );

@well_known_other_types = ( 'restriction', 'enforcement', 'destination_sign' );

The use of Perl module XML::Simple is too slow (~ 6 minutes for 42MB XML).
Replace by a faster module, e.g. one which knows about the light-weight structure of the XML-Overpass-API or XML-JOSM-OSM file - for instance Geo::Parse::OSM, which could do the job in 40 seconds

The last section in the output HTML should be split into two parts:

• "Berücksichtige 'network'-Werte" --> new, showing ???
• "Nicht berücksichtigte 'network'-Werte" -> exists, showing also their relations

Maybe add a "second" column showing how often the value/combination has been seen.

Tool prints "'network' is short form" if the value is "VRS" - this should be avoided, because that is what we expect.
We expect the 'network' being long for "NRW Regionalverkehr" and 'network' being short for "VRS".

Settings:
--expect-network-long
--expect-network-short-for="Verkehrsverbund Rhein-Sieg"
--network-long-regex=" Verkehrsverbund Rhein-Sieg|NRW Regionalverkehr"
--network-short-regex="VRS"

New option?
--accept-network-short-for="VRS" for this case of VRS

--accept-network-long-for"=xxxx xxx" for other cases

both options as regular expressions!

When analyzing a Route-Relation, check for existence of Route-Master(s):

• a Route-Master should have the same 'ref' value
• a Route-Master should have the same 'network' value
  report an issue if this is not the case.
  Mainly consider Route-Masters where the 'ref' or 'network' value does not match the wanted criterias of:
• network_long_regex / network_short_regex
• refs_of_interest

use htmldiff or something similar to show the differences between old and new HTML report as an HTML file with for instance red (deleted) and green (inserted).

Create a link on the OSM wiki page which points to the diff file.
Maybe also create an info on the OSM wiki page showing whether there is a new diff or not (time stamp?)

Maybe move the overview table to the target location, if it is easier to show that info or not

the trailing '.' in a link to a subsection is omitted when copying the URL.

Do not attach a '.' at the end of an anchor-link (TOC)

1. )The message is missleading, it does not really show what is wrong:
   "Route: 'network' = '%s' does not match regular expression for valid 'network' values ('$regex_network_long and $regex_nertwork_short'): %s"

2.) do not show this route under %suspicious_relations : that is even more missleading.

DE-BY-FRG-Analysis.html routes 6115 and 7595

For public transport vehicles running on rails or similar, this check can be easily implemented.
Applies to: tram, train, light_rail, subway, ferry, aerialway, funicular, monorail.
Does not apply to: bus, share_taxi, trolleybus.

avoid printing "wide characters", i.e. non-utf8 characters in the output.
These will cause warnings by "printf" and will prohibit the upload to the OSM Wiki via the wiki-page.pl srcipt. The wiki-page.pl script uses MediaWiki::API and this one LWP and those complain about those characters in the infile and on the upload

convert "wide characters" by their hex representation, i.e. sprintf "%x", ord($1).

Some donwloads via Overpass-API query do not include all data that is references in the relations.
I.e. there are some routes with way members, but details for the ways (tags and node-IDs) are not in the downloaded data. This may also the case for member nodes.
The code depends on completeness of data and does not check like if ( $WAYS($way_id)->{'first_node'} )

Before analyzing route relations, check for completeness of data.
If not complete, issue an error in the analysis file (for this route) and on STDERR specifying the Relation-ID.
Furthermore skip check which depend in completeness of data.

--check-motorway_link:

print error only if the route is well sorted, i.e. no gaps

First step towards realizing issue #45.

Move data to /osm/ptna/www/results/... on host everets.opnestreetmap.de via sftp
Accessible then via

• https://ptna.openstreetmap.de/results/DE/....
• https://ptna.openstreetmap.de/results/EU/...

Use WEB_URL == https://ptna.openstreetmap.de/
Use WEB_LOC == results/DE/ - depending on PREFIX (letters until first '-')
Use TARGET_HOST == [email protected]
Use TARGET_LOC == /osm/ptna/www/results/DE/ - depending on PREFIX (letters until first '-')

add code to the upload-wiki section of analyze-network.sh so that Wiki pages bigger than 2 MB are split into 2 parts and uploaded separately.
The string "- " can be inserted to the Routes file and will be printed in the analysis result, so that the mainatiner of the routes file can specify where to split. The maintaner has to take care for correct linking from the first to the second part then by entering [[xxx/yyy/zzz-2|Teil 2 der Analyse von xxx]].
Use "csplit" to automatically split the result file into max. 2 parts and upload the 2 parts to the Wiki.

as the title says: if the first route in a table is the same as the last on in the table before, no table-sub-header is generated

Make the tool multi-lingual by using ...

probably Locale::TextDomain

Mainly:

https://metacpan.org/pod/Locale::TextDomain
-> good example also with makefile: https://gist.github.com/shoorick/737400

msginit ; xgettext ; msgmerge ; msgfmt ...

The tool will be moved to an official server of OSM

• rename the repositiory to PTNA
• TARGETHOST is 'localhost', so no "sftp" if name is 'localhost'
• TARGETLOC is a path (no changes required)
• TARGETURL is then "https://ptna.openstreetmap.de/..."
• WIKI_USERNAME no longer required (read only)
• WIKI_PASSWORD no longer required (read only)
• Create table with Networks and last changes and ... locally (no push to OSM wiki)

Modify

• cronjob.sh
• analyze-all-networks.sh
• analyze-network.sh

Create a new option to allow/disallow route_master/route=coach

'coach' is used somewhere in Germany for "Flixbus" lines, but the value has never been agreed on, neither was there a proposal process.

This option shall treat 'coach' as supported route value, otherwise as 'suspicious'

Flixbus in Europe for instance is a far distance bus (route=coach) like Greyhound in the US

Sometimes, the wget call to the overpass-api fails with error "4xx Too many requests".
In this case and also in other cases, the downloaded data file *-Data.xml is empty.
Further analysis is not possible.

It would be helpful to specify an option to analyze-network.sh to start download and analysis only if the file does not exist or is empty - just to start a second attempt only if the first failed.

consider railway=tram|light_rail|train as equivalent to psv=yes when used together with highway. I.e. tram shares lane with cars,...

consider route=ferry on a way as allowed access for a relation with route=ferry.

if public_transport != 2 (or not defined), check also 'role' against 'platform*' or 'stop*' (yes non-PTv2 routes might also use this if PTv is simply not set)

Show these types of _issues only if there is at least one stop_psotion in the route

• stop positions are optional

an additional check with an extra option --check-from-to

apply this option also to checking 'from' and 'to' with 'name' of route

Show these types of issues only if there is at least one stop_position

• stop_positions are optional

"Stadtbusnetz Passau" is wrong, should read "Verkehrsbetriebsgesellschaft Passau" and "VBP"

This is an enhancement of issue #3.

For the network DE-SN-VMS, there are several routes with for instance ref=A and route=bus and operator=RVE in different villages/cities. (see: https://www.vms.de/fileadmin/user_upload/antraege_formulare/tbest_anlage_6_2018-02-26.pdf)

To distinguish and to propperly assign them, 'from' and 'to' of the route have to be compared with the "From" and "To" of the CSV routes file, e.g.

• "A;bus;Kleiner Stadtverkehr Annaberg-Buchholz;Barbara-Uthmann-Ring;Buchholz;RVE"
• "A;bus;Kleiner Stadtverkehr Marienberg;Markt;Gewerbegebiet;RVE"

The list "Nicht berücksichtigte 'network'-Werte" of not considered 'network' values is not complete.
It does not include the 'network' values of those relations where 'ref' ist not set.

instead of this

rel(br.R); out; rel.R; out; rel(r.R); out; way(r.R); out; node(r.R); out;

try that or something similar (taht one doesn't work)

rel(br.R)->M; rel(br.R); out; ( rel.R; rel.M; )->A; rel.A; out; rel(r.A); out; way(r.A); out; node(r.A); out;