hrbrmstr / overpass Goto Github PK

So thanks to the work of @Robinlovelace, we both know there is a great deal of compatibility between overpass and osmdatar. So much in fact that i'd suggest the two be merged. The good news as far as I can discern it is:

1. overpass seems to be primarily focussed on enabling a very precise tool for constructing and passing overpass queries within R, while ...
2. osmdatar has the crudest of all possible ways of constructing queries, and focusses instead on the fastest possible processing of OSM objects within an R call.

The two thus seem to be to be entirely complementary, and merging could and would only help both enormously (from your side as shown by the speed comparisons of @Robinlovelace). osmdatar effectively just Rcpp-ifies all of your internal dplyr-type manipulation of the overpass XML data. I did have an explicit timing comparison with your approach here. Although both our packages have developed since then, the general pattern remains.

The division of labour of a potential merge would seem pretty obvious: The interface end would remain almost unchanged from overpass, while the processing currently done in way_utils and node_utils could simply be replaced by the osmdatar code. (Also note that polygon processing in osmdatar is almost complete.)

If you're interested, we could at least start by discussing the direction of the merge ... for which my only concern from an osmdatar perspective would be that I'm attempting to merely use overpass to access data, with the package primarily serving those data in a usable format. (This will ultimately also include returning street networks in igraph form for routing queries.) Thus the name overpass doesn't really reflect the breadth of what I envision the package achieving. But that's a very minor concern, and other than that I look forward to your insights

seems to not be behaving as expected

Can we create such wrappers to auto-download certain common osm features for a given bbox?

• Should there be a message (perhaps using a verbose parameter) displayed when the query is issued and complete?
• Do we want to return a list by default with all the nodes, ways and relations and let the user sort it out and/or provide higher-level functions to work on those return types?
• ways can be lines or polygons ref. It doesn't make sense to only return a SpatialLinesDataFrame if so. Therefore it might be prudent to return a list of individual way objects (whether they be SpatialLines or SpatialPolygons) but we'd also need to keep the metadata with them.
• relations can hold pretty much anything and there's a concept of inner & outer things, so the return here might need to be a list of various Spatial objects

And I def need some collaboration on next steps since we can clearly get data out of this API :-)

I don't know if this is new but there is the page http://overpass-api.de/api/status indicating the rate limiting and the next time at which a slot will be available. Currently it seems to be about one slot every minute or so.

It was interesting to look at http://overpass-api.de/munin/localdomain/localhost.localdomain/cpu.html too, apparently the server is quite busy (or at least that's what I understand from looking at it?).

Went through osmar and there's salvageable code.

Also went through sp a bit and I think I've got an idea for how to make a new Spatial class to hold composite geometries.

*.overpassql is now the recommended file extension for Overpass QL: OSM Wiki: Overpass QL § File extension
(Disclaimer: I added that section after some discussion.)

I propose to rename all Overpass QL files in this repo to *.overpassql for consistency.

Context: GitHub's syntax highlighting engine requires that 200 repositories use the same file extension before they consider supporting that syntax and file extension: github-linguist/linguist#5890

@hrbrmstr,
i've opened this issue on overpass because we now officially need your explicit consent on this one. osmdata has been submitted to ropensci with intended simultaneous JOSS consideration. The latter is effected by this paper.md file - your name really ought to appear in this list, but for that we really do need your explicit consent. Please!

It'd be best if you could state your consent within this osmdata issue. I'll also take this opportunity to thank you again for your code which has helped make osmdata much better than what it would have been without. I'm mighty pleased with the 'end' result of this project!
cheers!
mark

Experimenting, I tried to get all proposed new national cycle paths in the UK:

from_robin <- '[out:xml][timeout:25];
(
  node["ncn"="proposed"](49.16,-13.4,60.85,1.76);
  way["ncn"="proposed"](49.16,-13.4,60.85,1.76);
  relation["ncn"="proposed"](49.16,-13.4,60.85,1.76);
);
out body;
>;
out skel qt;'

frb <- overpass_query(from_robin)

Error in `row.names<-.data.frame`(`*tmp*`, value = value) : 
  duplicate 'row.names' are not allowed
In addition: Warning message:
non-unique value when setting 'row.names': ‘299893104’ 

Can anyone reproduce this?

ref: http://stackoverflow.com/questions/41388724/overpass-does-not-find-all-cinemas-overpass-api-documentation?noredirect=1#comment69979906_41388724

Reproducible example:

# Requirements:
# devtools::install_github ('osmdatar/osmdatar')
library(osmdatar)
# devtools::install_github("hrbrmstr/overpass")
library(overpass)

b = structure(c(-1.80036221446736, 53.6990006231171, -1.29035539895422,
                53.9458885889733), .Dim = c(2L, 2L), .Dimnames = list(c("x",
                                                                        "y"), c("min", "max")))
# download road network
system.time({ # 8s
  r = get_lines(bbox = b, key = "highway", value = "primary") # slow
})

system.time({ # nearly 70 s
  from_robin <- '[out:xml][timeout:100];
(
  node["highway"="primary"](53.6990006231171,-1.80036221446736,53.9458885889733,-1.29035539895422);
  way["highway"="primary"](53.6990006231171,-1.80036221446736,53.9458885889733,-1.29035539895422);
  relation["highway"="primary"](53.6990006231171,-1.80036221446736,53.9458885889733,-1.29035539895422);
);
out body;
>;
out skel qt;'

  frb <- overpass_query(from_robin)
})

cc @mpadge - how is it so much faster? Even more weirdly

> object.size(frb) / 1000000
3.908632 bytes
> object.size(r) / 1000000
4.145232 bytes

Posit (alpha minimum working in 0.1.0.9000):

opq(bbox="43.0135509,-70.8229993,43.0996118,-70.7280563") %>% 
  add_feature("amenity", "pub", ) %>% 
  add_feature("amenity", "restaurant") %>% 
  add_feature("amenity", "library") %>% 
  issue_query() -> reading_noms

plot(reading_noms)