Giter VIP home page Giter VIP logo

overpass-api-python-wrapper's Introduction

Overpass API python wrapper

GitHub branch check runs PyPI - Version PyPI - Downloads PyPI - License PyPI - Python Version GitHub License Mastodon Follow

Python bindings for the OpenStreetMap Overpass API.

shell recording

Install it

pip install overpass

Usage

API() constructor

First, create an API object.

import overpass
api = overpass.API()

The API constructor takes several parameters, all optional:

endpoint

The default endpoint is https://overpass-api.de/api/interpreter but you can pass in another instance:

api = overpass.API(endpoint="https://overpass.myserver/interpreter")

timeout

The default timeout is 25 seconds, but you can set it to whatever you want.

api = overpass.API(timeout=600)

debug

Setting this to True will get you debug output.

Getting data from Overpass: get()

Most users will only ever need to use the get() method. There are some convenience query methods for common queries as well, see below.

response = api.get('node["name"="Salt Lake City"]')

response will be a dictionary representing the JSON output you would get from the Overpass API directly.

Note that the Overpass query passed to get() should not contain any out or other meta statements. See verbosity below for how to control the output.

Another example:

>>> print [(
...     feature['properties']['name'],
...     feature['id']) for feature in response["features"]]
[(u'Salt Lake City', 150935219), (u'Salt Lake City', 585370637)]

You can find more examples in the examples/ directory of this repository.

The get() method takes a few parameters, all optional having sensible defaults.

verbosity

You can set the verbosity of the Overpass query out directive using the same keywords Overpass does. In order of increased verbosity: ids, skel, body, tags, meta. As is the case with the Overpass API itself, body is the default.

>>> import overpass
>>> api = overpass.API()
>>> data = api.get('way(42.819,-73.881,42.820,-73.880);(._;>;)', verbosity='geom')
>>> [f for f in data.features  if f.geometry['type'] == "LineString"]

(from a question on GIS Stackexchange)

responseformat

You can set the response type of your query using get()'s responseformat parameter to GeoJSON (geojson, the default), plain JSON (json), CSV (csv), and OSM XML (xml).

response = api.get('node["name"="Salt Lake City"]', responseformat="xml")

If you choose csv, you will need to specify which fields you want, as described here in the Overpass QL documentation. An example:

response = api.get('node["name"="Springfield"]["place"]', responseformat="csv(name,::lon,::lat)")

The response will be a list of lists:

[
    ['name', '@lon', '@lat'],
    ['Springfield', '-3.0645656', '56.2952787'],
    ['Springfield', '0.4937446', '51.7487585'],
    ['Springfield', '-2.4194716', '53.5732892'],
    ['Springfield', '25.5454526', '-33.9814866'],
    ....
]

build

We will construct a valid Overpass QL query from the parameters you set by default. This means you don't have to include 'meta' statements like [out:json], [timeout:60], [out body], etcetera. You just supply the meat of the query, the part that actually tells Overpass what to query for. If for whatever reason you want to override this and supply a full, valid Overpass QL query, you can set build to False to make the API not do any pre-processing.

date

You can query the data as it was on a given date. You can give either a standard ISO date alone (YYYY-MM-DD) or a full overpass date and time (YYYY-MM-DDTHH:MM:SSZ, i.e. 2020-04-28T00:00:00Z). You can also directly pass a date or datetime object from the datetime library.

Pre-cooked Queries: MapQuery, WayQuery

In addition to just sending your query and parse the result, overpass provides shortcuts for often used map queries. To use them, just pass them like to normal query to the API.

MapQuery

This is a shorthand for a complete ways and relations query in a bounding box (the 'map call'). You just pass the bounding box to the constructor:

MapQuery = overpass.MapQuery(50.746,7.154,50.748,7.157)
response = api.get(MapQuery)

WayQuery

This is shorthand for getting a set of ways and their child nodes that satisfy certain criteria. Pass the criteria as a Overpass QL stub to the constructor:

WayQuery = overpass.WayQuery('[name="Highway 51"]')
response = api.get(WayQuery)

Testing

Using pytest.

py.test

FAQ

I need help or have an idea for a feature

Create a new issue.

Where did the CLI tool go?

The command line tool was deprecated in version 0.4.0.

See also

There are other python modules that do similar things.

overpass-api-python-wrapper's People

Contributors

andrewscarani avatar atlefren avatar b-jazz avatar dericke avatar donfaq avatar eumiro avatar fju avatar gendelf avatar johnjohndoe avatar jose1711 avatar luftj avatar mharnold avatar mvexel avatar nebulon42 avatar pe-perry avatar russbiggs avatar scott-vsi avatar t-g-williams avatar tbolender avatar themiurgo avatar thepolytheist avatar thomdietrich avatar trypy avatar ydong-vsi avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

overpass-api-python-wrapper's Issues

Error key error with center

With the following code I get

q = '(node["amenity"="restaurant"](around:10000,41.978436,2.813442);way["amenity"="restaurant"](around:10000,41.978436,2.813442);relation["amenity"="restaurant"](around:10000,41.978436,2.813442););out body center;'
print(api.Get(q))

this exception
Traceback (most recent call last):
File "/home/xevi/gits/overpass-api-python-wrapper/overpass/cli.py", line 29, in
print(api.Get(q))
File "/home/xevi/gits/overpass-api-python-wrapper/overpass/api.py", line 65, in Get
return self._asGeoJSON(response["elements"])
File "/home/xevi/gits/overpass-api-python-wrapper/overpass/api.py", line 131, in _asGeoJSON
for coords in elem["geometry"]:
KeyError: 'geometry'

Coding convention

I noticed that some parts differs from code style conventions like PEP-8. For methods camel case is used, attributes are only written in lower cases without underscores etc.
Regarding future work I would propose a small convention what to use when (and following parts of PEP-8 to make use more simple for others?)

Known maximum length of number of ways/nodes to request in single query

Seems that overpass will only accept a maximum of 1023 elements per query.

For instance this works fine:

import overpass
api = overpass.API()

#t looks something like (node(-1); node(-1)...;)
t = ('(' + ';'.join(['way(-1)' for v in xrange(0,1023)]) + ';)')
api.Get(t)`

But this gives an OverpassSyntaxError:

t = ('(' + ';'.join(['way(-1)' for v in xrange(0,1024)]) + ';)')
api.Get(t)`

I can't seem to find anything documentation on the maximum number of elements you can query at once. It would be nice to have a more detailed error if this is an exception that can be specifically handled

Add CI

Add this project to Travis or similar for CI, as suggested in #18

unable to use querries with alternatives

I expected that

import overpass
api = overpass.API()
query="node(31134192);node(31134242);"
print(api.Get(query, responseformat="xml"))

will return both http://www.openstreetmap.org/node/31134192 and http://www.openstreetmap.org/node/31134242

but only second node was returned

<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="Overpass API">
<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>
<meta osm_base="2017-09-25T09:14:02Z"/>

  <node id="31134242" lat="53.6688600" lon="17.4807500">
    <tag k="addr:postcode" v="77-300"/>
    <tag k="is_in:county" v="powiat człuchowski"/>
    <tag k="is_in:municipality" v="gmina Człuchów"/>
    <tag k="is_in:province" v="województwo pomorskie"/>
    <tag k="name" v="Brzeźno"/>
    <tag k="place" v="village"/>
    <tag k="population" v="258"/>
    <tag k="source:population" v="wikipedia"/>
    <tag k="teryt:rm" v="01"/>
    <tag k="teryt:simc" v="0742730"/>
    <tag k="teryt:stan_na" v="2009-01-01"/>
    <tag k="teryt:updated_by" v="teryt2osm combine.py v. 43"/>
    <tag k="wikipedia:en" v="Brzeźno, Człuchów County"/>
  </node>

</osm>

Adding proper user agent to HTTP requests

Currently, overpass api python wrapper shows up in the log files as user agent: python-requests/2.2.1 .... Any chance to have this set to something more meaningful by default?

please consider including tests/ dir in pypi release

Hello,
i noticed the github repo has a tests/ directory, while the tarball on pypi doesnt: could you consider adding it? it would be helpful for distributions to verify everything is working as expected and generally it's a nice practice to be able to run tests of a third-party modules.

thanks for considering,
Sandro

Encoding

I am having problem with german umlauts like ä, ö, ü or ß....; the response for one request is:

{u'elements': [{u'lat': 47.861882, u'lon': 15.202494, u'type': u'node', u'id': 2949081313, u'tags': {u'natural': u'peak', u'man_made': u'survey_point', u'alt_name': u'Vaterberg', u'name': u'\u0102\x96tscher', u'ele': u'1893'}}], u'version': 0.6, u'osm3s': {u'timestamp_osm_base': u'2015-08-27T11:14:02Z', u'copyright': u'The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.'}, u'generator': u'Overpass API'}

The name Tag shoule be "Ötscher"...but this seems to be no valid utf8 encoding as name.encode('utf-8') produces no valid results....

Output formats

I notice that it only accept json output format, while I would like to have the output in XML. Could the other output options be available as well?

Example bounding box map query call does not contain all the nodes

I've noticed that using the example in the readme that there are missing nodes for node ids referenced in ways . For example running this in Python:

api = overpass.API(timeout=600)
map_query = overpass.MapQuery(50.746,7.154,50.748,7.157)
response = api.Get(map_query, responseformat="json")
unique_nodes = []
way_nodes = []
for element in response['elements']:
    if element['type'] == 'way':
        way_nodes.extend([str(x) for x in element['nodes']])
    if element['type'] == 'node':
        id = str(element['id'])
        unique_nodes.append(id)
diff = list(set(way_nodes) - set(unique_nodes))
print diff

results in a set of node ids that are not found from the response call. Any thoughts on this?

Missing Nodes:

['1212226190', '2661510421', '4245927048', '1212226047', '2661537835', '2661510427', '1212226113', '1212226110', '4245927044', '4245927046', '2658130490', '2658130496', '1212226181', '4245926776', '1212226052', '1558201832', '2661510414', '2661510416', '1558201823', '1558201825', '1558201826', '1558201827', '291092263', '96140183', '4241842210', '2658130489', '1212226024', '2661510278', '2661510404', '4241842209', '4241842208', '1558201836', '1212226130', '1212226131', '1558201831', '96133778', '96133776', '1212226220', '2661510260', '2661510266', '96133775', '1212226038', '2661537867', '1212226031', '1212226030', '274246241', '1212226107', '1212226125', '32346787', '4241842211', '4083385469', '2658130507', '2661537830', '2661537832', '1212226085', '1212226159', '2464702124', '2464702125', '2464702122', '1212226042', '2464702129', '1212226203', '1212226139', '2661537846', '2661537849', '291089398', '291089396', '96133777', '96133774', '1647008854', '470930365', '2464702134', '2464702130', '2464702133', '2464702132', '2661537863', '2464702123', '2661537823', '4246007583', '4246007580', '1647095994', '2519501893', '1212226067', '2519501898', '1558201815', '2661537824', '1558201812', '291438873', '291438871', '2661537825', '291438874', '1212226166', '33078442', '2661538021', '2519501886', '2661537826', '291435959', '291435958', '4245927053', '2661537828', '4245927051', '1212226061', '4245927055']

In import please substitute httplib module with http.client module

Hi trying to run the code with debug, if i set debug on

in api = overpass.API(timeout=600, debug=True)

i get httplib module import error

to made it works in the file /usr/local/lib/python3.5/dist-packages/overpass/api.py

i must set this code
import http.client

and then i must change:
httplib.HTTPConnection.debuglevel = 1

with
http.client.HTTPConnection.debuglevel = 1

as below:
...
if self.debug:
import http.client
import logging
http.client.HTTPConnection.debuglevel = 1
...

search returning empty objects

Something have changed in either the code or the API since April/2016. A script that was working fantastically in April (and maybe in May) are only returning empty strings. Search queries [out:json];relation["type"="boundary"]["admin_level"="2"]["boundary"="administrative"](-34.0,-51.0,5.0,-28.0); and [out:json];relation["type"="boundary"]["admin_level"="2"]["boundary"="administrative"](-34.0,-51.0,5.0,-28.0);out body; and [out:json];relation["type"="boundary"]["admin_level"="2"]["boundary"="administrative"](-34.0,-51.0,5.0,-28.0);(._;>;);out body; all give empty response object back from the api. Running the last search in overpass-turbo returns correctly, the boundary relation for Brazil with all members. When the script was working, the first query sentence gave only the relation object of Brazil, which is what I need for further processing.

No changes have been done to the script since last successful run (except swapping between different alternatives of the searchString variable), though the PIP and the python version have been updated.

Below is a snippet from my log of the execution of the script:

2016/07/08 16:14:24: gpxupload INFO - Opened success.gpx/Upload/156VIRB0245.gpx
2016/07/08 16:14:24: gpxupload DEBUG - GPX File contains 1038 trackpoints with 1.423 km.
2016/07/08 16:14:24: gpxupload DEBUG - [out:json];relation["type"="boundary"]["admin_level"="2"]"boundary"="administrative";(._;>;);out body;
2016/07/08 16:14:25: requests.packages.urllib3.connectionpool INFO - Starting new HTTP connection (1): overpass-api.de
2016/07/08 16:14:39: requests.packages.urllib3.connectionpool DEBUG - "POST /api/interpreter HTTP/1.1" 400 461
2016/07/08 16:14:39: gpxupload ERROR - No search result returned

Add overpassify as a recommended libarary

If you did, then you could have an example like:

import overpass
from overpassify import overpassify

api = overpass.API()


@api.Get
@overpassify
def response():
    search = Area(3600134503)
    ways = Way(search, highway=...)
    nodes = Node(search)
    out(ways, geom=True, count=True)
    out(nodes, geom=True, count=True)
    noop()


print(response)

Everything is written in Python, yet you still get to make an OverpassQL query.

Timestamp

It would be great to include the timestamp field

Raising exceptions

Instead of constructing json error messages exceptions should be thrown to show that something went bad.

Example code in readme broken

When I try

print( [(feature['tags']['name'], feature['id']) for feature in response['elements']] )

I get the error

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'elements'

This works, though:

print( [(feature['properties']['name'], feature['id']) for feature in response['features']] )

don't require full QL syntax

Full Overpass QL includes stuff that we wouldn't want to bother the user with:

[out:json];
node["name"="Gielgen"];
out body;

The user should only have to pass in

node["name"="Gielgen"]

Documentation

I would like this project to have nice documentation on readthedocs. I did this with Sphinx on another project, maproulette-api-wrapper. See the docs/ folder there.

Support Python 3

I tried installing 98e731f with pip, but it fails with a SyntaxError on the use of the print statement.

searchString with ;out meta; give double positives

I notice that the search string URL automatically adds out body, this results that my search with out meta in the string returns double positives (each object two times with different level on appended data). Would it be possible to drop the auto-append of out body when search string ends with other out statements?

Adding compression option

I don't seem to find a way to have compression enabled for HTTP responses. Could you add that as an option?

Maintainer

Hi all,

I have too little time to maintain this project properly. Who would be interested in sharing this responsibility? @itiboi perhaps?

Get nodes within country borders

Hi,

Thanks so much for this excellent repo! I am trying to build a database of power plants from Open Street Map. This is the code I'm using:

import overpass
api = overpass.API(timeout=900)
response = api.Get('node["power" = "plant"]')
print(response)

I'm getting a nested dictionary of power plants, but only 187. I suspect there must be many more. Is there a better way to query? I eventually want to help build a database of power plants by country from Open Street Map and welcome any help on how to query by country as well.

Thank you so much!!

query not accepted

>>> api.Get('{{geocodeArea:Alabama}}->.state;relation[type=restriction](area.state);out body;>;')
[out:json];{{geocodeArea:Alabama}}->.state;relation[type=restriction](area.state);out body;>;out body geom;
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mvexel/dev/overpass-api-python-wrapper/overpass/api.py", line 45, in Get
    raw_response = self._GetFromOverpass(full_query)
  File "/Users/mvexel/dev/overpass-api-python-wrapper/overpass/api.py", line 108, in _GetFromOverpass
    raise OverpassSyntaxError(query)
overpass.errors.OverpassSyntaxError: [out:json];{{geocodeArea:Alabama}}->.state;relation[type=restriction](area.state);out body;>;out body geom;

Query runs fine when run from Overpass Turbo: http://overpass-turbo.eu/s/diB

not possible to use the recurse operator in OSM query

as query is composed, there is always a out body geom at the end of it, so it looks like it's not possible to use specify the recurse operator, at least based on what overpass translator proposes, which is <query>;out body;>;out skel;

Is it possible to get the boundary coordinates?

The overpass API mentions that it is possible to get the boundary coordinates of a location, for example, a city. I cannot figure out how to do this through this python module. Can anyone help me?

API(debug=True) is not python3 compatible

In [39]: api = overpass.API(timeout=3600, debug=True)
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-39-67bc1e87daf5> in <module>()
----> 1 api = overpass.API(timeout=3600, debug=True)

/usr/lib/python3/dist-packages/overpass/api.py in __init__(self, *args, **kwargs)
     26 
     27         if self.debug:
---> 28             import httplib
     29             import logging
     30             httplib.HTTPConnection.debuglevel = 1

ImportError: No module named 'httplib'

thats because httplib has been renamed to http.client in py3k, so the usual try/import/except/import dance is needed here

requests.exceptions.SSLError

Hi I am trying to hit overpass api using your wrapper package and am getting this error. It was working fine yesterday but today it weirdly stopped.

Would you have a solution to it ? Thanks

Traceback (most recent call last):
File "C:\HitOverpassAPI.py", line 152, in
response = api.Get('node["name"="Salt Lake City"]')
File "C:\ProgramData\Anaconda2\lib\site-packages\overpass\api.py", line 48, in Get
raw_response = self._GetFromOverpass(full_query)
File "C:\ProgramData\Anaconda2\lib\site-packages\overpass\api.py", line 101, in _GetFromOverpass
headers={'Accept-Charset': 'utf-8;q=0.7,*;q=0.7'}
File "C:\ProgramData\Anaconda2\lib\site-packages\requests\api.py", line 110, in post
return request('post', url, data=data, json=json, **kwargs)
File "C:\ProgramData\Anaconda2\lib\site-packages\requests\api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "C:\ProgramData\Anaconda2\lib\site-packages\requests\sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "C:\ProgramData\Anaconda2\lib\site-packages\requests\sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "C:\ProgramData\Anaconda2\lib\site-packages\requests\adapters.py", line 497, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)

self._timeout is used in the TimeoutError exception even if it has been change in self.timeout

In [27]: api = overpass.API(timeout=3)

In [28]: response = api.Get(s)
---------------------------------------------------------------------------
TimeoutError                              Traceback (most recent call last)
<ipython-input-28-cee6cde6e022> in <module>()
----> 1 response = api.Get(s)

/usr/lib/python3/dist-packages/overpass/api.py in Get(self, query, responseformat, verbosity)
     43 
     44         # Get the response from Overpass
---> 45         raw_response = self._GetFromOverpass(full_query)
     46 
     47         if responseformat == "xml":

/usr/lib/python3/dist-packages/overpass/api.py in _GetFromOverpass(self, query)
    100 
    101         except requests.exceptions.Timeout:
--> 102             raise TimeoutError(self._timeout)
    103 
    104         self._status = r.status_code

TimeoutError: 25

that's because it uses self._timeout (always set to 25) but the API class defines the timeout as

self.timeout = kwargs.get("timeout", self._timeout)

and it's the correct value used to determine if it has timed out or not, so i think that's the value to be used in the exception as well

support non-geo queries

I have some use cases that call for output that is not spatial (just relation ids and their tags and metadata).

Looking into supporting this, how do you think this would work best? Options I can think of:

  1. separate method
  2. parameter passed into Get() method
  3. automatic discovery

First would be easiest to implement, going to start a little prototyping based on that option, but please weigh in on how this should be implemented

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.