csparpa / pyowm Goto Github PK

The module utils.py concentrates utility functions about either time and temperature units conversions.

First, it should be better to split these functions into two distinct modules (i.e: temputils.py and the already existent timeutils.py)

Second, the whole codebase contains calls to the above-mentioned functions in a fashion similar to the following:

# Taken from weathers.py
def get_reference_time(self, timeformat='unix'):
  #[...docstring...]
  if timeformat == 'unix':
    return self._reference_time
  elif timeformat == 'iso':
    return converter.UNIXtime_to_ISO8601(self._reference_time)
  else:
    raise ValueError("Invalid value for parameter 'format'")

It would be better to move the IFs down to the utility functions and have this instead:

def get_reference_time(self, timeformat='unix'):
  #[...docstring...]
  return converter.format_time(self._reference_time, timeformat)

Parsing fails for some stations parsing fails with exception.

https://gist.github.com/alexschultze/5c94bb13422c44d06161

here is an example that fails:

{u'clouds': {u'all': 0}, u'name': u'Berlin Koepenick', u'visibility': 10000, u'sys': {u'country': u'DE', u'sunset': 1427304457, u'message': 0.0141, u'type': 1, u'id': 4878, u'sunrise': 1427259349}, u'weather': [{u'main': u'Clear', u'id': 800, u'icon': u'01n', u'description': u'Sky is Clear'}], u'coord': {u'lat': 52.44, u'lon': 13.55}, u'base': u'stations', u'dt': 1427327222, u'main': {u'pressure': 1004, u'temp_min': 282.15, u'temp_max': 284.26, u'temp': 283.39, u'humidity': 70}, u'id': 2885657, u'wind': {u'speed': 3.6, u'deg': 90}, u'cod': 200}

Modify function weather_history in module webapi25/owm25.py introducing a cnt parameter (defaulting to None|) which limits the number of items retrieved from the web API.

Caveat: the "1-offset" bug!

A cache mechanism should be added to the library, so that responses to OWM web API queries that had been issued time ago (the time amount should be somehow configurable) are returned to the clients directly from the cache and further OWM web API queries are not issued.

This saves time and bandwidth.

The cache mechanism should be enough abstract to let clients use their own cache providers (eg: memcached, redis, etc.), but a default simple cache implementation should be given by the library itself.

This code in the CityIDRegistry class doesn't work for me in python 3.3.4.

def _match_line(self, city_name, lines):
        for line in lines:
            if line.startswith(city_name.lower()):
                return line.strip()
        return None

So I can't lookup a city like this

registry = owm.city_id_registry()
result = registry.location_for("London")

Because it fails like this.

TypeError: startswith first arg must be bytes or a tuple of bytes, not str

I believe the problem is that the lines variable contains bytes, not strings. This means that if line.startswith(city_name.lower()): fails when city_name is a string.

To compound this, I can't simply pass in bytes like this.

registry = owm.city_id_registry()
result = registry.location_for(bytes("London", 'utf8'))

because CityIDRegistry._assess_subfile_from uses ord() which requires a string.

A solution may be to add a line like this.

def _match_line(self, city_name, lines):
        city_name = bytes(city_name.lower(), 'utf8')
        for line in lines:
            if line.startswith(city_name.lower()):
                return line.strip()
        return None

Class StationHistory should be broken down into two separate coooperating classes: a databox and an abstraction provider, similarly to the classes Forecast and Forecaster.

As read on the OWM API wiki:

JSON returns data from weather stations located within definite rectangle that is specified by geographical coordinates.

Parameters:

• bbox = bounding box, it's a list of numbers: lon of the top left point, lat of the top left point, lon of the bottom right point, lat of the bottom right point
• cluster = use server clustering of points, possible values are [yes, no]
• cnt = results pagination
  Example: http://api.openweathermap.org/data/2.5/box/station?cluster=no&cnt=5&bbox=65.0,41.0,69.21,49.26

{"cod":"200","cnt":2,"list":[{"id":7203,"dt":1415187000,"name":"UAOO","type":1,"coord":{"lat":44.7069,"lon":65.5925},"main":{"temp":5,"pressure":1021},"wind":{"speed":4,"deg":350},"rang":50,"clouds":[{"condition":"CAVOK"}]},{"id":7201,"dt":1415185200,"name":"UAKD","type":1,"coord":{"lat":47.7083,"lon":67.7333},"main":{"temp":3,"pressure":1021,"humidity":74},"wind":{"speed":6,"deg":350},"rang":50,"clouds":[{"distance":640,"condition":"BKN"},{"distance":823,"condition":"OVC"}]}]}

We need to introduce at least:

• owm25.weather_at_stations_in_bbox(lat_top_left, lon_top_left, lat_bottom_right, lon_bottom_right, cluster=False, cnt=None)

Currently there is no way to specify the paid OWM web API root URL but in code: this should be done via configuration.
The API root URL is in module pyowm.webapi25.configuration25.py

# OWM web API URLs
ROOT_API_URL = 'http://api.openweathermap.org/data/2.5'

There should be a way to specify the Pro OWM web API root URL (which is http://pro.openweathermap.org/data/2.5) via configuration

We want the PyOWM library to be usable also in proxied network environments.

We also want that the proxy configuration be made through application configuration, as well as with environment variables.

Reference: http://stackoverflow.com/questions/3168171/how-can-i-open-a-website-with-urllib-via-proxy-python

The actual wiki pages provided by GitHub are updated every time a new release is issued: this means that they get out of sync with previous library versions.

We need a way to keep track of the "old versions" of the Wiki and to relate them to a specific reference library version.

The idea is to implement the Wiki using a separate tool than GitHub and host it somewhere (eg: GitHub pages)
A good tool is mkdocs

It seems something has been changed.

Traceback (most recent call last):
File "pyo.py", line 9, in
observation = owm.weather_at_place('London,uk')
File "/usr/local/lib/python2.7/dist-packages/pyowm/webapi25/owm25.py", line 153, in weather_at_place
return self._parsers['observation'].parse_JSON(json_data)
File "/usr/local/lib/python2.7/dist-packages/pyowm/webapi25/observationparser.py", line 60, in parse_JSON
w = weather.weather_from_dictionary(d)
File "/usr/local/lib/python2.7/dist-packages/pyowm/webapi25/weather.py", line 430, in weather_from_dictionary
if 'distance' in d['visibility']:
TypeError: argument of type 'int' is not iterable

The portability assessment shall lead us to decide between one of the following actions:

• patch pyowm so that it is runnable under a Python 3.x env
• write a parallel pyowm releases branch specific for Python 3.x users

A good start is to run the 2to3.py utility

A more convenient way is to use tox

When executing:

>>> from pyowm import OWM
>>> owm = OWM('apikey', '2.5')

the library loads the configuration options found into the configurationXX.py relative to the OWM API version which is used (in this case, version 2.5).

The loading of this module is wired into the code, and this is bad. We want the configuration module's Python path to be injectable while instantiating the library main entry point, something like this:

>>> owm = OWM('apikey', '2.5', 'package.subpackage.my_configmodule' )

The user-provided config module shall be visible in the PYTHONPATH scope.
Of course, proper fallback default values shall be provided for config options that are not correctly set / can't be found into the user-provided module.

This might be the chance to revisit the main OWM factory interface - in view of a next major release

Benefits: it will make code contributions and code management easier.

References: @nvie's blog post

To prevent possible XML element name collisions in PyOWM clients, it would be good to provide a XMLNS (reference) for the XML strings returned by entity modules.

The solution could be providing an XSD file (reference) along with PyOWM code

Of course this needs a paid subscription.

Has anyone got a personal one that can be shared?

I want to use Sourcegraph code search and code review with pyowm. A project maintainer needs to enable it to set up a webhook so the code is up-to-date there.

Could you please enable pyowm on @sourcegraph by going to https://sourcegraph.com/github.com/csparpa/pyowm and clicking on Settings? (It should only take 15 seconds.)

Thank you!

I would be interested in adding support for latitude/longitude querying of forecasts as it now appears the API supports these parameters, seen here:
http://openweathermap.org/forecast

For example:
Call by geographic coordinats at JSON format:
http://api.openweathermap.org/data/2.5/forecast?lat=35&lon=139

The weatherutils.py provides utilities for checking detailed weather statuses against predefined values (i.e: check if status of a Weather object is 'rainy' or similar)

The checks are performed against several English word lists named *_KEYWORDS and located in module pyowm.webapi25.configuration25. This pattern is not robust and must be re-engineered in order to allow future multilingual support.

For example, a possible new type of check might be against OWM weather codes instead of English weather keywords

Related modules: forecaster.py

Class OWM25 is under focus. Make sure the function names are clear and convey what they really do.

This task could lead to a change in the library interface: this means that users must be notified of this non retro compatibility and docs must be updated, as well as code usage wiki and example snippets, and README.md

This also includes changes introduced with #17

As read on the OWM API wiki:

Get the most recent measurements from weather station, ID = STATION_ID
Data can be called in JSON or HTML.

Example: http://api.openweathermap.org/data/2.5/station?id=1000

{"station":{"name":"KPPQ","type":1,"status":50,"id":1000,"coord":{"lon":-90.47,"lat":39.38}},"last":{"main":{"temp":279.15,"pressure":1022,"humidity":87},"wind":{"speed":2.1,"deg":230},"visibility":{"distance":16093,"prefix":0},"calc":{"dewpoint":277.15,"humidex":278.15},"clouds":[{"condition":"CLR"}],"dt":1415187300},"params":["temp","pressure","humidity","wind","visibility"]}

We need to introduce at least:

• owm25.weather_at_station(id)
• a new specific parser StationObservationParser, as the JSON is quite different from the one returned when querying for weather on city IDs. Next step then could be to refactor commonalities between the 2 parsers

First, Great library!

Second, I found the position of lat long coordinates in weather_at_coords(long; lat) Counterintuitive, google api, openweather uses (lat, long) por geopoints.

Eg: next_hour, next_week

Check if codebase adheres to the style guidelines stated in PEP-8.

Also this guide is useful to read

Tools like pep8 should help.

Similar to Forecaster class. A few useful additions:

max_temperature()  # returns tuple (<max_temp>, <timestamp>)
max_humidity()  # as above
max_pressure()
max_rain()

min_temperature()  # returns tuple (<min_temp>, <timestamp>)
min_humidity()  # as above
min_pressure()
min_rain()

average_temperature() # returns avg temperature
average_humidity()
average_pressure()
average_rain()

API keys may contain non ASCII characters, and this would cause crashes

Entity classes such as Weather, Forecast, Location, etc.. should be equipped with a __repr__ hook so that they can be pretty-printed

It is possible to invoke OWM web API endpoints specifying a language parameter for output weather descriptions (reference), eg:

http://api.openweathermap.org/data/2.5/weather?q=London,uk&lang=zh_cn

As of version 0.X.Y, the PyOWM lib performs weather queries without specifying any language parameters, so the output weather descriptions are defaulted to English.

It would be desirable for end-users to be allowed to specify the language of output data: this shall be optional, of course.

This new feature requires a bit of refactoring, and fixing of issue #12

Lines such as

#!/usr/bin/env python

should be removed from all source files, as this is a library and cointains no directly-executable Python scripts (so no portability issues).

The library should be tested (unit-tests and functional-tests) against a Python 3.x installation and assessments should be made to understand if code can be patched in order to support Python 3.x while mantaining retro-compatibilty with 2.6/2.7 versions.

Possible steps:

1. install a virtualenv with a Python 3.0 interpreter
2. install PyOWM on it and run the the full test suite, tracking down what goes wrong

Hello,

unicode strings should be supported.

I'm using click as CLI parser http://click.pocoo.org/

@click.command()
@click.option('--api_key', default='', help=u"API Key for OWM")
@click.option('--place', default='Milan,it', help=u"Place")
def main(api_key, place):
    pass

I think your asserts to test if a parameter is a string are too restrictive and force people to convert to str which is not necessary for simple unicode string and which is not very "pythonic" (duck typing...).

For example I noticed
https://github.com/csparpa/pyowm/blob/master/pyowm/webapi25/owm25.py

assert type(API_key) is str, "If provided, 'API_key' must be a str"

it should be

assert(isintance(API_key, basestring))

Kind regards

Retrieval of currently observed weather and of daily/3h forecasts can be done on the OWM web API also using the city ID as search key.

We need to decide if this is reasonably going to be supported by PyOWM (does it make sense for the users? If users don't know the IDs they first need to retrieve it) and hence implement the new features.

This may also need - in case of implementation - a review in the naming of functions in module owm25.py.

In case the functionality won't be implemented, state the reasons for the choice on Github project's wiki.

As read on the OWM API wiki:

You can choose a geo point and expected number of weather station to get a list of weather stations > that are located around this geo point.
Example: http://api.openweathermap.org/data/2.5/station/find?lat=55&lon=37&cnt=2

[{"station":{"name":"rvb.name","type":5,"status":1,"user_id":15032,"id":80678,"coord":{"lon":37.5617,"lat":55.4033}},"distance":57.285,"last":{"main":{"temp":277.75,"humidity":73,"pressure":98971},"dt":1415188194}},{"station":{"name":"UUMO","type":1,"status":50,"id":7325,"coord":{"lon":37.5,"lat":55.5}},"distance":63.995,"last":{"main":{"temp":276.15,"pressure":1013,"humidity":100},"wind":{"speed":4,"deg":250},"visibility":{"distance":4000,"prefix":0},"calc":{"dewpoint":276.15,"humidex":274.85},"clouds":[{"distance":183,"condition":"OVC"}],"weather":[{"precipitation":false,"descriptor":false,"intensity":false,"obscuration":"BR"}],"dt":1415185200}}]

This will need to introduce:

• owm25.station_at_coords(lat, lon, cnt=None)
• new class Station
• relevant parsers

owm25.weather_history_at_id(id)

@csparpa Hi!

I use:

obs = owm.weather_at_id(2940132)

It works not with all IDs. For example this station: http://api.openweathermap.org/data/2.5/station?id=121479

obs = owm.weather_at_id(121479)
ERROR: OWM API: observation data not available - response payload: 
{"message": "Error: Not found city", "cod": "404"}

All works fine but I got following error when run it from the crontab:

File "/home/user/myapp/pyowm/commons/owmhttpclient.py", line 64, in call_API
raise api_call_error.APICallError(str(e.reason), e)
pyowm.exceptions.api_call_error.APICallError: Exception in calling OWM web API.
Reason:
Caused by: HTTP Error 510:

Hello,

I had a look at
https://github.com/csparpa/pyowm/blob/master/scripts/generate_city_id_files.py

You might use Pandas:

import pandas as pd
df = pd.read_csv("http://openweathermap.org/help/city_list.txt", sep="\t")
print df
df = df.sort(['countryCode', 'nm'], ascending=['True', 'True'])
df.to_excel("city_list.xls")
df.to_csv("city_list.csv")

Pandas could also be great for weather history:
see https://github.com/scls19fr/openweathermap_requests

Kind regards

Something like:

>>> owm.is_API_up()
True  # If API is responsive, False otherwise

The check should be made by using a short timeout for API responses, in order to avoid hang-up

By far, model entity classes (such as Location, Weather, etc.) don't return any DOM-oriented representation of themselves: you can only retrieve their XML/JSON representations as Unicode objects, which are not "traversable".

The public interface for entity classes should:

• allow clients to retrieve DOM-oriented representations and traverse them directly
• allow clients to retrieve Unicode XML/JSON string representations

Something like:

# Current interface
location.to_XML()    # <-- gives a Unicode XML-formatted string
location.to_JSON()  # <-- gives a Unicode JSON-formatted string

# Desired interface
location.to_XML()  # <-- gives a Unicode XML-formatted string
location.to_JSON()  # <-- gives a Unicode JSON-formatted string
location.to_DOM()   # <-- gives an xml.etree.ElementTree.ElementTree object

OWM web API responses to queries for weather data on specific meteostations also include calc and visibility JSON blobs.
The calc blob has subvalues:

• dewpoint
• humidex
• heatindex

and visibility has subvalues:

• distance

Note that the blobs are not included in JSON responses when invoking other endpoints.

Eg:

# JSON contains calc and visibility
http://api.openweathermap.org/data/2.5/station/find?lat=40&lon=-73&cnt=1

{"last":{"main":{"temp":285.15,"pressure":1022,"humidity":66},"wind":{"speed":2.1,"deg":0},"visibility":{"distance":16093,"prefix":0},"calc":{"dewpoint":279.15,"humidex":284.75},"clouds":[{"condition":"CLR"}],"dt":1415674260}

# JSON does not contain calc and visibility
http://api.openweathermap.org/data/2.5/weather?lat=35&lon=139

{"coord":{"lon":139,"lat":35},"sys":{"message":0.0208,"country":"JP","sunrise":1415654076,"sunset":1415691687},"weather":[{"id":501,"main":"Rain","description":"moderate rain","icon":"10n"}],"base":"cmc stations","main":{"temp":282.855,"temp_min":282.855,"temp_max":282.855,"pressure":981.73,"sea_level":1025.91,"grnd_level":981.73,"humidity":100},"wind":{"speed":1.67,"deg":10.5005},"clouds":{"all":92},"rain":{"3h":5},"dt":1415698000,"id":1851632,"name":"Shuzenji","cod":200}

These blobs are currently unsupported but should be.

Patches to be done in Weather class:

• functionweather_from_dictionary() will need to be patched accordingly. As usual, a Weather object will be returned even if these new JSON blobs cannot be parsed/are missing.
• add functions get_dewpoint(), get_humidex() , get_heatindex(), get_visibility_distance(). Each of them shall return the numeric value of the corresponding property or None if they are empty
• patch to_JSON() and to_XML() functions

Because I would like automatic update from server, could this be done in Flask or do I need to use Tornado? Thanks

Seb

This will affect version 3.0 as it is a breaking change in the library API.

We need to normalise the weather retrieval functions to the naming pattern:

• forecast_at_place(self, name, forecast_granularity, [limit])
• forecast_at_coords(self, lat, lon, forecast_granularity, [limit])
• forecast_at_id(self, id, forecast_granularity, [limit])

where "forecast_granularity" can be either "3h" or "daily"

A sample file describing the problem:

http://api.openweathermap.org/data/2.5/forecast/daily?q=Winnipeg&mode=json&units=metric&cnt=16

You can see the 'speed' key in the JSON with no 'wind'.

My program was working yesterday, and as far as I remember nothing was changed since then. Now all of a sudden I'm getting the above error. Someone else had the same issue but closed the topic before getting a response.

Exception in Tkinter callback
Traceback (most recent call last):
File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 1489, in call
return self.func(*args)
File "FSE code.py", line 65, in weatherwindow
app.weather() # call weather function
File "FSE code.py", line 46, in weather
f = owm.weather_at_id(6182962) # pull forcast for Windsor Ontario
File "build/bdist.linux-i686/egg/pyowm/webapi25/owm25.py", line 200, in weather_at_id
return self._parsers['observation'].parse_JSON(json_data)
File "build/bdist.linux-i686/egg/pyowm/webapi25/observationparser.py", line 60, in parse_JSON
w = weather.weather_from_dictionary(d)
File "build/bdist.linux-i686/egg/pyowm/webapi25/weather.py", line 430, in weather_from_dictionary
if 'distance' in d['visibility']:
TypeError: argument of type 'int' is not iterable

WeatherCodeRegistry should handle the following extreme weather conditions:

• 781 tornado
• 900 tornado
• 901 tropical storm
• 902 hurricane
• 960 storm
• 961 violent storm
• 962 hurricane

and Forecaster class should provide methods for querying "storms" and "tornadoes":

    when_storm()
    will_have_storm()
    will_be_stormy_at()

    when_tornado()
    will_have_tornado()
    will_be_tornado_at()

    when_hurricane()
    will_have_hurricane()
    will_be_hurricane_at()

Reference: tox website

All the methods in the public interface of classes should be tested.

Mainly we're missing unit tests for "to_JSON" and "to_XML" methods in almost every class that is JSON- and XML-printable.

Class "Weather". The check on positiveness should be removed

The library should be installed and tested on MacOS platforms. Relevant issues regarding both phases should be reported here.

Add two functions to class OWM25:

• three_hours_forecast_at_id(self, id, [limit])
• daily_forecast_at_id(self, id, [limit])

whose purposes are self I descriptive.