lgervasoni / urbansprawl Goto Github PK

Hello, I am trying to run the sprawl-overview example code but I am getting an error when I attempt to retreive the OSM data. Can you help? This is the error message:

JSONDecodeError Traceback (most recent call last)
~\AppData\Local\Continuum\anaconda3\lib\site-packages\osmnx\core.py in overpass_request(data, pause_duration, timeout, error_pause_duration)
337 try:
--> 338 response_json = response.json()
339 if 'remark' in response_json:

~\AppData\Local\Continuum\anaconda3\lib\site-packages\requests\models.py in json(self, **kwargs)
891 pass
--> 892 return complexjson.loads(self.text, **kwargs)
893

~\AppData\Local\Continuum\anaconda3\lib\json_init_.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
353 parse_constant is None and object_pairs_hook is None and not kw):
--> 354 return _default_decoder.decode(s)
355 if cls is None:

~\AppData\Local\Continuum\anaconda3\lib\json\decoder.py in decode(self, s, _w)
338 """
--> 339 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
340 end = _w(s, end).end()

~\AppData\Local\Continuum\anaconda3\lib\json\decoder.py in raw_decode(self, s, idx)
356 except StopIteration as err:
--> 357 raise JSONDecodeError("Expecting value", s, err.value) from None
358 return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Exception Traceback (most recent call last)
in ()
1 # Retrieve OSM data
2 region_args = {"north":north, "south":south, "west":west, "east":east}
----> 3 df_osm_built, df_osm_building_parts, df_osm_pois = get_processed_osm_data(city_ref, region_args)

~\src\core.py in get_processed_osm_data(city_ref_file, region_args, kwargs)
179 ##########################
180 # Query and update bounding box / polygon
--> 181 df_osm_built, polygon, north, south, east, west = create_buildings_gdf_from_input(date=date_query, polygon=polygon, place=place, which_result=which_result, point=point, address=address, distance=distance, north=north, south=south, east=east, west=west)
182 df_osm_built["osm_id"] = df_osm_built.index
183 df_osm_built.reset_index(drop=True, inplace=True)

~\src\osm\osm_overpass.py in create_buildings_gdf_from_input(date, polygon, place, which_result, point, address, distance, north, south, east, west)
96 p4 = (east,south)
97 polygon = Polygon( [p1,p2,p3,p4] )
---> 98 df_osm_built = buildings_from_polygon(date, polygon)
99 else:
100 log("Error: Must provide at least one input")

~\src\osm\osm_overpass.py in buildings_from_polygon(date, polygon, retain_invalid)
337 """
338
--> 339 return create_buildings_gdf(date=date, polygon=polygon, retain_invalid=retain_invalid)
340
341

~\src\osm\osm_overpass.py in create_buildings_gdf(date, polygon, north, south, east, west, retain_invalid)
235 """
236
--> 237 responses = osm_bldg_download(date, polygon, north, south, east, west)
238
239 vertices = {}

~\src\osm\osm_overpass.py in osm_bldg_download(date, polygon, north, south, east, west, timeout, memory, max_query_area_size)
201 '(poly:"{polygon}")["building"];(._;>;));out;')
202 query_str = query_template.format(polygon=polygon_coord_str, timeout=timeout, maxsize=maxsize)
--> 203 response_json = overpass_request(data={'data':query_str}, timeout=timeout)
204 response_jsons.append(response_json)
205 msg = ('Got all building footprints data within polygon from API in '

~\AppData\Local\Continuum\anaconda3\lib\site-packages\osmnx\core.py in overpass_request(data, pause_duration, timeout, error_pause_duration)
358 else:
359 log('Server at {} returned status code {} and no JSON data'.format(domain, response.status_code), level=lg.ERROR)
--> 360 raise Exception('Server returned no JSON data.\n{} {}\n{}'.format(response, response.reason, response.text))
361
362 return response_json

Exception: Server returned no JSON data.
<Response [400]> Bad Request

The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.

Error: line 1: parse error: ';' expected - ')' found.

Error: line 1: parse error: Unexpected end of input.

Error: line 1: static error: Element "print" cannot be subelement of element "union".

Hi,
There seems to be an issue with osm_cities notebook, as the process fails for some cities. As an example, for Paris, I get a

ReadTimeout: HTTPConnectionPool(host='overpass-api.de', port=80): Read timed out. (read timeout=180)

Plus, some other cities are really long to be addressed. How much time is the processing supposed to be?

Hello,

Thank your for sharing UrbanSprawl of Python.

Now I want to use it but it does not work due to update of dependency like scikit-learn.

I need your advice for solving the issue.

Sincerely,

In order to better understand the framework, the code needs to be reorganized.
Modules need to be correctly separated.

Running the first cell of the notebook raises a missing dependency error, as keras is not indicated in the project dependencies.

The OSM data gathering process is long and its reliability is hard to control. We could cut it into smaller steps, design a task dependency graph and verify each step realization sequentially.

In sprawl/landusemix.py the Entropy Index calculation results in NaN values if any of the inputs is zero, whereas the degree of mixed-use should be zero.

	if (x <= 0 or y <= 0): return np.nan

The employed logging can be enhanced

1. geopandas data farmes have no assigned names, where they still appear in the log.
2. The time elapsed can be shown for each step

The last exemple within osm_cities.ipynb failed as follows:

---------------------------------------------------------------
KeyError                      Traceback (most recent call last)
<ipython-input-4-2c1734dd0411> in <module>()
      8 kwargs={'retrieve_graph': True, 'default_height': 3, 'meters_per_level': 3, 'associate_landuses_m2': True, 'minimum_m2_building_area': 9, 'date': date}
      9 region_args = {"place":"Grenoble, France"}
---> 10 df_osm_built, df_osm_building_parts, df_osm_pois = get_processed_osm_data(city_ref_file = 'Grenoble_old', region_args = region_args, kwargs = kwargs)
     11 
     12 # Plot

~/src/urbansprawl/src/core.py in get_processed_osm_data(city_ref_file, region_args, kwargs)
    306                 default_height = kwargs["default_height"]
    307                 meters_per_level = kwargs["meters_per_level"]
--> 308                 mixed_building_first_floor_activity = kwargs["mixed_building_first_floor_activity"]
    309                 compute_landuses_m2(df_osm_built, df_osm_building_parts, df_osm_pois, default_height=default_height, meters_per_level=meters_per_level, mixed_building_first_floor_activity=mixed_building_first_floor_activity)
    310 

KeyError: 'mixed_building_first_floor_activity'

It seems that we have to add 'mixed_building_first_floor_activity': True in the kwargs dict definition.

When buildings are retrieved for a region of interest, and there exists no height tags for any of them, the associate_level function in osm/osm_surface.py fails.
It expects a dict to process the available height tags, but a NaN is sent in some cases (due to missing columns).
The method should consider these cases, and set a default height to the buildings.
The problem is reproducible chossing the following region of interest: "El Cairo, Egypt".

One should extend the installation instructions for the cases of Miniconda and pip

The compute_grid_accessibility method fails during execution. After reading the source code, it seems that the problem comes from line 96:

max_processes = int( ''.join(numbers) )

The value of numbers is probably not int-compatible (I suspect a null value), as a the log highlights a missing dependency (psutil).

Hi,

I'm working on a setup.py file. It may be useful for:

1. install the urbansprawl project/package
2. list all dependencies
3. distribute the package on Pypi

I also suggest to rename the folder src to urbansprawl. When you distribute a Python package, you have to find a name to this package. It shouldn't be src. This implies to update all imports where there is src.

What do you think @lgervasoni ?

Thanks,
Damien

In order to better understand the framework, the code needs to be reorganized. Modules need to be correctly separated.
As well, a main executable should be added, something like the following:
python urbansprawl [method] [arguments]
where the method can be either the OSM processed data, the spatial sprawl indices, or the population downscaling.

The composed classification calculation is unnecessary. During the pre-processing step, buildings classification are already given according to their composed classification, considering both containing Points of interest and building parts.

Code in population/urban_features.py

# Calculate the composed classification
df_osm_built.containing_poi = df_osm_built.containing_poi.apply(lambda x: x if isinstance(x,list) else [])
df_osm_built.activity_category = df_osm_built.activity_category.apply(lambda x: x if isinstance(x,list) else [])

df_osm_built['composed_classification'] = df_osm_built.apply(lambda x: get_composed_classification( x, df_osm_pois.loc[x.containing_poi] ).classification, axis=1 )
df_osm_built.loc[ df_osm_built.containing_poi.apply(lambda x: len(x)==0 ), "containing_poi" ] = np.nan
df_osm_built.loc[ df_osm_built.activity_category.apply(lambda x: len(x)==0 ), "activity_category" ] = np.nan

A main executable can be added, in order to be able to do something like the following:
python urbansprawl [method] [arguments]
where the method can be either the 1) OSM processed data, 2) the spatial sprawl indices, or 3) the population downscaling.

As such, the following file can be included in /urbansprawl/__main__.py:

import sys

# Usage: "python urbansprawl [method] [arguments]"

def main():
	print(sys.argv)

	if (sys.argv[1] == "osm_data"):
		print("Retrieve OSM data")
		# TODO

	if (sys.argv[1] == "indices"):
		print("Retrieve sprawl indices")
		# TODO

	if (sys.argv[1] == "population_downscaling"):
		print("Perform population downscaling")
		# TODO

if __name__== "__main__":
	main()

In the second code cell, I got:

FileNotFoundError: [Errno 2] No such file or directory: 'data/training'

We need a data/training subdir in get_Y_X_features_population_data fonction, however it seems that its creation is not managed at all in the source code.