mapproxy / mapproxy Goto Github PK
View Code? Open in Web Editor NEWMapProxy is a tile cache and WMS proxy
Home Page: http://mapproxy.org
License: Apache License 2.0
MapProxy is a tile cache and WMS proxy
Home Page: http://mapproxy.org
License: Apache License 2.0
Mapnik often doesn't load many fonts by default, so most mapnik-using applications have a config option for a directory to load fonts from.
For example, in mod_tile:
https://github.com/openstreetmap/mod_tile/blob/master/renderd.conf#L9
In nik2img.py:
http://code.google.com/p/mapnik-utils/source/browse/trunk/nik2img/nik2img.py#153
In Tilemill, a bunch of OS-specific places are checked automatically, which is an alternative approach
https://github.com/mapnik/node-mapnik/blob/master/lib/mapnik.js#L33
The code in nik2img that registers a directory of fonts recursively:
http://code.google.com/p/mapnik-utils/source/browse/trunk/nik2img/mapnik_utils/fonts.py
Something similar for mapproxy would be very useful.
The image on the left is the original tile image. The image on the right is the same tile returned by MapProxy using MapProxy's WMS service. Both tiles are 256x256 pixels, but it can be seen that the last row of the tile on the right is transparent.
The configuration file required to reproduce the bug is https://gist.github.com/1851b26a9ed0438feb9a, it should work out-of-the box (the original tiles are stored in S3).
The WMS request is:
http://localhost:8080/service/?FORMAT=image%2Fpng&LAYERS=VelolandRoutenNational&TRANSPARENT=TRUE&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&STYLES=&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A21781&BBOX=586400,196400,752800,362800&WIDTH=256&HEIGHT=256
The original tile is
http://schweizmobil-test.s3.amazonaws.com/tpayne-20121206/VelolandRoutenNational/14/1/1.png
Useful information:
Could this be a rounding error somewhere?
cc @fredj
I'm trying to proxy OSM source as WMS service with MapProxy.
I've set up the source for OSM tiles url, and the grid with origin: nw, but I get tiles in incorrect y-axis order.
What else I have to add to the configuration file to set up tiles in proper order?
My configuration:
services:
demo:
kml:
tms:
wmts:
wms:
srs: ['EPSG:4326', 'EPSG:900913']
image_formats: ['image/jpeg', 'image/png']
layers:
caches:
osm_orig_cache:
grids: [global_mercator_osm]
sources: [osm_orig_src]
sources:
osm_orig_src:
type: tile
url: http://c.tile.openstreetmap.org/%(tms_path)s.png
grids:
global_mercator_osm:
base: GLOBAL_MERCATOR
num_levels: 20
origin: nw
globals:
cache:
base_dir: './cache_data'
lock_dir: './cache_data/locks'
image:
resampling_method: nearest
When MapProxy caches the response of a WMS request it should use either max-age within the Cache-Control header (preferable), or the Expires header, to compute a timestamp to store in its cache. When a second request is made for the same tile the stored timestamp should be consulted first to determine if the request needs to be reissued to the service providing the WMS.
Note: This was unabashedly borrowed from a MapCache issue request, but is equally as valid (MapServer/mapcache#36).
LayerMerge currently only supports RGBA, not 8-bit paletted imagery.
mapserver
source does not allow to set environment variables for mapserv executable within yaml file. It is necessary for handy setting mapserver PATH
, PROJ_LIB
, GDAL_DATA
, MS_ERRORFILE
, etc.
According to the WMS GetFeatureInfo Request specification the QUERY_LAYERS parameter has a "comma-separated list of one or more layers to be queried". When firing such a request to Mapproxy only the first layer will be queried, the others are ignored.
Since many clients e.q. OpenLayers, will combine layers which are accessable through the same service url GetFeatureInfo through Mapproxy isn't working as excpected.
--Gerard
UPDATE: lxml package solved this issue. My mistake to not testing this first. May be the optional install in the doc could be changed to (highly) recommended.
maproxy config source:
layers:
- name: cadaster
title: cadaster layer
sources: [ wms_cadaster]
- name: cadaster1
title: cadaster layer
sources: [wms_cadaster1]
sources:
wms_cadaster:
type: wms
wms_opts:
version: 1.1.1
featureinfo: true
req:
url: http://localhost/cgi-bin/mapserv?map=map/wms_cadaster.map&
layers: cadaster
transparent: true
format: image/png
radius: 10
wms_cadaster1:
type: wms
wms_opts:
version: 1.1.1
featureinfo: true
req:
url: http://localhost/cgi-bin/mapserv?map=map/wms_cadaster.map&
layers: cadaster1
transparent: true
format: image/png
radius: 10
Openlayers code:
cadaster = new OpenLayers.Layer.WMS("cadaster", "/mapproxy/service?", {layers: "cadaster", format: 'image/png'});
cadaster1 = new OpenLayers.Layer.WMS("cadaster1", "/mapproxy/service?", {layers: "cadaster1", format: 'image/png'});
clickCadasterGood = new OpenLayers.Control.WMSGetFeatureInfo({
layers: [cadaster1],
queryVisible: true,
infoFormat: 'application/vnd.ogc.gml',
maxFeatures: 1,
autoActivate: true,
vendorParams: {
radius: 15
},
eventListeners: {
}
});
clickCadasterBad = new OpenLayers.Control.WMSGetFeatureInfo({
layers: [cadaster1,cadaster],
queryVisible: true,
infoFormat: 'application/vnd.ogc.gml',
maxFeatures: 1,
autoActivate: true,
vendorParams: {
radius: 15
},
eventListeners: {
}
});
when I make a request to one layer (clickCadasterGood), get the answer in the correct encoding: responseText: "<msGMLOutput xmlns:gml="http://www.opengis.net/gml"...."
when I make a request to multiple layers (clickCadasterBad), do not get the correct encoding: responseText: "<msGMLOutput xmlns:gml="http://www.opengis.net/gml"...."
seems at parsing lost tag: ""
Another approach to Issue 80 would be to allow the seeder to cleanup a specific items from the cache, or to remove all items newer than a specific date. We have a situation where images are updated throughout the day, but may already have been requested and cached. We need a mechanism to directly remove an image (and its related pyramid of images) from the cache, or to just flush everything that's newer than 2 hours, for example.
I have a PNG TMS cache in geodetic from zoom levels 0-9 and I want to convert it to mbtiles as PNG/JPEG mixed mode.
I setup a cache for the TMS folder and then setup a cache for the mbtiles using the TMS cache as the source. When I try to view this mbtiles cache using mapproxy-util serve-develop, I receive the error listed in the title.
Tested thoroughly and this only occurs when the grids match. Using GLOBAL_GEODETIC for each cache will produce this error, however giving it a custom grid (global_geodetic_sqrt2 for instance) will generate the cache as expected.
I will provide Gists if requested, thanks!
I have a cleanup task configured to delete all tiles from level 10 onwards for a cache that's located on another machine. globals.cache.base_dir is set to a UNC path and when I run mapproxy-seed I get this error http://twitpic.com/ab44s1/full.
All of the paths I've configured are UNC paths, so I'm not sure where it's getting confused with non-UNC paths.
MapProxy 1.4.0 on Windows Server 2008.
MapProxy tries to remove tiles from cache when it fails to load it, to allow re-retrieving of broken tiles.
https://github.com/mapproxy/mapproxy/blob/master/mapproxy/image/tile.py#L70
But this only works for filebased caches but not MBTiles, etc.
It's not possible to set the image format with mapproxy-util export
.
Add either a custom or auto-generated index to the mbtiles caches created by MapProxy. This would make the mbtiles created by MapProxy usable to libraries that need an index to perform searches.
I made the following cache:
caches:
kabul:
sources: []
grids: [GLOBAL_GEODETIC]
format: image/png
cache:
type: file
directory_layout: tms
directory: '/data/result'
Tiles are valid and appear when zoomed into the extent of the cached data. However, when changing the grid to the following:
grids:
kabul_grid:
srs: "EPSG:4326"
bbox: [69.0007369,34.3603824,69.4629450,34.6855883]
bbox_srs: "EPSG:4326"
No tiles are rendered. So, in order to specify an extent of the data AND have it render correctly, I must keep the kabul cache at GLOBAL_GEODETIC and then make an additional cache with kabul as the source, applying the kabul_grid to it. Is this working as intended?
EDIT: This was tested in both GLOBAL_GEODETIC and in GLOBAL_MERCATOR. Both produce this result.
All throughout the documentation the http.ssl_no_cert_check is referred to in its singular form, whereas the name of the property in reality is the plural http.ssl_no_cert_checks.
Affected pages:
http://mapproxy.org/docs/1.4.0/configuration.html
http://mapproxy.org/docs/1.4.0/sources.html
http://mapproxy.org/docs/1.4.0/configuration_examples.html
If I try to transform POINT(71 50)
from EPSG:4326 to EPSG:32644 in PostGIS or cs2cs utility I get the POINT(-216025.317866142 5586720.84493467)
. But when I set up grid in mapproxy config:
services:
demo:
tms:
sources:
my_mapnik_source:
type: mapnik
use_mapnik2: True
mapfile: /home/rykovd/mapnik-config.xml
grids:
utfgrid:
srs: EPSG:32644
tile_size: [256, 256]
origin: ll
res_factor: sqrt2
bbox: [71,50,87,54]
bbox_srs: EPSG:4326
caches:
ex1_cache:
cache:
type: file
directory_layout: tms
sources: [my_mapnik_source]
grids: [utfgrid]
layers:
- name: merged_layer
sources: [ex1_cache]
and try to query TileMap resource of TMS using http://10.22.0.9:8080/tms/1.0.0/merged_layer/EPSG32644
URL I get response which contains the following bbox information:
<BoundingBox minx="-216025.317866" miny="5540547.36994" maxx="929865.502423" maxy="6029925.90671"/>
As you can see minx value is the same as in PostGIS, but miny is different - 5540547.36994 vs 5586720.84493467.
Hello,
I use MapProxy with Autocad Map to view a tile source..
At initialisation the client zoom to bbox of wms and request quadkey((0,0,0)) who return an empty string, and fail to initialyse the wms layer.
My whorkaround is to mydify the quadkey function (mapproxy/client/tile.py) to return "0"
def quadkey(tile_coord):
"""
>>> quadkey((0, 0, 1))
'0'
>>> quadkey((1, 0, 1))
'1'
>>> quadkey((1, 2, 2))
'21'
"""
x, y, z = tile_coord
quadKey = ""
for i in range(z,0,-1):
digit = 0
mask = 1 << (i-1)
if (x & mask) != 0:
digit += 1
if (y & mask) != 0:
digit += 2
quadKey += str(digit)
# workaround for (0,0,0) tile
if quadkey == "":
quadkey = "0"
return quadKey
It is a bug or Is there any way to set min level to 1 in the configuration file ?
Installed MapProxy in Osgeo4w on Windows (tried both Windows7 and Windows Server 2008R2)
setting following environment variables:
OSGEO4W_ROOT=c:/osgeo4w
PYTHONHOME=%OSGEO4W_ROOT%\apps\Python27
PATH = %PATH%;%OSGEO4W_ROOT%\bin;%OSGEO4W_ROOT%\apps\Python27\Scripts;
If you create a wsgi app as written in docs, the demo apps are not working if called from osgeo4w apache. The maps are not showing because MapProxy cannot use PIL
But if wsgi is called via serve-develop all is working fine
To me it appears that something is wrong with loading PIL module if running in apache/mod_wsgi mode. Looking into error logs of apache immediately after starting there is a user warning that PIL is not available (raised in image.py). And when a map is served a real import error about PIL is shown.
[imported from https://bitbucket.org/olt/mapproxy/issue/35/transparency-and-opacity-fix]
This proposal for enhancement is relative to mapproxy 1.2.0.
Mapproxy does not work well when adding opacity to layers which also have transparent area's. The transparent area's become either semi-transparent black or semi-transparent-white.
The problem is described in: http://lists.osgeo.org/pipermail/mapproxy/2012-January/000913.html
and http://lists.osgeo.org/pipermail/mapproxy/2012-January/000914.html.
I've created a patch which solves the problem:
#!python
mapproxy/image/__init__.py:
layer_img = layer.as_image()
if (layer.image_opts and layer.image_opts.opacity is not None
and layer.image_opts.opacity < 1.0):
- layer_img = layer_img.convert(img.mode)
- img = Image.blend(img, layer_img, layer.image_opts.opacity)
+ if layer_img.mode == 'RGBA':
+ alpha = layer_img.split()[3]
+ alpha = ImageEnhance.Brightness(alpha).enhance(layer.image_opts.opacity)
+ layer_img.putalpha(alpha)
+ img.paste(layer_img, (0, 0), layer_img)
+ else:
+ layer_img = layer_img.convert(img.mode)
+ img = Image.blend(img, layer_img, layer.image_opts.opacity)
else:
if layer_img.mode == 'RGBA':
# paste w transparency mask from layer
mapproxy/platform/image.py:
Everywhere I've substituted:
Image, ImageColor, ImageDraw, ImageFont, ImagePalette, ImageChops
by:
Image, ImageColor, ImageDraw, ImageFont, ImagePalette, ImageChops, ImageEnhance
I propose to add this to mapproxy.
i've got a wms-server http://gis.susu.ac.ru/wms1? in EPSG:28411 but i can't see it in MapProxy viewer, e. g. in EPSG:4326 or EPSG:900913. What do you think about a probably reasons of that problem? Thank you.
It would be handy to display the number of tiles for each level/resolution in the mapproxy-util grids output.
When looking at layers, rendered from source
mpn:
type: mapnik
mapfile: /work/mapnik/mpn.xml
the background is black.
But in the xml file, I have
When rendering with just a python script or OGCServer, the background is transparent.
Is this a bug or just an incorrect setup?
We've just noticed what seems to be a a minor issue with multiple service initialisations happening when a bunch of requests for tiles for an uninitialised (or changed) service come in simultaneously (log attached).
I'm thinking ideally only the first call should initialise the service and the others should wait around until it's initialised. Not sure how hard that would be, though. Open to other ideas if you have any though, Oliver. Thanks.
For reference, we're running MapProxy as MultiApp via WSGI in Apache.
2013-04-03 14:40:51,005 - INFO - mapproxy.multiapp - initializing project app fwew with e:\mapproxy\keith\projects\fwew.yaml
2013-04-03 14:40:51,005 - INFO - mapproxy.config - reading: e:\mapproxy\keith\projects\fwew.yaml
2013-04-03 14:40:53,427 - INFO - mapproxy.config - reading: e:\mapproxy\keith\projects\base-config\base-mapproxy.yaml
2013-04-03 14:40:54,895 - INFO - mapproxy.multiapp - initializing project app fwew with e:\mapproxy\keith\projects\fwew.yaml
2013-04-03 14:40:54,911 - INFO - mapproxy.config - reading: e:\mapproxy\keith\projects\fwew.yaml
2013-04-03 14:40:56,848 - INFO - mapproxy.config - reading: e:\mapproxy\keith\projects\base-config\base-mapproxy.yaml
2013-04-03 14:40:58,677 - INFO - mapproxy.multiapp - initializing project app fwew with e:\mapproxy\keith\projects\fwew.yaml
2013-04-03 14:40:58,677 - INFO - mapproxy.config - reading: e:\mapproxy\keith\projects\fwew.yaml
2013-04-03 14:41:00,615 - INFO - mapproxy.config - reading: e:\mapproxy\keith\projects\base-config\base-mapproxy.yaml
2013-04-03 14:41:02,115 - INFO - mapproxy.multiapp - initializing project app fwew with e:\mapproxy\keith\projects\fwew.yaml
2013-04-03 14:41:02,115 - INFO - mapproxy.config - reading: e:\mapproxy\keith\projects\fwew.yaml
2013-04-03 14:41:04,459 - INFO - mapproxy.config - reading: e:\mapproxy\keith\projects\base-config\base-mapproxy.yaml
2013-04-03 14:41:05,802 - INFO - mapproxy.multiapp - initializing project app fwew with e:\mapproxy\keith\projects\fwew.yaml
2013-04-03 14:41:05,802 - INFO - mapproxy.config - reading: e:\mapproxy\keith\projects\fwew.yaml
2013-04-03 14:41:07,522 - INFO - mapproxy.config - reading: e:\mapproxy\keith\projects\base-config\base-mapproxy.yaml
I have added support within MapProxy for a new SQLite-based cached data format called geopackage. There is talk of initiating a pull request to the MapProxy repo in order to facilitate adoption and spread the word about MapProxy software.
Are you open to merging support for new cache data formats within MapProxy if I were to generate a pull request? This would hinge on approval from my stakeholders to give this code back, but so far everyone seems to be on-board to move forward.
Here are some links talking about geopackage:
http://www.opengeospatial.org/projects/groups/geopackageswg
http://www.weogeo.com/blog/The_GeoPackage.html
If you create a source and use the seed:only parameter MapProxy do not use the coverages in the services. In this case MapProxy creates a dummy source without coverages. So the bounding box in the capabilities covers the whole world.
If seed:only is set MapProxy only use the coverages for seeding.
Currently does not works. Throws exception on start:
Traceback (most recent call last):
File "./fcgi-socket", line 4, in <module>
application = make_wsgi_app('/etc/mapproxy/mapproxy.yaml')
File "/usr/local/lib/python2.6/dist-packages/mapproxy/wsgiapp.py", line 99, in make_wsgi_app
conf = load_configuration(mapproxy_conf=services_conf, ignore_warnings=ignore_config_warnings)
File "/usr/local/lib/python2.6/dist-packages/mapproxy/config/loader.py", line 1412, in load_configuration
conf_dict = load_configuration_file([os.path.basename(mapproxy_conf)], conf_base_dir)
File "/usr/local/lib/python2.6/dist-packages/mapproxy/config/loader.py", line 1430, in load_configuration_file
current_dict = load_yaml_file(conf_file)
File "/usr/local/lib/python2.6/dist-packages/mapproxy/util/yaml.py", line 29, in load_yaml_file
return load_yaml(f)
File "/usr/local/lib/python2.6/dist-packages/mapproxy/util/yaml.py", line 38, in load_yaml
return yaml.load(doc, Loader=yaml.CLoader)
File "/usr/local/lib/python2.6/dist-packages/yaml/__init__.py", line 73, in load
loader.dispose()
AttributeError: 'CLoader' object has no attribute 'dispose'
If our Mapproxy-running server has more IP adresses it would be nice to be able use them all to fight traffic shapers on remote WMS.
http://stackoverflow.com/questions/1150332/source-interface-with-python-and-urllib2
While trying to use the mbcheck
program (from the node-mbtiles
package), I've discovered that the metadata
table is empty in the MBTiles databases produced by MapProxy.
There is indeed a method called update_metadata()
in cache/mbtiles.py
but it doesn't seem to be called.
Furthermore, it doesn't support the bounds
property, which is recommended by the MBTiles spec.
Producing fully populated MBTiles databases would help when working with third-party tools.
A bit of an edge case here (apologies for the pun).
We've noticed an issue with requests from MapProxy that hit the boundary between eastern and western hemispheres. MapProxy is stopping the requests at the boundary, but in cases where the request is in the eastern hemisphere the upper right corner of the BBOX is being set to -180, not 180 (the eastern hemisphere's side of the boundary). This is causing some unfortunate display issues with our backend (MapServer).
For example:
Reprojected to EPSG:900913 (note the low quality of the image)
http://srss-ows-1.landgate.wa.gov.au/wxs/firewatch.php?styles=&format=image%2Fpng&height=1184&bbox=14636773.6723,-5400734.67052,20037508.3428,391357.58482&transparent=True&layers=layer37_MODIS_Global_Mosaic.ecw&service=WMS&width=1104&request=GetMap&srs=EPSG%3A900913&version=1.1.1
The same request in native EPSG:4326 fails because the BBOX is invalid
http://srss-ows-1.landgate.wa.gov.au/wxs/firewatch.php?styles=&format=image%2Fpng&height=1184&bbox=131.484375,-43.580391,-180,3.513421&transparent=True&layers=layer37_MODIS_Global_Mosaic.ecw&service=WMS&width=1104&request=GetMap&srs=EPSG:4326&version=1.1.1
But if we replace the -180 with 180 we get the desired outcome:
http://srss-ows-1.landgate.wa.gov.au/wxs/firewatch.php?styles=&format=image%2Fpng&height=1184&bbox=131.484375,-43.580391,180,3.513421&transparent=True&layers=layer37_MODIS_Global_Mosaic.ecw&service=WMS&width=1104&request=GetMap&srs=EPSG:4326&version=1.1.1
This is possibly something MapServer should deal with, but we thought we'd ask for a fix from MapProxy first since it seems like such an easy fix from your POV.
We've worked around it for the time being by reducing our meta_size to 3,3 from 4,4 so only New Zealand is being pixelated, but a longer-term fix would be great.
We have a .yaml configuration file with ~600 layers in it and have noticed that each time it changes (every 5 minutes in our case) that httpd.exe adds ~18MB of memory.
Steps to reproduce:
This is what the log looked like after 4 iterations with httpd.exe going from ~14MB to ~94MB
2013-05-03 15:01:39,039 - INFO - mapproxy.multiapp - initializing project app firewatch with e:\mapproxy\keith\projects\firewatch.yaml
2013-05-03 15:01:39,039 - INFO - mapproxy.config - reading: e:\mapproxy\keith\projects\firewatch.yaml
2013-05-03 15:01:40,631 - INFO - mapproxy.config - reading: e:\mapproxy\keith\projects\base-config\base-mapproxy.yaml
2013-05-03 15:01:40,803 - INFO - mapproxy.system - using pyproj for coordinate transformation
2013-05-03 15:01:55,882 - INFO - mapproxy.multiapp - initializing project app firewatch with e:\mapproxy\keith\projects\firewatch.yaml
2013-05-03 15:01:55,882 - INFO - mapproxy.config - reading: e:\mapproxy\keith\projects\firewatch.yaml
2013-05-03 15:01:57,648 - INFO - mapproxy.config - reading: e:\mapproxy\keith\projects\base-config\base-mapproxy.yaml
2013-05-03 15:02:08,414 - INFO - mapproxy.multiapp - initializing project app firewatch with e:\mapproxy\keith\projects\firewatch.yaml
2013-05-03 15:02:08,414 - INFO - mapproxy.config - reading: e:\mapproxy\keith\projects\firewatch.yaml
2013-05-03 15:02:10,071 - INFO - mapproxy.config - reading: e:\mapproxy\keith\projects\base-config\base-mapproxy.yaml
2013-05-03 15:02:58,321 - INFO - mapproxy.multiapp - initializing project app firewatch with e:\mapproxy\keith\projects\firewatch.yaml
2013-05-03 15:02:58,321 - INFO - mapproxy.config - reading: e:\mapproxy\keith\projects\firewatch.yaml
2013-05-03 15:03:00,023 - INFO - mapproxy.config - reading: e:\mapproxy\keith\projects\base-config\base-mapproxy.yaml
Tested with MapProxy 1.5.0 (with bac0464 integrated) and 1.6.0a with the same result.
We're running MapProxy as MultiApp via WSGI in Apache on Windows Server 2008.
Have confirmed that it happens when using mapproxy-util serve-multiapp-develop, too.
There is a report that MapProxy does processes cascaded GeoTIFF requests from MapServer sources and thus removing the georeference information.
http://lists.osgeo.org/pipermail/mapproxy/2012-August/001173.html
Hey there,
I have configured mapnik rendering like this:
layers: - name: overlay_mapnik title: overlay_mapnik sources: [overlay_mapnik] caches: overlay_mapnik: sources: [overlay_mapnik] sources: overlay_mapnik: type: mapnik mapfile: osm_overlay.xml use_mapnik2: true transparent: true
If I access the MapProxy WMS server through the demo-page, I can see the tiles being rendered correctly and the server returns HTTP 200 with the correct images.
However, when I try to access an (uncached) area via TMS (or by the tiles/ URL for OSM-like zoom levels), the tiles are not being rendered. Previousely cached tiles that were created by the WMS access will be returned, but no rendering will be done.
The request hangs for a while and apache will finally return HTTP 500.
Version 1.5.0, SuSE 12.2, running on mod_wsgi.
EDIT: This only seems to happen when running through mod_wsgi.
Accessing via localhost:8080 seems to work correctly and triggers the rendering.
regards,
Adrian
Hi,
I've been looking at implementing an AWS S3 Cache backend based on the FileCache, I now have a working prototype which uses boto and I'd appreciate any thoughts and feedback. The code can be found here in the s3 branch: https://github.com/walkermatt/mapproxy/tree/s3.
If you'd like to set up an S3 cache then you'll need the boto package and a cache of type 's3' and a valid bucket name. For instance to configure the demo osm_cache to use an S3 cache:
caches:
osm_cache:
grids: [GLOBAL_MERCATOR, global_geodetic_sqrt2]
sources: [osm_wms]
cache:
type: s3
bucket: uniquebucketid
The bucket ID must be unique across all of S3 as described here so choose something unique. At present new buckets are created in the EU zone, I'll no doubt make that configurable soon.
Even though you can store any arbitrary data in S3 against a key I've chosen to effectively store files with the keys being the effective file system path. This may allow an existing cache to be uploaded to S3 using something like s3fs or even offline import then used / maintained by the S3 cache.
BTW The original motivation for looking at an S3 cache is to set up a redundant shared cache used by several MapProxy instances all running the same configuration behind a load balancer. At present I'm using file based cache stored on EBS volumes which works well but results in duplication of the cache data. I've experimented with using s3fs with a file based cache but I've had several stability issues where the s3fs mount disappeared periodically.
Cheers,
Matt.
The export tool tries to use existing tiles (without opening them) when the cache and export girds are compatible. This fails when the image formats are different:
Exception in thread Thread-1:
Traceback (most recent call last):
File "D:\Python27\Lib\threading.py", line 551, in __bootstrap_inner
self.run()
File "d:\mapproxy_venv\lib\site-packages\mapproxy\seed\seeder.py", line 92, in run
self.work_loop()
File "d:\mapproxy_venv\lib\site-packages\mapproxy\seed\seeder.py", line 104, in work_loop
exceptions=(SourceError, IOError), ignore_exceptions=(LockTimeout, ))
File "d:\mapproxy_venv\lib\site-packages\mapproxy\seed\util.py", line 216, in exp_backoff
result = func(*args, **kw)
File "d:\mapproxy_venv\lib\site-packages\mapproxy\cache\tile.py", line 126, in load_tile_coords
created_tiles = creator.create_tiles(uncached_tiles)
File "d:\mapproxy_venv\lib\site-packages\mapproxy\cache\tile.py", line 213, in create_tiles
created_tiles = self._create_single_tiles(tiles)
File "d:\mapproxy_venv\lib\site-packages\mapproxy\cache\tile.py", line 237, in _create_single_tiles
created_tiles.extend(self._create_single_tile(tile))
File "d:\mapproxy_venv\lib\site-packages\mapproxy\cache\tile.py", line 253, in _create_single_tile
source = self._query_sources(query)
File "d:\mapproxy_venv\lib\site-packages\mapproxy\cache\tile.py", line 274, in _query_sources
return self.sources[0].get_map(query)
File "d:\mapproxy_venv\lib\site-packages\mapproxy\source\tile.py", line 95, in get_map
return CacheMapLayer.get_map(self, query)
File "d:\mapproxy_venv\lib\site-packages\mapproxy\layer.py", line 390, in get_map
self._check_tiled(query)
File "d:\mapproxy_venv\lib\site-packages\mapproxy\layer.py", line 409, in _check_tiled
raise MapError("invalid tile format, use %s" % self.tile_manager.format)
MapError: invalid tile format, use jpeg
When I issue http://myserver/mapproxy/demo
The apache2 error log shows the error from the title twice:
[Tue Jan 15 22:11:49 2013] [error] No handlers could be found for logger "mapproxy.service.wmts"
[Tue Jan 15 22:11:49 2013] [error] No handlers could be found for logger "mapproxy.service.wmts"
There is an easy fix for this which I have tested and will send in through a pull request
Often imagery source is very slow and cannot handle many requests at same time. So when someone wants to warm up it's own cache, he cannot control download speed and imagery source can crash, especially this happens with ESRI products. That's why mapproxy-seed needs some throttling mechanism. It could be based on request rate per second or downloaded data size per second, or maybe time pauses between requests could help.
Throttling would be very useful option for mapproxy-seed.
Has anyone tried to implement utfgrid output?
Where to start?
When doing a featureinfo request the source service returns a charset encoding in the http-headers like this:
Content-Type:application/vnd.ogc.gml;charset=UTF-8
mapproxy throws an NotImplementedError() if a xsl transformation on the result should be done.
Suggested fix in featureinfo.py:
change:
def create_featureinfo_doc(content, info_format):
if info_format in ('text/xml', 'application/vnd.ogc.gml'):
return XMLFeatureInfoDoc(content)
if info_format == 'text/html':
return HTMLFeatureInfoDoc(content)
to:
def create_featureinfo_doc(content, info_format):
if info_format.split(';')[0] in ('text/xml', 'application/vnd.ogc.gml'):
return XMLFeatureInfoDoc(content)
if info_format.split(';')[0] == 'text/html':
return HTMLFeatureInfoDoc(content)
[Imported from https://bitbucket.org/olt/mapproxy/issue/43/mapproxy-crashes-when-built-against-os-x]
I've got an OS X 10.7 system with Python 2.7.3 installed as a Framework (that's default for OS X). I've installed PIL, PyYAML and MapProxy. Serve-develop starts with default configuration (base-config). Server crashes on the first WMS GetMap or TMS request with no error message - it just returns to the command line.
The last message is:
mapproxy.source.request - INFO - GET http://osm.omniscale.net/proxy/service?layers=osm&width=512&version=1.1.1&bbox=-20037508.3428,-20037508.3428,20037508.3428,20037508.3428&service=WMS&format=image%2Fpng&styles=&srs=EPSG%3A900913&request=GetMap&height=512 200 23.3 166
MapProxy doesn't crash when built against Homebrew installed python. This could be ok, but I'm planning to use mapnik source with mapproxy. Mapnik requires boost library, and boost library builds against OS default python only. So I don't get mapnik bindings in my homebrew python. So the only solution left is to recompile boost against non-default python on OS X, but this isn't a trivial task at all.
Maybe something can be changed in MapProxy code, so it won't crash when getting wms/tms requests on OS X?
ResourceURL template in in WMTSCapabilities does not conform to WMTS standard. MapProxy uses double braces '{{' and '}}' to mark variable names whereas the standard uses single braces '{' and '}'. As far as I can see the standard does not explicitly forbid double braces, but it means that for instance QGIS is not able to consume WMTS from MapProxy.
MapProxy example:
Example from the spec:
We are using MapProxy 1.4.0
When MapProxy receives a GetFeatureInfo request for a WMS source that contains the FEATURE_COUNT property it doesn't get passed on to the source service. As the default for FEATURE_COUNT is usually 1 this is resulting in single features being returned.
MapProxy 1.4.0.
Tile access (TMS/KML/WMTS) is not possible when using the watermark feature. I guess the image_opts get lost when the watermarks are added.
services:
demo:
tms:
layers:
- name: osm
title: OpenStreetMap
sources: [osm_cache]
caches:
osm_cache:
grids: [global_mercator_osm]
sources: [osm_tms]
disable_storage: true
watermark:
text: '@ OpenStreetMap contributors'
sources:
osm_tms:
type: tile
grid: global_mercator_osm
url: http://c.tile.openstreetmap.org/%(tms_path)s.png
grids:
global_mercator_osm:
base: GLOBAL_MERCATOR
num_levels: 18
origin: nw
The problem is mapnik-specific and is well described here:
https://groups.google.com/forum/?hl=ru&fromgroups=#!topic/mapnik/NVpQHfICdas
and here:
mapnik/mapnik#1550
This is MapProxy output:
http://db.tt/m2K8mBR3
So the problem is that Map element in mapnik config file is configured with srs="+init:epsg:3857"
which is then reprojected somewhere in MapProxy to EPSG:3857 (while it shouldn't). If raster reprojection occurs, then it triggers mapnik bug with corrupted raster edges.
Fixed that in mapnik OGCServer by changing the long proj string to just +init:epsg:3857, so it doesn't reproject. Hoped that helps with MapProxy too, but unfortunately it didn't.
Hi
Mapproxy doesn't offer acces to root resource with TMS client.
Some softwares particularly needs Services information :
< Services > < TileMapService title="Example Static Tile Map Service" version="1.0.0" href="http://www.osgeo.org/services/tilemapservice.xml" / >< /Services >
as described here http://wiki.osgeo.org/wiki/Tile_Map_Service_Specification
grids:
swiss_grid_new:
srs: 'EPSG:21781'
bbox: [420000, 30000, 900000, 360000]
bbox_srs: 'EPSG:21781'
res: [250, 100, 50, 20, 10, 5, 2.5, 2, 1.5, 1, 0.5 ,0.25, 0.125, 0.0625]
origin: 'nw'
The WMTS GetCapabilities document that MapProxy generates includes
TileMatrix blocks that look like this:
<TileMatrix>
<ows:Identifier>00</ows:Identifier>
<ScaleDenominator>892857.142857</ScaleDenominator>
<TopLeftCorner>420000.0 40000.0</TopLeftCorner>
<TileWidth>256</TileWidth>
<TileHeight>256</TileHeight>
<MatrixWidth>8</MatrixWidth>
<MatrixHeight>6</MatrixHeight>
</TileMatrix>
Initialising a service with a large number of layers takes a long time. Follow on from this post on the mailing list (with data and timings included): http://osgeo-org.1560.n6.nabble.com/MapProxy-GetCapabilities-request-very-slow-td5039049.html
Gists of a config file with 200 layers that was taking 60s to init via 'mapproxy-utils serve-develop':
https://gist.github.com/keithamoss/d06b55e44ae98cbc083a
https://gist.github.com/keithamoss/48d3080a093f619fb94b
I have a working mapproxy setup that has a RGBA PNG cache which works fine when accessed through the web.
When I try to run mapproxy-seed
for the cache I get the error
An error occured. Retry in 2 seconds: IOError('decoder zip not available',)
An error occured. Retry in 2 seconds: IOError('decoder zip not available',)
An error occured. Retry in 4 seconds: IOError('decoder zip not available',)
...
I have mapproxy installed in a virtualenv.
We use 2 mapproxy Nodes behind a Loadbalancer without sticky sessions.
mappproxy cache directory is storage to the same NFS share, is locatet at a Netapp device.
The NFS is mounted under this options: rw,noatime,nodiratime,noacl,noac
When mapproxy write a tile, lockfiles are storage to the NFS share directory under /cache/mapproxy/locks, that is configured on both nodes.
The format of file is eb543c83f5f73bc5abf13e6573dba30d.lck0
This format is always the same....
We to suspect that on node delete the lck file from the other node...
Here ist the issue from the apache error logfile:
[Thu Feb 28 11:19:08 2013] [error] [client _] Traceback (most recent call last):, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] File "/usr/lib/pymodules/python2.6/mapproxy/wsgiapp.py", line 158, in call, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] resp = self.handlers[handler_name].handle(req), referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] File "/usr/lib/pymodules/python2.6/mapproxy/service/ows.py", line 38, in handle, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] return self.services[service].handle(req), referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] File "/usr/lib/pymodules/python2.6/mapproxy/service/base.py", line 30, in handle, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] return handler(parsed_req), referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] File "/usr/lib/pymodules/python2.6/mapproxy/service/wms.py", line 111, in map, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] renderer.render(merger), referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] File "/usr/lib/pymodules/python2.6/mapproxy/service/wms.py", line 371, in render, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] layer_merger), referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] File "/usr/lib/pymodules/python2.6/mapproxy/util/async.py", line 204, in _single_call, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] result = func(_args), referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client _] File "/usr/lib/pymodules/python2.6/mapproxy/service/wms.py", line 412, in _render_layer, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] layer_img = layer.get_map(self.query), referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] File "/usr/lib/pymodules/python2.6/mapproxy/layer.py", line 289, in get_map, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] result = self._image(query), referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] File "/usr/lib/pymodules/python2.6/mapproxy/layer.py", line 322, in _image, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] tile_collection = self.tile_manager.load_tile_coords(affected_tile_coords), referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] File "/usr/lib/pymodules/python2.6/mapproxy/cache/tile.py", line 126, in load_tile_coords, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] created_tiles = creator.create_tiles(uncached_tiles), referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] File "/usr/lib/pymodules/python2.6/mapproxy/cache/tile.py", line 227, in create_tiles, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] created_tiles = self._create_meta_tiles(meta_tiles), referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] File "/usr/lib/pymodules/python2.6/mapproxy/cache/tile.py", line 293, in _create_meta_tiles, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] return self._create_threaded(self._create_meta_tile, meta_tiles), referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] File "/usr/lib/pymodules/python2.6/mapproxy/cache/tile.py", line 243, in _create_threaded, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] for new_tiles in async_pool.imap(create_func, tiles):, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] File "/usr/lib/pymodules/python2.6/mapproxy/util/async.py", line 54, in _result_iter, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] for result in results:, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] File "/usr/lib/pymodules/python2.6/mapproxy/util/async.py", line 191, in map_each, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] for value in self._get_results(next_result, results, raise_exceptions):, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] File "/usr/lib/pymodules/python2.6/mapproxy/util/async.py", line 235, in _get_results, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] for i, value in self._fetch_results(raise_exceptions):, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] File "/usr/lib/pymodules/python2.6/mapproxy/util/async.py", line 144, in run, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] result = func(_args), referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client _**] File "/usr/lib/pymodules/python2.6/mapproxy/cache/tile.py", line 304, in _create_meta_tile, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] with self.tile_mgr.lock(main_tile):, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] File "/usr/lib/pymodules/python2.6/mapproxy/platform/cpython/lock.py", line 42, in enter, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] self.lock(), referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client ] File "/usr/lib/pymodules/python2.6/mapproxy/platform/cpython/lock.py", line 65, in lock, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] self._lock = self.try_lock(), referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] File "/usr/lib/pymodules/python2.6/mapproxy/platform/cpython/lock.py", line 56, in _try_lock, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] return LockFile(self.lock_file), referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] File "/usr/lib/pymodules/python2.6/mapproxy/util/ext/lockfile.py", line 124, in init, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] pid = fp.read().strip()[:20], referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
[Thu Feb 28 11:19:08 2013] [error] [client *] IOError: [Errno 116] Stale NFS file handle, referer: http://isk.geobasis-bb.de/BrandenburgViewer/basiskarte.html
is mapproxy Loadbalancer ready?
There are a LB howto for mapproxy?
many thx
Karsten
I created two caches:
Conf is here: https://gist.github.com/stevendlander/5782462
I created a seed for cache 2, specifying the Geodetic grid and a custom coverage for my data in EPSG:4326
Seed conf: https://gist.github.com/stevendlander/5782483
I then kick off a seed task for this job: mapproxy-seed -f conf.yaml -s seed.yaml
I show the following stack trace: https://gist.github.com/stevendlander/5782514
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.