aliflux / maptilesdownloader Goto Github PK

nice work btw 😉

Strange issue.... I exported some data, and imported into geoserver (as mbtiles) and the output is not correctly aligned.
This could be also a geoserver problem, but I suspect rather that something is wrong with the sqlite format.

Any idea about this?
I was using the docker to export the mbtiles.

Hi,
How we can add apple map service to this project? I want to download apple map tiles for a location.

In the method "gettilerect"（main.js）, C1 represents northwest and C2 represents southeast. C1 and C2 are stored in mapboxgl.lnglatbounds, but the two parameters in lnglatbounds should be southwest and northeast. Is it wrong to use them? Still didn't explain clearly

The OSMF tile usage policy states

In particular, downloading significant areas of tiles at zoom levels 17 and higher for offline or later usage is forbidden without prior consultation with a System Administrator.

I didn't identify MapTilesDownloader as a heavy user in openstreetmap/operations#371, but it's still doing something not allowed.

Thanks for that really nice peace of work!

I use it to fill up a simple nginx-cache, that buffers tiles from OpenTopoMap.
(OpenTopoMap is slooooow, so the cache makey my life easier).
For that I modified main.js and added the caches URL as tile-server.

While downloading with MapTilesDownloader many errors occur. Most of them are "Error while relaying tile".
What exactly this error means?

As mentioned in my last issue I just want to fill my cache and don't need the downloaded tiles.
Do you see an easy way for me to NOT create the output on filesystem?

Whenever the tool tries to download the tiles I'm getting error "error while relaying tile". In the python console the error output looks like so:

----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 59134)
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\socketserver.py", line 650, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\ProgramData\Anaconda3\lib\socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\ProgramData\Anaconda3\lib\socketserver.py", line 720, in __init__
    self.handle()
  File "C:\ProgramData\Anaconda3\lib\http\server.py", line 426, in handle
    self.handle_one_request()
  File "C:\ProgramData\Anaconda3\lib\http\server.py", line 414, in handle_one_request
    method()
  File "server.py", line 81, in do_POST
    postvars = parse_multipart(self.rfile, pdict)
  File "C:\ProgramData\Anaconda3\lib\cgi.py", line 220, in parse_multipart
    headers['Content-Length'] = pdict['CONTENT-LENGTH']
KeyError: 'CONTENT-LENGTH'
----------------------------------------

An error occurred when I opened the “http://localhost:8080/”
FileNotFoundError: [Errno 2] No such file or directory: './UI/favicon.ico'.
I searched the entire engineering project but couldn't find it.
I pray you can give me a solution.

Actually, I'm a newby working with Map using Cesium. I found that Cesium needs a map provider as WMS and I've tried MapTileDownloader and it works great except it cannot output images as Web Map Tiles Service (WMTS or WMS) which is the format that Cesium supports.
I cannot find any software that can export map tiles in WMS as well.
So, how can we add an option to the downloader the tiles for WMS?
Thanks.

Traceback (most recent call last):
File "C:\Users\aaves\Desktop\maptiles\src\server.py", line 236, in
run()
File "C:\Users\aaves\Desktop\maptiles\src\server.py", line 228, in run
httpd = serverThreadedHandler(server_address, serverHandler)
File "C:\Users\aaves\AppData\Local\Programs\Python\Python39\lib\socketserver.py", line 452, in init
self.server_bind()
File "C:\Users\aaves\AppData\Local\Programs\Python\Python39\lib\http\server.py", line 138, in server_bind
socketserver.TCPServer.server_bind(self)
File "C:\Users\aaves\AppData\Local\Programs\Python\Python39\lib\socketserver.py", line 466, in server_bind
self.socket.bind(self.server_address)
OSError: [WinError 10013] An attempt was made to access a socket in a way forbidden by its access permissions

Hi Great program here.... thanks

When i edit to other mapbox style in Main.js - the map stop loading and blank mapview - do i do something wrong or ??

Thanks
Frank

I'm using my own mapbox account to download the maps. Google Map works fine, however, I would need OpenStreetMap. But regardless of the zoom level, I encountered this error "403 Error downloading tile".

...
127.0.0.1 - - [29/Apr/2020 00:15:04] "POST /download-tile HTTP/1.1" 200 -
HIT: https://a.tile.openstreetmap.org/{z}/{x}/{y}.png
RETURN: 429
...

You need find a way for add header at request 'urllib.request.urlretrieve', like this: {"User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0"} ?
Your code should seem like a Webbrowser for OSM servers, not a script.

I've cloned the repo and tried to run:
python3 ./src/server.py

but got a
ModuleNotFoundError: No module named 'PIL'

error.

Do I need to install packages like in the javascript world?

How can i get Jpeg output

I'd like to be able to combine the output into one high res image, or better yet, have that as an output option. Are you willing to implement this, or have any recommendations on how I would go about doing this?

How to change the map source and add your own map source published with mapbox

Hi there, I want to archive all the tiles on an encient tile provider (since I am afraid that they will shut down their tile server and I have an app that I cannot change that is still using them)

the tile server is here: http://tiles.cdnst.net/base/

maps are organized in /z/x/y.png (I think)
and are using modestmaps-as3 (Modest Maps)
examples: http://tiles.cdnst.net/base/6/34/23.png

http://tiles.cdnst.net/base/6/34/24.png
http://tiles.cdnst.net/base/6/34/25.png
http://tiles.cdnst.net/base/6/34/26.png
http://tiles.cdnst.net/base/6/34/27.png
http://tiles.cdnst.net/base/6/34/28.png
http://tiles.cdnst.net/base/6/34/29.png
http://tiles.cdnst.net/base/6/34/30.png

http://tiles.cdnst.net/base/6/35/24.png
http://tiles.cdnst.net/base/7/35/24.png

so does someone know how can I download them all (including all zoom settings)?

if downloading is not an option, can I just render them and make them look like the above?

Thanks for Anwsering and Best Regards

When loading the app, the map screen allowing you to select a geographical area no longer works. The following is displayed in the CMD prompt.

127.0.0.1 - - [30/Jan/2023 19:14:04] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [30/Jan/2023 19:14:04] "GET /materialize/css/materialize.min.css HTTP/1.1" 200 -
127.0.0.1 - - [30/Jan/2023 19:14:04] "GET /fonts/fonts.css HTTP/1.1" 200 -
127.0.0.1 - - [30/Jan/2023 19:14:04] "GET /style.css HTTP/1.1" 200 -
127.0.0.1 - - [30/Jan/2023 19:14:04] "GET /jquery.min.js HTTP/1.1" 200 -
127.0.0.1 - - [30/Jan/2023 19:14:04] "GET /mapbox-gl-draw-rectangle-mode.min.js HTTP/1.1" 200 -
127.0.0.1 - - [30/Jan/2023 19:14:04] "GET /async.min.js HTTP/1.1" 200 -
127.0.0.1 - - [30/Jan/2023 19:14:04] "GET /turf.min.js HTTP/1.1" 200 -
127.0.0.1 - - [30/Jan/2023 19:14:05] "GET /materialize/js/materialize.min.js HTTP/1.1" 200 -
127.0.0.1 - - [30/Jan/2023 19:14:05] "GET /main.js HTTP/1.1" 200 -
127.0.0.1 - - [30/Jan/2023 19:14:05] "GET /fonts/open-sans-v15-latin-700.woff2 HTTP/1.1" 200 -
127.0.0.1 - - [30/Jan/2023 19:14:05] "GET /fonts/open-sans-v15-latin-600.woff2 HTTP/1.1" 200 -
127.0.0.1 - - [30/Jan/2023 19:14:05] "GET /fonts/open-sans-v15-latin-regular.woff2 HTTP/1.1" 200 -
127.0.0.1 - - [30/Jan/2023 19:14:05] "GET /favicon.ico HTTP/1.1" 200 -

Exception occurred during processing of request from ('127.0.0.1', 50470)
Traceback (most recent call last):
File "C:\Users\alanr\AppData\Local\Programs\Python\Python311\Lib\socketserver.py", line 691, in process_request_thread
self.finish_request(request, client_address)
File "C:\Users\alanr\AppData\Local\Programs\Python\Python311\Lib\socketserver.py", line 361, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Users\alanr\AppData\Local\Programs\Python\Python311\Lib\socketserver.py", line 755, in init
self.handle()
File "C:\Users\alanr\AppData\Local\Programs\Python\Python311\Lib\http\server.py", line 432, in handle
self.handle_one_request()
File "C:\Users\alanr\AppData\Local\Programs\Python\Python311\Lib\http\server.py", line 420, in handle_one_request
method()
File "C:\Users\alanr\Desktop\MapTilesDownloade\src\server.py", line 219, in do_GET
with open(file, "rb") as f:
^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: './UI/favicon.ico'

using docker version and got this error

@AliFlux I think this ability is not working, because if I start downloading, stop, and after that start again it is downloading already downloaded files

Yandex map uses EPSG3395 CRS.
https://core-renderer-tiles.maps.yandex.net/tiles?l=map&x={x}&y={y}&z={z}

MTD downloads tiles incorrectly.

When using the tool for larger tilesets, I ran into some easily fixable issues.

Rendering problems

During downloading the process gets appended in the textarea in the sidebar. This causes lagging until the page reacts very slow or even freezes complitely after some time. The issue here is a very long text inside the textarea.
You can work around it, if you limit the content to a certain length, which can be achived with a small call to a substring function like so:
function logItemRaw(text) {
var logger = $('#log-view');
var value = logger.val() + '\n' + text;
var maxLength = 10000;
if(value.length > maxLength){
value = value.substring(value.length - maxLength, value.length);
}
logger.val(value);
logger.scrollTop(logger[0].scrollHeight);
}

Out of memory

When downloading for longer periods of time, I ran into an issue, where the tab in Chrome crashes, with an 'Out of memory' message. I noticed that the time until this happens it is related to how large the image tiles are.
The problem is that every request with its result (which also contains the base64 tile) is stored in an array but never removed again. This causes a constant memory increase over time.
You can fix this by removing the data from the array, when it is not necessary anymore.
This can be achived by these lines inside the 'always' function of the ajax request:
var removeIndex = requests.indexOf(request);
if(removeIndex >= 0) requests.splice(removeIndex, 1);

The complete part sould look like this:
.always(function(data) {
i++;
removeLayer(boxLayer);
updateProgress(i, allTiles.length);
done();
var removeIndex = requests.indexOf(request);
if(removeIndex >= 0) requests.splice(removeIndex, 1);
if(cancellationToken) {
return;
}
});

All in all I have to say that your tool is great! Not only by getting the job done, but doing it in a beautifully visualized way!
I could watch it download all day long :)
Thanks for publishing it 👍

Hello,
I made a fantasy game map (not on earth) using QGIS and mapbox (for style) and I'd like to get offline tiles of my map.
How could I add mapbox provider for instance to get tiles?
Thank you

Looks like you have some Windows OS specific code...

$ python3 server.py
Starting Server...
Running Server...
Traceback (most recent call last):
  File "server.py", line 167, in <module>
    run()
  File "server.py", line 162, in run
    os.startfile('UI\\index.htm', 'open')
AttributeError: module 'os' has no attribute 'startfile'

Running Windows 10, can get the app loaded and select area but no download.
Exception happened during processing of request from ('127.0.0.1', 11709)
Traceback (most recent call last):
File "C:\Users\cwilson.KORBEN\AppData\Local\Programs\Python\Python36\lib\socketserver.py", line 639, in process_request_thread
self.finish_request(request, client_address)
File "C:\Users\cwilson.KORBEN\AppData\Local\Programs\Python\Python36\lib\socketserver.py", line 361, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Users\cwilson.KORBEN\AppData\Local\Programs\Python\Python36\lib\socketserver.py", line 696, in init
self.handle()
File "C:\Users\cwilson.KORBEN\AppData\Local\Programs\Python\Python36\lib\http\server.py", line 418, in handle
self.handle_one_request()
File "C:\Users\cwilson.KORBEN\AppData\Local\Programs\Python\Python36\lib\http\server.py", line 406, in handle_one_request
method()
File "server.py", line 153, in do_POST
self.writerByType(outputType).addMetadata(lock, os.path.join("output", outputDirectory), filePath, outputFile, "Map Tiles Downloader via AliFlux", "png", boundsArray, centerArray, minZoom, maxZoom, "mercator", 256 * outputScale)
AttributeError: 'NoneType' object has no attribute 'addMetadata'

does this look pretty accurate?
https://www.rainviewer.com/api/geotiff.html#:~:text=It%20is%20quite%20easy%20to%20convert%20a%20PNG,-co%20compress%3Dlzw%20-of%20Gtiff%20-a_srs%20EPSG%3A3857%20%7Bsource_image%7D%20%7Boutput_image%7D

I am also trying to see if geotiff can then be converted to a different format, say NAIP.

Love this app! However, when dealing with tens of thousands of tiles in a single go, the browser UI eventually crashes (Chrome says, "Aw, Snap! Something went wrong while displaying this webpage."). I'd say 50k tiles seems to be the limit.

I thought maybe it's a memory leak related to the ever-expanding log-view container. But I commented out the logItemRaw() function and it still happens. It's worth noting that the logging definitely slows down the app over time. It would be better to implement a fixed sized queue that only shows the last 25 log events.

Hi!
I tried to use your script for downloading map tiles. I got an error after selecting a region and trying to download the tiles.
"9652,12319,15 : Error while relaying tile
All requests are done"

In terminal it looks like this:

File "C:\Users\KK\AppData\Local\Programs\Python\Python37\lib\http\server.py", line 426, in handle
self.handle_one_request()
Exception happened during processing of request from ('127.0.0.1', 61176)
File "C:\Users\KK\AppData\Local\Programs\Python\Python37\lib\http\server.py", line 414, in handle_one_request
method()
Traceback (most recent call last):
File "C:\Users\KK\Desktop\Nowy folder (3)\server.py", line 81, in do_POST
postvars = parse_multipart(self.rfile, pdict)
File "C:\Users\KK\AppData\Local\Programs\Python\Python37\lib\socketserver.py", line 650, in process_request_thread
self.finish_request(request, client_address)
File "C:\Users\KK\AppData\Local\Programs\Python\Python37\lib\socketserver.py", line 360, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Users\KK\AppData\Local\Programs\Python\Python37\lib\socketserver.py", line 720, in init
self.handle()
File "C:\Users\KK\AppData\Local\Programs\Python\Python37\lib\http\server.py", line 426, in handle
self.handle_one_request()
File "C:\Users\KK\AppData\Local\Programs\Python\Python37\lib\http\server.py", line 414, in handle_one_request
method()
File "C:\Users\KK\Desktop\Nowy folder (3)\server.py", line 81, in do_POST
postvars = parse_multipart(self.rfile, pdict)
File "C:\Users\KK\AppData\Local\Programs\Python\Python37\lib\cgi.py", line 220, in parse_multipart
headers['Content-Length'] = pdict['CONTENT-LENGTH']
KeyError: 'CONTENT-LENGTH'
File "C:\Users\KK\AppData\Local\Programs\Python\Python37\lib\cgi.py", line 220, in parse_multipart
headers['Content-Length'] = pdict['CONTENT-LENGTH']

KeyError: 'CONTENT-LENGTH'

Any idea of what the issue is here? I have no experience in coding etc. Would be great to hear back from you!
Krzysztof