mapbox / rio-mbtiles Goto Github PK
View Code? Open in Web Editor NEWA plugin command for the Rasterio CLI that exports a raster dataset to an MBTiles 1.1 SQLite file
License: MIT License
A plugin command for the Rasterio CLI that exports a raster dataset to an MBTiles 1.1 SQLite file
License: MIT License
Switch entry point to rasterio.rio_plugins and require Rasterio >= 0.25.
Output tiles are written by GDAL, remember.
Create high resolution mbtiles to serve in mapbox studio and connect with a R shiny leaflet app. The tiles are geotiff orthomosiacs collected by drone. The objects of interest are small. Zoom levels > 18 are the only levels of interest.
The uploaded tiles are too pixelated at high resolutions. I can see several interpolation methods. Is it possible to control the desired minimum resolution of tiles at a given zoom level?
(mapbox) [b.weinstein@login4 6th Bridge]$ rio mbtiles 6thBridge_03182020_webmercator.tif -o /orange/ewhite/everglades/mapbox/riotest.mbtiles --zoom-levels 17..22 -j 4 -f PNG --overwrite
Ubuntu HPC terminal. Clean conda env. Installed from pypi.
(mapbox) [b.weinstein@login4 6th Bridge]$ rio mbtiles --version
1.4.2
(mapbox) [b.weinstein@login4 6th Bridge]$ gdalinfo 6thBridge_03182020_webmercator.tif
Driver: GTiff/GeoTIFF
Files: 6thBridge_03182020_webmercator.tif
Size is 49490, 72283
Coordinate System is:
PROJCS["WGS 84 / Pseudo-Mercator",
GEOGCS["WGS 84",
DATUM["WGS_1984",
SPHEROID["WGS 84",6378137,298.257223563,
AUTHORITY["EPSG","7030"]],
AUTHORITY["EPSG","6326"]],
PRIMEM["Greenwich",0,
AUTHORITY["EPSG","8901"]],
UNIT["degree",0.0174532925199433,
AUTHORITY["EPSG","9122"]],
AUTHORITY["EPSG","4326"]],
PROJECTION["Mercator_1SP"],
PARAMETER["central_meridian",0],
PARAMETER["scale_factor",1],
PARAMETER["false_easting",0],
PARAMETER["false_northing",0],
UNIT["metre",1,
AUTHORITY["EPSG","9001"]],
AXIS["X",EAST],
AXIS["Y",NORTH],
EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs"],
AUTHORITY["EPSG","3857"]]
Origin = (-8966045.884076893329620,3014557.724904019385576)
Pixel Size = (0.011984415262908,-0.011984415262908)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left (-8966045.884, 3014557.725) ( 80d32'36.10"W, 26d 7'29.62"N)
Lower Left (-8966045.884, 3013691.455) ( 80d32'36.10"W, 26d 7' 4.46"N)
Upper Right (-8965452.775, 3014557.725) ( 80d32'16.92"W, 26d 7'29.62"N)
Lower Right (-8965452.775, 3013691.455) ( 80d32'16.92"W, 26d 7' 4.46"N)
Center (-8965749.330, 3014124.590) ( 80d32'26.51"W, 26d 7'17.04"N)
Band 1 Block=49490x1 Type=Byte, ColorInterp=Red
Unit Type: metre
Band 2 Block=49490x1 Type=Byte, ColorInterp=Green
Unit Type: metre
Band 3 Block=49490x1 Type=Byte, ColorInterp=Blue
Unit Type: metre
I recognize that the size of image I am using is larger than recommended. I had previously tried using gdal, but received difficult to debug messages on mapbox studio after uploading. I only mention in case it is somehow related:
gdalwarp 6thBridge_03182020.tif 6thBridge_03182020_webmercator.tif -t_srs "EPSG:3857"
gdal_translate 6thBridge_03182020_webmercator.tif /orange/ewhite/everglades/mapbox/test.mbtiles -of mbtiles -co ZLEVEL=9
mapbox upload bweinstein.6thBridge_03182020_tiles /orange/ewhite/everglades/mapbox/test.mbtiles
yields from the mapbox studio processing notifications
center z value must be an integer between 0 and 30
All feedback welcome.
Thanks for this plugin. Directly uploading to mapbox studio mysteriously failed at high resolutions in leaflet and I couldn't get gdal mbtiles to behave, so this was some hope. I'm still searching for a good strategy.
New features like an option to keep empty tiles and create an alpha band for RGB input which has a mask of some kind.
Not a fan of underscored module names, sets a poor example for other plugin projects.
Target date: 2020-10-31.
I install rio-mbtiles on Ubuntu 14.04.3 and try create mbtiles from my raster and raster in tests/data/ folder:
rio mbtiles RGB.byte.tif -o raster.mbtiles -j 4 --format PNG --image-dump /home/box/rio-mbtiles/tests/data
mbtiles and image tiles successfully created but image tiles (png and jpg) don't opened in standart image files viewer. Also mbtiles file don't opened in QGIS (opened as tables (tiles and metadata)) or another mbtiles viewer. Example output data available on link https://www.dropbox.com/s/kfg2pywe9q21adx/rio-mbtiles_data.zip?dl=0
How I can get valid mbtiles?
with rasterio.open('/vsimem/tileimg', 'w', **kwds) as tmp:
as used in mbtiles/init.py doesn't work with Rasterio 0.31. Reverting to 0.29 is a temporary work around.
I'd like to export datasets as tiles files within python calls, rather than using the CLI. Essentially thinking something that (using all default arguments) looked like:
import mbtiles
mbtiles.export(tiff_filename, tiles_filename)
Is this of any interest? Back when I did this for rasterio.merge
(rasterio/rasterio#460), keeping file i/o out of the core merge functionality was important. It doesn't seem like there's any useful version of the mbtiles.export
I'm suggesting that wouldn't actually take a file in and generate a tiles file.
Should be mbtiles. This is a critical bug; the plugin is broken.
Better to make rio-mbtiles faster by default and have num_workers default to the number of available CPUs (like GNU Parallel).
Specifically, add an --append
option (short -r
like tar, maybe?) and if true, neither delete nor create the existing db and table.
Version 1.3 of MBTiles allows WebP tiles. I expect this to be an easy addition.
I have two questions to help with:
512x512 tiles play better with the Mapbox platform and are a win for users that want to upload mbtiles and make Mapbox tilesets.
Rasterio 1.0a1 allows multiband reprojection, which will make rio-mbtiles 2-3x faster.
rio mbtiles
currently does not complain if a user passes in multiple input files, but takes the first input file, converts to mbtiles, and ignores the rest. I took a very quick crack at support for multiple input files, expected it to fail where one tile included data from two (or more) input files, and confirmed that it failed. I have not spent any more time looking into how difficult it would be to support inserting data from multiple files. @sgillies Given the CLI takes multiple input files, I thought it might be something that you'd already given some thought to, so wanted to check in for your sense of feasibility and desirability of this before proceeding any farther on this myself.
The use case I ran into myself that motivated this was generating mbtiles for a very large raster, of which only about 5% were data pixels. Processing these in multiple files that were each basically 100% data was considerably faster, and for certain situations I had, I actually got acceptable results in splitting up the processing to be done with multiple files rather than one. It seems plausible that there's another way of going about this by differently handling areas of nodata within a large raster.
Line 40 in f557854
Allow any version new than 1.7
"shapely>=1.7.0",
I really must automate the building of these platters. Soon.
when i use the command
rio mbtiles --zoom-levels 1..5 -j 8 1.tif -o 2.mbtiles
it comes out with a few warnings like
D:\Anaconda\envs\rio\lib\site-packages\mercantile\__init__.py:79: FutureWarning: Mercantile 2.0 will require tile x and y to be within the range (0, 2 ** zoom) FutureWarning, D:\Anaconda\envs\rio\lib\site-packages\mercantile\__init__.py:79: FutureWarning: Mercantile 2.0 will require tile x and y to be within the range (0, 2 ** zoom) FutureWarning, D:\Anaconda\envs\rio\lib\site-packages\mercantile\__init__.py:79: FutureWarning: Mercantile 2.0 will require tile x and y to be within the range (0, 2 ** zoom) FutureWarning,
And the mbtiles format result cannot be opened for missing tiles table.
I know few about this field, please help me if you know why this happens, thank you!
At mbtiles/__init__.py:62: in process_tile
the following code
tile_window.col_off -= 1
results in an attr.exceptions.FrozenInstanceError
.
I am trying to generate mbtiles from geotiff using rio-mbtiles.
Although the installation (pip install rio-mbtiles) shows success and reinstallation says Requirement already satisfied, I am unable to run rio mbtiles command
When run rio mbtiles time GDAL_CACHEMAX=256 rio mbtiles input.tif -o ne.mbtiles --zoom-levels 1..5 -j 4
It errors out saying, No such command 'mbtiles'
Usage: rio [OPTIONS] COMMAND [ARGS]...
Rasterio command line interface.
Options:
-v, --verbose Increase verbosity.
-q, --quiet Decrease verbosity.
--aws-profile TEXT Select a profile from the AWS credentials file
--aws-no-sign-requests Make requests anonymously
--aws-requester-pays Requester pays data transfer costs
--version Show the version and exit.
--gdal-version
--help Show this message and exit.
Commands:
blocks Write dataset blocks as GeoJSON features.
bounds Write bounding boxes to stdout as GeoJSON.
calc Raster data calculator.
clip Clip a raster to given bounds.
convert Copy and convert raster dataset.
edit-info Edit dataset metadata.
env Print information about the Rasterio environment.
gcps Print ground control points as GeoJSON.
info Print information about a data file.
insp Open a data file and start an interpreter.
mask Mask in raster using features.
merge Merge a stack of raster datasets.
overview Construct overviews in an existing dataset.
rasterize Rasterize features.
rm Delete a dataset.
sample Sample a dataset.
shapes Write shapes extracted from bands or masks.
stack Stack a number of bands into a multiband dataset.
transform Transform coordinates.
warp Warp a raster dataset.
System Details:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.3 LTS
Release: 20.04
Codename: focal
Python : python --version
Python 3.8.13
Any help is appreciated.
Also, I am not able to figure out if there is a way to run this from inside python(except for subprocess / os.system to call the CLI )
Recent Rasterio versions require explicit flags to overwrite existing files.
Two problems here:
--force-overwrite
at the CLI should we want this behavior.In cli.py
we use from rasterio.rio.helpers import resolve_inout
which takes a force_overwrite
keyword argument. We could pass True
and keep the old behavior if needed.
I have a list of .tif filenames as strings and I want to create a bunch of mbtiles. I can't figure out how to do it from a python script.
This version will require Rasterio 1.x and will fix issue #39.
rasterio/_base.c:287:22: fatal error: cpl_conv.h: No such file or directory
- looks like no python-dev. In fact, looks like this repo is set up to use old linux wheels.
@sgillies maybe this would be a good place to employ the manylinux wheels for 1.0a1?
#26 is fixed. Next I'll tag and publish to PyPI.
Rasterio isn't at 1.0 yet, but this is quite solid. Let's go straight to 1.0.
Running:
rio mbtiles -o file.mbtiles -f WEBP --rgba --zoom-levels 6..15 file.tif
results with error:
Traceback (most recent call last):
File "/usr/local/bin/rio", line 8, in <module>
sys.exit(main_group())
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1062, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 763, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/mbtiles/scripts/cli.py", line 575, in mbtiles
process_tiles(
File "/usr/local/lib/python3.9/site-packages/mbtiles/cf.py", line 53, in process_tiles
tile, contents = future.result()
File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 438, in result
return self.__get_result()
File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
raise self._exception
rasterio._err.CPLE_IllegalArgError: panSrcBands[3] = 4 ... out of range for dataset
Specifically: require concurrent.futures with ThreadPool initializer and initargs.
Works OK on pyproj==2.2.*
but not on pyproj>=2.3.*
pyproj
is the key library involved in this bug, not entirely sure about thatpyproj==2.2.*
and pyproj==2.3.*
should both use proj==6.1.*
$ pip freeze | grep -E -i 'rasterio|proj|gdal|geopandas|crs'
GDAL==3.0.1
geopandas==0.6.2
pyproj==2.3.1
rasterio==1.1.0
I should add that conda is used to install almost everything and subsequently pip is used to install rio-mbtiles
(maybe a conda-forge package for rio-mbtiles
is an option?).
The command with -j 3
on a 4-physical-core system fails (backtrace below) and with -j 1
it is OK, i.e. the command works with:
rio mbtiles -j 1 \
--format PNG --rgba --overlay --overwrite \
--resampling nearest \
--title "${title}" \
"$input" -o "$output"
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/opt/conda/envs/gis-dataprocessing/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/opt/conda/envs/gis-dataprocessing/lib/python3.7/site-packages/mbtiles/__init__.py", line 89, in process_tile
resampling=resampling)
File "/opt/conda/envs/gis-dataprocessing/lib/python3.7/site-packages/rasterio/env.py", line 394, in wrapper
return f(*args, **kwds)
File "/opt/conda/envs/gis-dataprocessing/lib/python3.7/site-packages/rasterio/env.py", line 620, in wrapper
return f(*args, **kwds)
File "/opt/conda/envs/gis-dataprocessing/lib/python3.7/site-packages/rasterio/warp.py", line 347, in reproject
warp_mem_limit=warp_mem_limit, **kwargs)
File "rasterio/_warp.pyx", line 480, in rasterio._warp._reproject
File "rasterio/_warp.pyx", line 460, in rasterio._warp._reproject
File "rasterio/_err.pyx", line 205, in rasterio._err.exc_wrap_pointer
rasterio._err.CPLE_NotSupportedError: Cannot find coordinate operations from 'PROJCRS["unnamed",BASEGEOGCRS["Normal Sphere (r=6370997)",DATUM["unknown",ELLIPSOID["sphere",6370997,0,LENGTHUNIT["metre",1,ID["EPSG",9001]]]],PRIMEM["Greenwich",0,ANGLEUNIT["degree",0.0174532925199433,ID["EPSG",9122]]]],CONVERSION["unnamed",METHOD["Lambert Conic Conformal (2SP)",ID["EPSG",9802]],PARAMETER["Latitude of false origin",19.6181716918945,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8821]],PARAMETER["Longitude of false origin",-153.800003051758,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8822]],PARAMETER["Latitude of 1st standard parallel",10,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8823]],PARAMETER["Latitude of 2nd standard parallel",30,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8824]],PARAMETER["Easting at false origin",0,LENGTHUNIT["metre",1],ID["EPSG",8826]],PARAMETER["Northing at false origin",0,LENGTHUNIT["metre",1],ID["EPSG",8827]]],CS[Cartesian,2],AXIS["easting",east,ORDER[1],LENGTHUNIT["metre",1,ID["EPSG",9001]]],AXIS["northing",north,ORDER[2],LENGTHUNIT["metre",1,ID["EPSG",9001]]]]' to 'EPSG:3857'
"""
Using -j 2
got a different error, i.e.
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "rasterio/_crs.pyx", line 363, in rasterio._crs._CRS.from_user_input
File "rasterio/_err.pyx", line 194, in rasterio._err.exc_wrap_ogrerr
rasterio._err.CPLE_BaseError: OGR Error code 6
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/envs/gis-dataprocessing/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/opt/conda/envs/gis-dataprocessing/lib/python3.7/site-packages/mbtiles/__init__.py", line 65, in process_tile
with memfile.open(**kwds) as tmp:
File "/opt/conda/envs/gis-dataprocessing/lib/python3.7/site-packages/rasterio/env.py", line 394, in wrapper
return f(*args, **kwds)
File "/opt/conda/envs/gis-dataprocessing/lib/python3.7/site-packages/rasterio/io.py", line 136, in open
nodata=nodata, sharing=sharing, **kwargs)
File "rasterio/_io.pyx", line 2000, in rasterio._io.BufferedDatasetWriterBase.__init__
File "rasterio/_io.pyx", line 1247, in rasterio._io.DatasetWriterBase._set_crs
File "/opt/conda/envs/gis-dataprocessing/lib/python3.7/site-packages/rasterio/crs.py", line 459, in from_user_input
obj._crs = _CRS.from_user_input(value, morph_from_esri_dialect=morph_from_esri_dialect)
File "rasterio/_crs.pyx", line 367, in rasterio._crs._CRS.from_user_input
rasterio.errors.CRSError: The WKT could not be parsed. OGR Error code 6
"""
There seems to be some multiprocessing (IPC?) problems.
Bug fixes. Target date: 2018-10-17.
I have a GeoTIFF in which nodata values are represented with 0. If I use rio mbtiles file.tif -o file.mbtiles --src-nodata 0
this produces black in areas of no data, rather than the transparency that I was hoping for. Is it possible to produce transparency? If so, what am I doing wrong?
Would work like --cutline
but without masking. Possible values:
[x, y, z]
web mercator tile objectBug fixes.
tested with rasterio v1.0.2 on Ubuntu 16.04
$ rio mbtiles --help
Warning: entry point could not be loaded. Contact its author for help.
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/click_plugins/core.py", line 37, in decorator
group.add_command(entry_point.load())
File "/home/kn/.local/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2321, in load
return self.resolve()
File "/home/kn/.local/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2327, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/usr/local/lib/python3.5/dist-packages/mbtiles/scripts/cli.py", line 13, in <module>
from rasterio.rio.options import force_overwrite_opt, output_opt
ImportError: cannot import name 'force_overwrite_opt'
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/data/bukapeta/python-dev/ENV/lib/python3.6/site-packages/mbtiles/__init__.py", line 66, in process_tile
tile_window = tile_window.round_offsets().round_shape()
File "/data/bukapeta/python-dev/ENV/lib/python3.6/site-packages/rasterio/windows.py", line 691, in round_offsets
pixel_precision is not None else self.col_off),
ValueError: cannot convert float NaN to integer
I think it's because rasterio.warp.transform_bounds
returned infinity
number
https://github.com/mapbox/rio-mbtiles/blob/master/mbtiles/__init__.py#L58
import rasterio
rasterio.warp.transform_bounds('EPSG:3857', 'EPSG:32749', 0.0, -20037508.342789255, 20037508.342789244, -7.081154551613622e-10)
returned (-14601517.568011362, -9093361.687045071, inf, inf)
Adds one new format and the --co
option.
Thank you for making rio-mbtiles, it's great.
I'm wondering if it would be possible to add support for PNG8, which produces drastically smaller file sizes for png tiles.
Thanks.
It would be useful to be able to pass a cutline to rio mbtiles
. Ideally, something like:
$ rio mbtiles --cutline extents.geojson --zoom-levels 10..3 in.vrt out.mbtiles
When I run the following:
rio --verbose mbtiles --force-overwrite --title '' --description '' --overlay --format PNG --zoom-levels 1..20 -j 4 --src-nodata 0 raster.tif output.mbtiles
where raster.tif
has an Alpha band
For example using NE2_50M_SR and running:
gdalwarp -of GTiff -dstalpha NE2_50M_SR/NE2_50M_SR.tif raster.tif
creating a 4th alpha band:
Band 1 Block=10800x1 Type=Byte, ColorInterp=Red
Mask Flags: PER_DATASET ALPHA
Band 2 Block=10800x1 Type=Byte, ColorInterp=Green
Mask Flags: PER_DATASET ALPHA
Band 3 Block=10800x1 Type=Byte, ColorInterp=Blue
Mask Flags: PER_DATASET ALPHA
Band 4 Block=10800x1 Type=Byte, ColorInterp=Alpha
rio produces the output:
Traceback (most recent call last):
File "/usr/local/bin/rio", line 11, in <module>
sys.exit(main_group())
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 716, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 534, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/mbtiles/scripts/cli.py", line 187, in mbtiles
for tile, contents in pool.imap_unordered(process_tile, tiles):
File "/usr/lib/python2.7/multiprocessing/pool.py", line 668, in next
raise value
IndexError: tuple index out of range
tif size:3G
ram:centos 7 1G VMware
time rio mbtiles town.tif -o ttt.mbtiles --zoom-levels 11..18 -j 3 -f PNG
the result:
WARNING:rasterio._env:CPLE_NotSupported in driver PNG does not support creation option SHARING WARNING:rasterio._env:CPLE_NotSupported in driver PNG does not support creation option SHARING WARNING:rasterio._env:CPLE_NotSupported in driver PNG does not support creation option SHARING WARNING:rasterio._env:CPLE_NotSupported in driver PNG does not support creation option SHARING WARNING:rasterio._env:CPLE_NotSupported in driver PNG does not support creation option SHARING Traceback (most recent call last): File "/usr/bin/rio", line 11, in <module> load_entry_point('rasterio==1.0.24', 'console_scripts', 'rio')() File "/usr/lib/python2.7/site-packages/Click-7.0-py2.7.egg/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/usr/lib/python2.7/site-packages/Click-7.0-py2.7.egg/click/core.py", line 717, in main rv = self.invoke(ctx) File "/usr/lib/python2.7/site-packages/Click-7.0-py2.7.egg/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python2.7/site-packages/Click-7.0-py2.7.egg/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/lib/python2.7/site-packages/Click-7.0-py2.7.egg/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/usr/lib/python2.7/site-packages/Click-7.0-py2.7.egg/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/rio_mbtiles-1.4.2-py2.7.egg/mbtiles/scripts/cli.py", line 214, in mbtiles for tile, contents in pool.imap_unordered(process_tile, tiles): File "/usr/lib64/python2.7/multiprocessing/pool.py", line 655, in next raise value MemoryError
The tool cause my vm to slow...
so i develop a tif tile tool with node-gdal:
/opt/gdaltest/tiles/18/159699/135273.png /opt/gdaltest/tiles/18/159699/135274.png /opt/gdaltest/tiles/18/159699/135275.png /opt/gdaltest/tiles/18/159699/135276.png /opt/gdaltest/tiles/18/159699/135277.png /opt/gdaltest/tiles/18/159699/135278.png /opt/gdaltest/tiles/18/159699/135279.png /opt/gdaltest/tiles/18/159699/135280.png 切片耗时: 303600.441ms
the tool runs ok and run faster(Asynchronous io?),so i think the rio mbtiles so slow may be had performance problem
https://www.sqlite.org/threadsafe.html
Code to review in this regard is mostly in a few lines at
rio-mbtiles has deadlocks and the primary suspect is using a multiprocessing pool with sqlite, so
-j 1
is given, do not use a multiprocessing Pool
and run everything in the main threadimport psutil
#: physical cores (at least 1, or Ncores-1)
cpu_cores = max(1, psutil.cpu_count(logical=False) - 1)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.