Giter VIP home page Giter VIP logo

rio-mbtiles's Introduction

rio-mbtiles

image

A plugin for the Rasterio CLI that exports a raster dataset to the MBTiles (version 1.3) format. Features include automatic reprojection and concurrent tile generation.

Usage

Usage: rio mbtiles [OPTIONS] INPUT [OUTPUT]

  Export a dataset to MBTiles (version 1.3) in a SQLite file.

  The input dataset may have any coordinate reference system. It must have
  at least three bands, which will be become the red, blue, and green bands
  of the output image tiles.

  An optional fourth alpha band may be copied to the output tiles by using
  the --rgba option in combination with the PNG or WEBP formats. This option
  requires that the input dataset has at least 4 bands.

  The default quality for JPEG and WEBP output (possible range: 10-100) is
  75. This value can be changed with the use of the QUALITY creation option,
  e.g. `--co QUALITY=90`.  The default zlib compression level for PNG output
  (possible range: 1-9) is 6. This value can be changed like `--co
  ZLEVEL=8`.  Lossless WEBP can be chosen with `--co LOSSLESS=TRUE`.

  If no zoom levels are specified, the defaults are the zoom levels nearest
  to the one at which one tile may contain the entire source dataset.

  If a title or description for the output file are not provided, they will
  be taken from the input dataset's filename.

  This command is suited for small to medium (~1 GB) sized sources.

  Python package: rio-mbtiles (https://github.com/mapbox/rio-mbtiles).

Options:
  -o, --output PATH               Path to output file (optional alternative to
                                  a positional arg).

  --append / --overwrite          Append tiles to an existing file or
                                  overwrite.

  --title TEXT                    MBTiles dataset title.
  --description TEXT              MBTiles dataset description.
  --overlay                       Export as an overlay (the default).
  --baselayer                     Export as a base layer.
  -f, --format [JPEG|PNG|WEBP]    Tile image format.
  --tile-size INTEGER             Width and height of individual square tiles
                                  to create.  [default: 256]

  --zoom-levels MIN..MAX          A min...max range of export zoom levels. The
                                  default zoom level is the one at which the
                                  dataset is contained within a single tile.

  --image-dump PATH               A directory into which image tiles will be
                                  optionally dumped.

  -j INTEGER                      Number of workers (default: number of
                                  computer's processors).

  --src-nodata FLOAT              Manually override source nodata
  --dst-nodata FLOAT              Manually override destination nodata
  --resampling [nearest|bilinear|cubic|cubic_spline|lanczos|average|mode|gauss|max|min|med|q1|q3|rms]
                                  Resampling method to use.  [default:
                                  nearest]

  --version                       Show the version and exit.
  --rgba                          Select RGBA output. For PNG or WEBP only.
  --implementation [cf|mp]        Concurrency implementation. Use
                                  concurrent.futures (cf) or multiprocessing
                                  (mp).

  -#, --progress-bar              Display progress bar.
  --covers TEXT                   Restrict mbtiles output to cover a quadkey
  --cutline PATH                  Path to a GeoJSON FeatureCollection to be
                                  used as a cutline. Only source pixels within
                                  the cutline features will be exported.

  --oo NAME=VALUE                 Format driver-specific options to be used
                                  when accessing the input dataset. See the
                                  GDAL format driver documentation for more
                                  information.

  --co, --profile NAME=VALUE      Driver specific creation options. See the
                                  documentation for the selected output driver
                                  for more information.

  --wo NAME=VALUE                 See the GDAL warp options documentation for
                                  more information.

  --exclude-empty-tiles / --include-empty-tiles
                                  Whether to exclude or include empty tiles
                                  from the output.

  --help                          Show this message and exit.

Performance

The rio-mbtiles command is suited for small to medium (~1 GB) raster sources. On a MacBook Air, the 1:10M scale Natural Earth raster (a 21,600 x 10,800 pixel, 700 MB TIFF) exports to MBTiles (levels 1 through 5) in 45 seconds.

$ time GDAL_CACHEMAX=256 rio mbtiles NE1_HR_LC.tif \
> -o ne.mbtiles --zoom-levels 1..5 -j 4

real    0m44.925s
user    1m20.152s
sys     0m22.428s

Installation

pip install rio-mbtiles

rio-mbtiles's People

Contributors

geowurster avatar jdmcbr avatar sgillies avatar spatialhast avatar yoninachmany avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

rio-mbtiles's Issues

Support cutlines

It would be useful to be able to pass a cutline to rio mbtiles. Ideally, something like:

$ rio mbtiles --cutline extents.geojson --zoom-levels 10..3 in.vrt out.mbtiles

Error when source has alpha band

When I run the following:

rio --verbose mbtiles --force-overwrite --title ''  --description '' --overlay --format PNG --zoom-levels 1..20 -j 4 --src-nodata 0 raster.tif output.mbtiles

where raster.tif has an Alpha band

For example using NE2_50M_SR and running:

gdalwarp -of GTiff -dstalpha NE2_50M_SR/NE2_50M_SR.tif raster.tif

creating a 4th alpha band:

Band 1 Block=10800x1 Type=Byte, ColorInterp=Red
  Mask Flags: PER_DATASET ALPHA 
Band 2 Block=10800x1 Type=Byte, ColorInterp=Green
  Mask Flags: PER_DATASET ALPHA 
Band 3 Block=10800x1 Type=Byte, ColorInterp=Blue
  Mask Flags: PER_DATASET ALPHA 
Band 4 Block=10800x1 Type=Byte, ColorInterp=Alpha

rio produces the output:

Traceback (most recent call last):
  File "/usr/local/bin/rio", line 11, in <module>
    sys.exit(main_group())
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/mbtiles/scripts/cli.py", line 187, in mbtiles
    for tile, contents in pool.imap_unordered(process_tile, tiles):
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 668, in next
    raise value
IndexError: tuple index out of range

1.0 release

Rasterio isn't at 1.0 yet, but this is quite solid. Let's go straight to 1.0.

  • Release Rasterio 0.23
  • Tag
  • Build dists
  • Upload to PyPI

Require Python 3.7+

Specifically: require concurrent.futures with ThreadPool initializer and initargs.

Consider thread safety for sqlite

https://www.sqlite.org/threadsafe.html

Code to review in this regard is mostly in a few lines at

rio-mbtiles has deadlocks and the primary suspect is using a multiprocessing pool with sqlite, so

  • please consider the thread safety of sqlite
  • when -j 1 is given, do not use a multiprocessing Pool and run everything in the main thread
  • switch from multiprocessing.cpu_count to use psutil instead, esp. when using threads because the GIL will limit it anyway (right?), e.g.
import psutil

#: physical cores (at least 1, or Ncores-1)
cpu_cores = max(1, psutil.cpu_count(logical=False) - 1)

need way to force overwrite, rasterio 1.0 compat

Recent Rasterio versions require explicit flags to overwrite existing files.

Two problems here:

  1. Our test suite tries to overwrite the test fixtures. They should probably just be written to temp space, right?
  2. We have no way to specify --force-overwrite at the CLI should we want this behavior.

In cli.py we use from rasterio.rio.helpers import resolve_inout which takes a force_overwrite keyword argument. We could pass True and keep the old behavior if needed.

4 ... out of range for dataset

Running:

rio mbtiles -o file.mbtiles -f WEBP --rgba --zoom-levels 6..15 file.tif

results with error:

Traceback (most recent call last):
  File "/usr/local/bin/rio", line 8, in <module>
    sys.exit(main_group())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/mbtiles/scripts/cli.py", line 575, in mbtiles
    process_tiles(
  File "/usr/local/lib/python3.9/site-packages/mbtiles/cf.py", line 53, in process_tiles
    tile, contents = future.result()
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 438, in result
    return self.__get_result()
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
rasterio._err.CPLE_IllegalArgError: panSrcBands[3] = 4 ... out of range for dataset

Recent Rasterio breaks rio-mbtiles

with rasterio.open('/vsimem/tileimg', 'w', **kwds) as tmp:

as used in mbtiles/init.py doesn't work with Rasterio 0.31. Reverting to 0.29 is a temporary work around.

wrond dataset of mbtiles and png/jpg files

I install rio-mbtiles on Ubuntu 14.04.3 and try create mbtiles from my raster and raster in tests/data/ folder:
rio mbtiles RGB.byte.tif -o raster.mbtiles -j 4 --format PNG --image-dump /home/box/rio-mbtiles/tests/data
mbtiles and image tiles successfully created but image tiles (png and jpg) don't opened in standart image files viewer. Also mbtiles file don't opened in QGIS (opened as tables (tiles and metadata)) or another mbtiles viewer. Example output data available on link https://www.dropbox.com/s/kfg2pywe9q21adx/rio-mbtiles_data.zip?dl=0
How I can get valid mbtiles?

Is it possible to mantain native resolution at high zoom levels?

Goal

Create high resolution mbtiles to serve in mapbox studio and connect with a R shiny leaflet app. The tiles are geotiff orthomosiacs collected by drone. The objects of interest are small. Zoom levels > 18 are the only levels of interest.

Problem

The uploaded tiles are too pixelated at high resolutions. I can see several interpolation methods. Is it possible to control the desired minimum resolution of tiles at a given zoom level?

Screen Shot 2020-04-28 at 3 43 14 PM

Code

(mapbox) [b.weinstein@login4 6th Bridge]$ rio mbtiles 6thBridge_03182020_webmercator.tif -o /orange/ewhite/everglades/mapbox/riotest.mbtiles --zoom-levels 17..22 -j 4 -f PNG --overwrite

System Details

Ubuntu HPC terminal. Clean conda env. Installed from pypi.

(mapbox) [b.weinstein@login4 6th Bridge]$ rio mbtiles --version
1.4.2

Image Details

(mapbox) [b.weinstein@login4 6th Bridge]$ gdalinfo 6thBridge_03182020_webmercator.tif
Driver: GTiff/GeoTIFF
Files: 6thBridge_03182020_webmercator.tif
Size is 49490, 72283
Coordinate System is:
PROJCS["WGS 84 / Pseudo-Mercator",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Mercator_1SP"],
    PARAMETER["central_meridian",0],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["X",EAST],
    AXIS["Y",NORTH],
    EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs"],
    AUTHORITY["EPSG","3857"]]
Origin = (-8966045.884076893329620,3014557.724904019385576)
Pixel Size = (0.011984415262908,-0.011984415262908)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (-8966045.884, 3014557.725) ( 80d32'36.10"W, 26d 7'29.62"N)
Lower Left  (-8966045.884, 3013691.455) ( 80d32'36.10"W, 26d 7' 4.46"N)
Upper Right (-8965452.775, 3014557.725) ( 80d32'16.92"W, 26d 7'29.62"N)
Lower Right (-8965452.775, 3013691.455) ( 80d32'16.92"W, 26d 7' 4.46"N)
Center      (-8965749.330, 3014124.590) ( 80d32'26.51"W, 26d 7'17.04"N)
Band 1 Block=49490x1 Type=Byte, ColorInterp=Red
  Unit Type: metre
Band 2 Block=49490x1 Type=Byte, ColorInterp=Green
  Unit Type: metre
Band 3 Block=49490x1 Type=Byte, ColorInterp=Blue
  Unit Type: metre

I recognize that the size of image I am using is larger than recommended. I had previously tried using gdal, but received difficult to debug messages on mapbox studio after uploading. I only mention in case it is somehow related:

gdalwarp 6thBridge_03182020.tif  6thBridge_03182020_webmercator.tif -t_srs "EPSG:3857" 
gdal_translate 6thBridge_03182020_webmercator.tif /orange/ewhite/everglades/mapbox/test.mbtiles -of mbtiles -co ZLEVEL=9
mapbox upload bweinstein.6thBridge_03182020_tiles /orange/ewhite/everglades/mapbox/test.mbtiles

yields from the mapbox studio processing notifications

center z value must be an integer between 0 and 30

All feedback welcome.

Thanks for this plugin. Directly uploading to mapbox studio mysteriously failed at high resolutions in leaflet and I couldn't get gdal mbtiles to behave, so this was some hope. I'm still searching for a good strategy.

1.6.0 release

New features like an option to keep empty tiles and create an alpha band for RGB input which has a mask of some kind.

Add --covers option to define output extent

Would work like --cutline but without masking. Possible values:

  • A GeoJSON FeatureCollection
  • A single web mercator quadkey string
  • A single JSON-encoded [x, y, z] web mercator tile object

Create non-CLI version of tile export

I'd like to export datasets as tiles files within python calls, rather than using the CLI. Essentially thinking something that (using all default arguments) looked like:

import mbtiles
mbtiles.export(tiff_filename, tiles_filename)

Is this of any interest? Back when I did this for rasterio.merge (rasterio/rasterio#460), keeping file i/o out of the core merge functionality was important. It doesn't seem like there's any useful version of the mbtiles.export I'm suggesting that wouldn't actually take a file in and generate a tiles file.

How to call mbtiles from python?

I have a list of .tif filenames as strings and I want to create a bunch of mbtiles. I can't figure out how to do it from a python script.

possible to produce transparency in tiles?

I have a GeoTIFF in which nodata values are represented with 0. If I use rio mbtiles file.tif -o file.mbtiles --src-nodata 0 this produces black in areas of no data, rather than the transparency that I was hoping for. Is it possible to produce transparency? If so, what am I doing wrong?

Add 512x512 tile option

512x512 tiles play better with the Mapbox platform and are a win for users that want to upload mbtiles and make Mapbox tilesets.

Failed to create zoom 1 tile when source crs is in UTM projection

Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/data/bukapeta/python-dev/ENV/lib/python3.6/site-packages/mbtiles/__init__.py", line 66, in process_tile
    tile_window = tile_window.round_offsets().round_shape()
  File "/data/bukapeta/python-dev/ENV/lib/python3.6/site-packages/rasterio/windows.py", line 691, in round_offsets
    pixel_precision is not None else self.col_off),
ValueError: cannot convert float NaN to integer

I think it's because rasterio.warp.transform_bounds returned infinity number
https://github.com/mapbox/rio-mbtiles/blob/master/mbtiles/__init__.py#L58

import rasterio
rasterio.warp.transform_bounds('EPSG:3857', 'EPSG:32749', 0.0, -20037508.342789255, 20037508.342789244, -7.081154551613622e-10)

returned (-14601517.568011362, -9093361.687045071, inf, inf)

missing tile table

when i use the command
rio mbtiles --zoom-levels 1..5 -j 8 1.tif -o 2.mbtiles

it comes out with a few warnings like
D:\Anaconda\envs\rio\lib\site-packages\mercantile\__init__.py:79: FutureWarning: Mercantile 2.0 will require tile x and y to be within the range (0, 2 ** zoom) FutureWarning, D:\Anaconda\envs\rio\lib\site-packages\mercantile\__init__.py:79: FutureWarning: Mercantile 2.0 will require tile x and y to be within the range (0, 2 ** zoom) FutureWarning, D:\Anaconda\envs\rio\lib\site-packages\mercantile\__init__.py:79: FutureWarning: Mercantile 2.0 will require tile x and y to be within the range (0, 2 ** zoom) FutureWarning,

And the mbtiles format result cannot be opened for missing tiles table.

I know few about this field, please help me if you know why this happens, thank you!

Latest rasterio could not load mbtiles plugin.

tested with rasterio v1.0.2 on Ubuntu 16.04

$ rio mbtiles --help
  Warning: entry point could not be loaded. Contact its author for help.

  Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/click_plugins/core.py", line 37, in decorator
      group.add_command(entry_point.load())
    File "/home/kn/.local/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2321, in load
      return self.resolve()
    File "/home/kn/.local/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2327, in resolve
      module = __import__(self.module_name, fromlist=['__name__'], level=0)
    File "/usr/local/lib/python3.5/dist-packages/mbtiles/scripts/cli.py", line 13, in <module>
      from rasterio.rio.options import force_overwrite_opt, output_opt
  ImportError: cannot import name 'force_overwrite_opt'

travis fails to build Rasterio

rasterio/_base.c:287:22: fatal error: cpl_conv.h: No such file or directory - looks like no python-dev. In fact, looks like this repo is set up to use old linux wheels.

@sgillies maybe this would be a good place to employ the manylinux wheels for 1.0a1?

rasterio._err.CPLE_NotSupportedError: Cannot find coordinate operations

Works OK on pyproj==2.2.* but not on pyproj>=2.3.*

  • assuming pyproj is the key library involved in this bug, not entirely sure about that
  • pyproj==2.2.* and pyproj==2.3.* should both use proj==6.1.*
$ pip freeze | grep -E -i 'rasterio|proj|gdal|geopandas|crs'
GDAL==3.0.1
geopandas==0.6.2
pyproj==2.3.1
rasterio==1.1.0

I should add that conda is used to install almost everything and subsequently pip is used to install rio-mbtiles (maybe a conda-forge package for rio-mbtiles is an option?).

The command with -j 3 on a 4-physical-core system fails (backtrace below) and with -j 1 it is OK, i.e. the command works with:

    rio mbtiles -j 1 \
      --format PNG --rgba --overlay --overwrite \
      --resampling nearest \
      --title "${title}" \
      "$input" -o "$output"
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/opt/conda/envs/gis-dataprocessing/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/opt/conda/envs/gis-dataprocessing/lib/python3.7/site-packages/mbtiles/__init__.py", line 89, in process_tile
    resampling=resampling)
  File "/opt/conda/envs/gis-dataprocessing/lib/python3.7/site-packages/rasterio/env.py", line 394, in wrapper
    return f(*args, **kwds)
  File "/opt/conda/envs/gis-dataprocessing/lib/python3.7/site-packages/rasterio/env.py", line 620, in wrapper
    return f(*args, **kwds)
  File "/opt/conda/envs/gis-dataprocessing/lib/python3.7/site-packages/rasterio/warp.py", line 347, in reproject
    warp_mem_limit=warp_mem_limit, **kwargs)
  File "rasterio/_warp.pyx", line 480, in rasterio._warp._reproject
  File "rasterio/_warp.pyx", line 460, in rasterio._warp._reproject
  File "rasterio/_err.pyx", line 205, in rasterio._err.exc_wrap_pointer
rasterio._err.CPLE_NotSupportedError: Cannot find coordinate operations from 'PROJCRS["unnamed",BASEGEOGCRS["Normal Sphere (r=6370997)",DATUM["unknown",ELLIPSOID["sphere",6370997,0,LENGTHUNIT["metre",1,ID["EPSG",9001]]]],PRIMEM["Greenwich",0,ANGLEUNIT["degree",0.0174532925199433,ID["EPSG",9122]]]],CONVERSION["unnamed",METHOD["Lambert Conic Conformal (2SP)",ID["EPSG",9802]],PARAMETER["Latitude of false origin",19.6181716918945,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8821]],PARAMETER["Longitude of false origin",-153.800003051758,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8822]],PARAMETER["Latitude of 1st standard parallel",10,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8823]],PARAMETER["Latitude of 2nd standard parallel",30,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8824]],PARAMETER["Easting at false origin",0,LENGTHUNIT["metre",1],ID["EPSG",8826]],PARAMETER["Northing at false origin",0,LENGTHUNIT["metre",1],ID["EPSG",8827]]],CS[Cartesian,2],AXIS["easting",east,ORDER[1],LENGTHUNIT["metre",1,ID["EPSG",9001]]],AXIS["northing",north,ORDER[2],LENGTHUNIT["metre",1,ID["EPSG",9001]]]]' to 'EPSG:3857'
"""

Using -j 2 got a different error, i.e.

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "rasterio/_crs.pyx", line 363, in rasterio._crs._CRS.from_user_input
  File "rasterio/_err.pyx", line 194, in rasterio._err.exc_wrap_ogrerr
rasterio._err.CPLE_BaseError: OGR Error code 6

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/envs/gis-dataprocessing/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/opt/conda/envs/gis-dataprocessing/lib/python3.7/site-packages/mbtiles/__init__.py", line 65, in process_tile
    with memfile.open(**kwds) as tmp:
  File "/opt/conda/envs/gis-dataprocessing/lib/python3.7/site-packages/rasterio/env.py", line 394, in wrapper
    return f(*args, **kwds)
  File "/opt/conda/envs/gis-dataprocessing/lib/python3.7/site-packages/rasterio/io.py", line 136, in open
    nodata=nodata, sharing=sharing, **kwargs)
  File "rasterio/_io.pyx", line 2000, in rasterio._io.BufferedDatasetWriterBase.__init__
  File "rasterio/_io.pyx", line 1247, in rasterio._io.DatasetWriterBase._set_crs
  File "/opt/conda/envs/gis-dataprocessing/lib/python3.7/site-packages/rasterio/crs.py", line 459, in from_user_input
    obj._crs = _CRS.from_user_input(value, morph_from_esri_dialect=morph_from_esri_dialect)
  File "rasterio/_crs.pyx", line 367, in rasterio._crs._CRS.from_user_input
rasterio.errors.CRSError: The WKT could not be parsed. OGR Error code 6
"""

There seems to be some multiprocessing (IPC?) problems.

support for multifile tile conversion

rio mbtiles currently does not complain if a user passes in multiple input files, but takes the first input file, converts to mbtiles, and ignores the rest. I took a very quick crack at support for multiple input files, expected it to fail where one tile included data from two (or more) input files, and confirmed that it failed. I have not spent any more time looking into how difficult it would be to support inserting data from multiple files. @sgillies Given the CLI takes multiple input files, I thought it might be something that you'd already given some thought to, so wanted to check in for your sense of feasibility and desirability of this before proceeding any farther on this myself.

The use case I ran into myself that motivated this was generating mbtiles for a very large raster, of which only about 5% were data pixels. Processing these in multiple files that were each basically 100% data was considerably faster, and for certain situations I had, I actually got acceptable results in splitting up the processing to be done with multiple files rather than one. It seems plausible that there's another way of going about this by differently handling areas of nodata within a large raster.

Feature request: add support for PNG8

Thank you for making rio-mbtiles, it's great.
I'm wondering if it would be possible to add support for PNG8, which produces drastically smaller file sizes for png tiles.

Thanks.

MBtiles command not found

I am trying to generate mbtiles from geotiff using rio-mbtiles.

Although the installation (pip install rio-mbtiles) shows success and reinstallation says Requirement already satisfied, I am unable to run rio mbtiles command

When run rio mbtiles time GDAL_CACHEMAX=256 rio mbtiles input.tif -o ne.mbtiles --zoom-levels 1..5 -j 4
It errors out saying, No such command 'mbtiles'

Usage: rio [OPTIONS] COMMAND [ARGS]...

  Rasterio command line interface.

Options:
  -v, --verbose           Increase verbosity.
  -q, --quiet             Decrease verbosity.
  --aws-profile TEXT      Select a profile from the AWS credentials file
  --aws-no-sign-requests  Make requests anonymously
  --aws-requester-pays    Requester pays data transfer costs
  --version               Show the version and exit.
  --gdal-version
  --help                  Show this message and exit.

Commands:
  blocks     Write dataset blocks as GeoJSON features.
  bounds     Write bounding boxes to stdout as GeoJSON.
  calc       Raster data calculator.
  clip       Clip a raster to given bounds.
  convert    Copy and convert raster dataset.
  edit-info  Edit dataset metadata.
  env        Print information about the Rasterio environment.
  gcps       Print ground control points as GeoJSON.
  info       Print information about a data file.
  insp       Open a data file and start an interpreter.
  mask       Mask in raster using features.
  merge      Merge a stack of raster datasets.
  overview   Construct overviews in an existing dataset.
  rasterize  Rasterize features.
  rm         Delete a dataset.
  sample     Sample a dataset.
  shapes     Write shapes extracted from bands or masks.
  stack      Stack a number of bands into a multiband dataset.
  transform  Transform coordinates.
  warp       Warp a raster dataset.

System Details:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.3 LTS
Release: 20.04
Codename: focal

Python : python --version
Python 3.8.13

Any help is appreciated.

Also, I am not able to figure out if there is a way to run this from inside python(except for subprocess / os.system to call the CLI )

rio mbtiles MemoryError

tif size:3G
ram:centos 7 1G VMware
time rio mbtiles town.tif -o ttt.mbtiles --zoom-levels 11..18 -j 3 -f PNG

the result:
WARNING:rasterio._env:CPLE_NotSupported in driver PNG does not support creation option SHARING WARNING:rasterio._env:CPLE_NotSupported in driver PNG does not support creation option SHARING WARNING:rasterio._env:CPLE_NotSupported in driver PNG does not support creation option SHARING WARNING:rasterio._env:CPLE_NotSupported in driver PNG does not support creation option SHARING WARNING:rasterio._env:CPLE_NotSupported in driver PNG does not support creation option SHARING Traceback (most recent call last): File "/usr/bin/rio", line 11, in <module> load_entry_point('rasterio==1.0.24', 'console_scripts', 'rio')() File "/usr/lib/python2.7/site-packages/Click-7.0-py2.7.egg/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/usr/lib/python2.7/site-packages/Click-7.0-py2.7.egg/click/core.py", line 717, in main rv = self.invoke(ctx) File "/usr/lib/python2.7/site-packages/Click-7.0-py2.7.egg/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python2.7/site-packages/Click-7.0-py2.7.egg/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/lib/python2.7/site-packages/Click-7.0-py2.7.egg/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/usr/lib/python2.7/site-packages/Click-7.0-py2.7.egg/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/rio_mbtiles-1.4.2-py2.7.egg/mbtiles/scripts/cli.py", line 214, in mbtiles for tile, contents in pool.imap_unordered(process_tile, tiles): File "/usr/lib64/python2.7/multiprocessing/pool.py", line 655, in next raise value MemoryError

The tool cause my vm to slow...

so i develop a tif tile tool with node-gdal:
/opt/gdaltest/tiles/18/159699/135273.png /opt/gdaltest/tiles/18/159699/135274.png /opt/gdaltest/tiles/18/159699/135275.png /opt/gdaltest/tiles/18/159699/135276.png /opt/gdaltest/tiles/18/159699/135277.png /opt/gdaltest/tiles/18/159699/135278.png /opt/gdaltest/tiles/18/159699/135279.png /opt/gdaltest/tiles/18/159699/135280.png 切片耗时: 303600.441ms
the tool runs ok and run faster(Asynchronous io?),so i think the rio mbtiles so slow may be had performance problem

1.2 release

Switch entry point to rasterio.rio_plugins and require Rasterio >= 0.25.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.