consbio / mbtileserver Goto Github PK

is it possible to serve other files; i.e. like a standard http server?
In particular I am trying to use mapbox-gl.js which uses a .json file to store the style and also the mbtiles source location.
How can I serve the .json file from http://localhost?
Thanks.

Usage:
  mbtileserver [flags]

Flags:
      --cachesize int   Size of cache in MB. (default 250)
  -c, --cert string     X.509 TLS certificate filename.  If present, will be used to enable SSL on the server.
  -d, --dir string      Directory containing mbtiles files. (default "./tilesets")
  -k, --key string      flag usage
  -p, --port int        Server port. (default 8000)

key option should not be 'flag usage'

Use case:
time-enabled tilesets exported from multidimensional climate (and other) data sources in NetCDF.

Some options:
NASA example, where time is part of the URL ahead of zoom level:
https://wiki.earthdata.nasa.gov/pages/viewpage.action?pageId=14518214

Sentinel example, with time as a query param:
http://www.sentinel-hub.com/apps/wms/wmts-parameters
(note, we wouldn't expose z, x, y as query params)

Right now, tiles are dropping out when zooming in beyond maxzoom, which is not helpful for proofing the generated tiles.

To use tiles in a page served on another server or port, we need CORS enabled.

Right now, we only have Mapbox GL styles for polygons and lines.

This is nothing against Leaflet, which remains as awesome as ever.

However, for easier maintenance and enhancements, it would be best to limit our map preview technology to a single type.

This is with regard to maplibre/maputnik#184:

/cc @lukasmartinelli

I'd like to propose to refactor this repository so that it consists of main package and a non-mainpackage so that the tile server can be vendored and used in other Go projects.

I'd be happy to contribute to the necessary effort if @consbio and @brendan-ward are interested.

mbtileserver doesn't work correctly behind a proxy, e.g., Caddy.

It order to do this properly, we may need to allow setting the proxy hostname, port, and scheme.

The map endpoint works fine, but the endpoints created for tiles, etc in the tileJson are not correct.

Starting mbtileserver on a tileset that is >50 GB takes a couple minutes. Since we are using running a few basic queries to determine contents of the tileset, it should not take this long.

Right now we do lots of comparisons on string representation of format. We could do better using an enum and a higher level property.

We need to do this more than ideal because vector tiles require different handling if a few places than image tiles.

Here's the traceback of a sample error:

time="2018-03-28T20:33:10Z" level=error msg="Unable to create service set: could not open mbtiles file \"/root/tiles/ace_aqua_rept_ntv.mbtiles\": database disk image is malformed" 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7fc8a1]

goroutine 1 [running]:
github.com/consbio/mbtileserver/handlers.(*ServiceSet).Handler(0x0, 0xa29650, 0x1, 0xc4203e8ca0, 0xc42000e1b8)
	/go/src/github.com/consbio/mbtileserver/handlers/handlers.go:409 +0x61
main.serve()
	/go/src/github.com/consbio/mbtileserver/main.go:161 +0x59f
main.glob..func1(0xea67e0, 0xc4203d7cb0, 0x0, 0x1)
	/go/src/github.com/consbio/mbtileserver/main.go:34 +0x20
github.com/consbio/mbtileserver/vendor/github.com/spf13/cobra.(*Command).execute(0xea67e0, 0xc42000c070, 0x1, 0x1, 0xea67e0, 0xc42000c070)
	/go/src/github.com/consbio/mbtileserver/vendor/github.com/spf13/cobra/command.go:651 +0x23d
github.com/consbio/mbtileserver/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xea67e0, 0x40c62c, 0xc420024118, 0x0)
	/go/src/github.com/consbio/mbtileserver/vendor/github.com/spf13/cobra/command.go:726 +0x2d4
github.com/consbio/mbtileserver/vendor/github.com/spf13/cobra.(*Command).Execute(0xea67e0, 0x0, 0xbea71cdb785791df)
	/go/src/github.com/consbio/mbtileserver/vendor/github.com/spf13/cobra/command.go:685 +0x2b
main.main()
	/go/src/github.com/consbio/mbtileserver/main.go:66 +0x2d

Would be nice to add tilesets (and possibly remove / replace them) without restarting the server process.

Some options:

1. Add a filesystem watcher and update the map of services on change
2. Add an API endpoint (and associated auth, perhaps an API key) that we call on the running process to request update

Some challenges:

• we are not using a concurrency safe map. (concurrent-map might work)

Should be able to provide rendering info via JSON or using HTML stored in the mbtiles file (like this ).

While the spec includes HTML to make it really easy to drop in place in HTML, it conflates data (the mapping of values to colors) with presentation / layout in a way that seems less than portable.

Hi and thanks for this great project :)

There are 2 things that didn't work for me while following the README:

1. The local installation output was blank (go get github.com/consbio/mbtileserver) and while trying to run mbtileserver I receive a command not found message
2. The docker image seems to not be working at all. While trying to pull the docker image I receive this:

docker pull consbio/mbtileserver:latest
Error response from daemon: pull access denied for consbio/mbtileserver, repository does not exist or may require 'docker login'

Any help would be very appreciated.

Similar to what we did in clover, add a dynamically created Leaflet map using this tile service. Should be as simple as creating a template, injecting variables, and serving that on a known URL.

maybe via go-bindata. See: https://github.com/jmcfarlane/golang-templates-example

https://github.com/brendan-ward/tilemerge

Use case is to make static maps from Leaflet or other interactive maps.

Possible REST API:
/services/<service_name>/merge/z/<x0>:<x1>/<y0>:<y1>.<ext>?xOff=<xOff>&yOff=<yOff>&mapWidth=<mapWidth>&mapHeight=<mapHeight>

where x0:x1, y0:y1 is a tile range, e.g., tiles x0,y0 to x1,y1.

xOff and yOff are offsets either within the image, or outside the image. mapWidth and mapHeight refer to the dimensions from the offset point.

How to handle wrapping the antimeridian is TBD.

Apparently tile services published to ArcGIS Online are not full map services with all capabilities, they are more minimal tile-only services.

Example:
http://tiles.arcgis.com/tiles/QVENGdaPbd4LUkLV/arcgis/rest/services/usfws_critical_habitat_final/MapServer?f=json

This will require providing a few JSON endpoints.

The map preview templates look in the root of the domain for the static assets. This breaks if the server is sitting behind a proxy that has control over the root domain.

Per https://docs.sentry.io/hosted/clients/go/

Unable to create service set: could not open mbtiles file "/tmp/test.mbtiles": Could not detect tile format
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x43f7601]

goroutine 1 [running]:
github.com/consbio/mbtileserver/handlers.(*ServiceSet).Handler(0x0, 0x468d6d8, 0x1, 0xc4204121e0, 0xc4200be068)
	/Users/bcward/go/src/github.com/consbio/mbtileserver/handlers/handlers.go:405 +0x61
main.serve()
	/Users/bcward/go/src/github.com/consbio/mbtileserver/main.go:161 +0x55a
main.glob..func1(0x49203c0, 0xc420395400, 0x0, 0x4)
	/Users/bcward/go/src/github.com/consbio/mbtileserver/main.go:34 +0x20
github.com/consbio/mbtileserver/vendor/github.com/spf13/cobra.(*Command).execute(0x49203c0, 0xc4200c0100, 0x4, 0x4, 0x49203c0, 0xc4200c0100)
	/Users/bcward/go/src/github.com/consbio/mbtileserver/vendor/github.com/spf13/cobra/command.go:651 +0x23d
github.com/consbio/mbtileserver/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x49203c0, 0x400646c, 0xc42009c058, 0x0)
	/Users/bcward/go/src/github.com/consbio/mbtileserver/vendor/github.com/spf13/cobra/command.go:726 +0x2d4
github.com/consbio/mbtileserver/vendor/github.com/spf13/cobra.(*Command).Execute(0x49203c0, 0x0, 0xbed3325b39422aa3)
	/Users/bcward/go/src/github.com/consbio/mbtileserver/vendor/github.com/spf13/cobra/command.go:685 +0x2b
main.main()
	/Users/bcward/go/src/github.com/consbio/mbtileserver/main.go:66 +0x2d

Test file:
test.mbtiles.zip

We should not publish an invalid service, but we also shouldn't completely crash when this occurs.

There is a limit on the number of unique filenames that can be stored and easily used in a given folder.

For a folder structure like:

<tileroot>/a.mbtiles
<tileroot>/b/c.mbtiles

would be published as:

<domain>/a
<domain>/b/c

However, a folder name that is the same as a sibling mbtiles file name would not be allowed, e.g.,

<tileroot>/a.mbtiles
<tileroot>/a/b.mbtiles

reported by @ka7eh

Working on generating a test file to isolate issue. May not be isolated to vector tiles.

I have several MBTiles and I wnat the server to combine them on-the-fly. Is it possible?

Currently, any tileset that is invalid or raises an error during initial processing exits the process via log.Fatal.

Instead, errors should be raised, and tileset excluded from further processing.

Related to #64

In order to better support self-hosted tilesets and styles, we need to support serving other static assets likely to be referenced from those styles, including:

• fonts
• sprites

This returns an error:

http://frozen-island-41032.herokuapp.com/arcgis/rest/services/geography-class-png/MapServer/tile/0/0/0

Hello, I download mbtiles file from openmaptiles, use the command as follow:
mbmatch -debug true -path 2017-07-03_us_new-york.mbtiles.
but when I open the http://localhost:7000/, I can't see anything on browser.

These should go into their own package, and provide an interface instead of being glued into the current implementation.

I often use the preview maps for inspecting data before consuming in projects.

It would be very helpful if I could click on a feature, and show info for that feature in a sidebar, popup, etc. This way I could inspect certain features, and verify that they have the right attributes present, and right attribute values.

the logrus repo was renamed from github.com/Sirupsen/logrus to lowercased github.com/sirupsen/logrus so go build does not work for me anymore because logrus_sentry has changed it already .. what a hickup i can tell you.

After renaming the imports to lowercase in main.go and mbtileserver it compiles.

Is there any chance that you can tackle this issue?

Using vector tiles or UTF8-grids in mbtiles file alongside image tiles.

Given a geographic location and info about map extent / width:

1. determine zoom level
2. find appropriate tile
3. geometric operation (e.g., point in polygon) or lookup into UTF8-grid to find feature(s) at location
4. format attribute data and return

Following the instructions:

You can install this project with

go get github.com/consbio/mbtileserver

This will create and install an executable called mbtileserver.

However, the command does not install anything. Running it with -v only gives:

C:\>go get -v github.com/consbio/mbtileserver
github.com/consbio/mbtileserver (download)

Thank you for your help

Steps to reproduce

$ # mb-util ➠ https://github.com/mapbox/mbutil
$ mb-util ~/png-tiles ./mbtiles/testdata/test.mbtiles
$ docker-compose up -d
$ curl http://127.0.0.1:8080/services/test | jq
{
  "format": "png",
  "id": "test",
  "map": "http://127.0.0.1:8080/services/test/map",
  "maxzoom": 18,
  "minzoom": 0,
  "scheme": "xyz",
  "tilejson": "2.1.0",
  "tiles": [
    "http://127.0.0.1:8080/services/test/tiles/{z}/{x}/{y}.png"
  ]
}
$ curl http://127.0.0.1:8080/services/test/map
Internal Server Error
$ docker-compose logs mbtileserver
...
mbtileserver     | time="2019-11-14T22:57:50Z" level=error msg="could not parse template asset \"map\": html/template: cannot Parse after Execute"
mbtileserver     | time="2019-11-14T22:57:52Z" level=error msg="template: \"map\" is an incomplete or empty template"
mbtileserver     | time="2019-11-14T22:57:53Z" level=error msg="template: \"map\" is an incomplete or empty template"

I want to first of all say that I am extremely happy that someone has put this together and want to say what a great job you have done with this.

Upon using this there is one thing that has come up which is if there is not a tile for something(which is completely normal) but console logs an error of 404 not found. Upon trying to see if i could get mapbox gl to suppress it I came to find out that its not so much a mapbox issue as it is the server returning 404. They recommended having the server send back 204, which is a successful request with no content.

Let me know your thoughts.

Thanks

https://github.com/goji/goji

Need a new command line argument (or some), but should be pretty straightforward:

https://echo.labstack.com/cookbook/auto-tls

Will need to determine requirements for passing email address to Let's Encrypt.

These need to be derived from the min and max zoom levels

We use vendorized deps, no need to go get them now.

Want to avoid future pain from minor API upgrades as had to be fixed in #9

See solutions outlined here

We should better support docker-enabled workflows

Basemap tiles break because they are hardcoded to HTTP in the map preview templates