Comments (12)
This sounds like a good idea, and maputnik
looks like a great project. I'd be happy to help do this.
Before we begin, it would be good to have an understanding of how we want to split mbtileserver
into packages to best support this.
Splitting out mbtiles.go
into its own package seems like an easy first step; this is very self contained and provides the core interface to mbtiles
files.
However, I'm unclear on how you would want what is now in main.go
split out. @fawick would you mind identifying the core capabilities that are lurking in main.go
that are most useful as a library for use in other applications?
At a high level, we have:
- CLI related operations
- HTTP server operations
- in-memory caching
- things that help format the responses with the tile data (could theoretically be split out from the HTTP handlers themselves)
from mbtileserver.
Thank your being open to this, @brendan-ward, that means a lot! 👍
I agree that the type Mbtiles
and its methods are a good place to start. It makes sense to put that into a dedicated package. I'd also move the import of go-sqlite3
from main.go
to there as its the only user of sqlite. That package would then already take care of the most important use case for other users: Being able to read tiles from a Mbtiles server.
Ages ago I wrote a MBTiles implementation for storing raster tiles from mapnik: https://github.com/fawick/go-mapnik/blob/master/maptiles/mbtilesgenerator.go#L16. That code contains members to create a new MBTiles DB and its schema, I could sponsor that code for the new package that is about to be created here, too. That way you'd have an mbtiles packages that could read from and write to the DB.
Another idea would be some convenience methods that wrap Mbtiles.ReadTile
et al., like this
// WriteTileTo reads a tile with coordinates z, x, and y from tileset and writes it to w.
// The first error encountered is returned. If no error occurs, the returned error will be nil.
func (tileset * Mbtiles) WriteTileTo(w io.Writer, z uint8, x uint64, y uint64)
In the same vein I'd remove the calls logrus.Error
and logrus.Errorf
. Returning the error is enough and whether to handle them via logging should be a choise of the package user.
As for main.go
: I believe the CLI is specific to mbtileserver and should remain in package main
. Same goes for TLS support.
I'd refactor the HTTP operations into an own package that only uses net/http.Handler
to lift the hard requirement to labstack/echo
. That way a user could use any HTTP router / framework. For the main
package of mbtileserver I'd stick with echo, as you have probably chosen it for a reason.
I have no good idea for the caching, yet. Maybe it can be implemented as a middleware as well? The caching part of GetTile
would be a http.Handler
and the cacheGetter would then call the net/http.Handler
from above. and cache the body of the response.
What do you think?
from mbtileserver.
I started to work on this here: https://github.com/fawick/mbtileserver/tree/splitIntoPackages (removed)
Would you prefer me to submit atomic PRs after each step or just one PR in the end? Personally, I'd prefer the first, but it's your call.
from mbtileserver.
Thanks for starting on this @fawick !
I'd like atomic PRs; that will help us refine as we go. Don't feel like you have to do it all - I'm happy to jump in and do this too.
I wonder if we shouldn't upframe this a little bit. Let me know your thoughts on this:
- new repo for
mbtiles
I/O. (I'd prefer this under an organization account rather than personal account. Usingconsbio
for org is an option. Creating a new org is also an option) - include that as a library used by
mbtileserver
, which could provide basic HTTP operations in anhttp
package undermbtileserver
. This would include the response handlers for both regular tiles and ArcGIS type tile services. main.go
here could then use CLI and web framework of choice for starting server and muxing requests?
This seems like it would give us better separation of concerns between a repo intended to be an I/O library for mbtiles, a library for formatting HTTP responses for tile / ArcGIS endpoints, and an executable that runs a server.
from mbtileserver.
Re: caching. Our use of groupcache
is total overkill for how we are using it here.
I like the idea of a handling this in the application rather than the library (cache the response data rather than the tile data), perhaps as middleware.
from mbtileserver.
Great, it looks like we have pretty much the same understanding on how to proceed.
I'm happy to bootstrap all of this and shape it in a way so I can use it easily it maputnik/desktop.
Only one comment: For the moment I'd rather keep development in this repo until things are fleshed out. That would mean to keep the mbtiles
package here for the moment. Moving that package to another repo later is easy.
I already have a commit that refactors mbtiles.go
into its own package and starts using it in main.go
. This is going to be my first PR which I create in a moment.
from mbtileserver.
Thanks for your reviews on the PRs #41 and #42. I've included the requested changes.
Further on, I am currenty adding a type ServiceSet
to package handlers, which will be providing the http.Handler
. This will probably take a few days as I can only do this in my limited personal time.
I understand that the GZIP compression is currently done with labstack/echo/middleware.Gzip
. I could not get anything about this out of mbtiles-spec, so I wonder: Is the GZIP compression mandatory for vector tiles? If so, I was considering to use https://github.com/NYTimes/gziphandler instead, as it provides a method function that is compatible to the API we aim for in package handlers
.
PS: gorilla/handlers
has one, too.
from mbtileserver.
Okay, scratch that question, I read the code more closely and saw that the GZIP middleware is not applied to the GET of the tile itself.
from mbtileserver.
@brendan-ward Thank you very much for your reviews and your many great suggestions to make the code better.
With #50 now being merged, the next thing I should probably start working on are the ArcGIS endpoints. Would you agree that these should also be in package handlers
, or is it better to start a new package for them?
I do not have access to an ArcGIS installation (and I have never worked with ArcGIS), so I am unsure how to do an manual integration test for these endpoints. Can you recommend some tool for that? Would they work with, say, QuantumGIS?
Finally, I have some additional exported methods for package handlers
in mind, that would allow to only install some of the endpoints. For example, in maplibre/maputnik#148 we only need the endpoints for serving the TileJSON and the tiles, but not the templates.
Another thing probably would be to start (re)implementing a caching middleware in the main
package.
from mbtileserver.
@fawick Thank YOU for doing all the work to make this better!!
The arcgis endpoints can go into handlers/arcgis.go
since they are handlers, but I suppose for use in maputnik/editor
we wouldn't necessarily want those endpoints available by default? I'm not sure exactly what you have in mind for conditionally installing endpoints, but maybe that would work here as well?
The arcgis endpoints cold also go into a dedicated package under handlers handlers/arcgis
if that seems like a better way of isolating them.
For testing them, I always just hit the endpoints directly (manually) to make sure they respond as expected. There is no need to try and consume them from another application. This is where I really need to learn how to write tests in Go that exercise HTTP endpoints, just haven't had the time.
Do you want me to run with updating those? They seem orthogonal to your needs for integration with maputnik/editor
.
I think next up should be your conditional installation of endpoints.
Is the caching middleware of the tiles something that would be at all useful for maputnik/editor
? Seems to me like it would be useful there, and it would just be up to the consumer of the library to choose to use it. There might be a really simple, in-memory implementation that we could use here (controlling write locks would probably be hardest part). My original use of groupcache
was probably orders of magnitude more complex than we actually needed.
If we are open to swapping out one 3rd party dependency for another, these look promising but would need investigation:
from mbtileserver.
I'd be grateful and glad if you like to take a shot at the ArcGIS handlers, as I relly know next to nothing about ArcGIS and am only guessing the meaning and use cases of these endpoint from the current implementation.
I can start on the smaller endpoint set. I will implement this as a function very similar to ServiceSet.Handler
, but leaving out the /map
endpoint and the service lister endpoint. I even consider making the return value an *http.ServeMux
and then calling this method from Handler
to simply add the maps and the lister again and return the extended http.ServeMux
as the http.Handler
.
For the caching middleware: I am inclined to put the caching middleware into the main
package, and even basing it on labstack/echo
and groupcache
but with the request URL path as the key. But I am without preferences for the 3rdparty cache package. The main reason I suggested the caching was that I did not want to take that feature away from mbtileserver, as you had probably added for a good reason in the first case. For maputnik/editor
(and for another project with mbiles
and handler
) I do not plan to use any caching.
What do you think?
from mbtileserver.
@brendan-ward Thank you very much again for being so open and for all the collaboration on this!
from mbtileserver.
Related Issues (20)
- BUG: invalid tiles cause panic on sqlite connections
- BUG: Map preview fails when 'center' property is not present in metadata HOT 7
- caching options HOT 1
- Build Error with GCC11 HOT 5
- DOC: Add note that user needs to increase max open files limit if using several tilesets
- Deploy arm64 image to Docker Hub HOT 4
- Consider alternative basemap tile providers HOT 9
- Add multiple subdomain support HOT 1
- Missing tiles returned as 256x256 transparent tiles HOT 11
- Serve tiles from blob-storage HOT 1
- it is not support EPSG 4326 HOT 2
- Migrate from Docker Hub to Github Packages registery
- Include version in CLI
- mbtileserver vs Martin Comparison? HOT 2
- Consider adding a /metrics endpoint HOT 1
- Map preview does not work when using reverse proxy with HTTPS HOT 6
- Use of stadiamaps? HOT 9
- Docker 404 Error HOT 4
- Crash when issue with one tile HOT 4
- Enhancement: support TileJSON 3.0.0
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mbtileserver.