Giter VIP home page Giter VIP logo

tileserver's Introduction

Macrostrat's tile server

A dynamic tile server for geologic maps

Version 2 changes

  • Python-based instead of Node-based
  • A "dynamic tiler" based heavily on TimVT
  • Uses a PostgreSQL caching backend (this is the primary simplification)
  • Uses Mapnik for legacy image-tile generation
  • Optionally, can use Varnish as a "L2" API cache

Installing

This module depends on tile utilitiesUW-Macrostrat/postgis-tile-utils. This dependency is packaged as a git submodule. To install it, run:

git submodule update --init

The module can be run locally using Poetry, but there may be problems with Mapnik. We're working on simplifying this process and making Mapnik optional.

To install in Docker (preferred), build the image:

docker build -t macrostrat/tileserver .

Running the tile server

Then run it with the appropriate environment variables and port bindings:

docker run macrostrat/tileserver
-e POSTGRES_DB=postgresql://user:[email protected]:5432
-p 8000:8000

To serve tile layers, the fixtures (housed in macrostrat_tileserver/fixtures) must be created on the database. There is a bundled tileserver CLI that will create the layers. In Docker:

docker run macrostrat/tileserver
-e POSTGRES_DB=postgresql://user:[email protected]:5432
tileserver create-fixtures

Or in the running docker container:

docker exec tileserver create-fixtures

Accessing tiles

Once the tileserver is running, you should be able to access docs:

curl localhost:8000/docs

And tiles:

curl localhost:8000//{z}/{x}/{y}

Macrostrat core layers:

Defining new layers

  • New layers can be defined using SQL or PL/PGSQL functions.
  • Currently, layers must be initialized by editing the macrostrat_tileserver/main.py file to add the appropriate initialization function. This will be improved in the future.

tileserver's People

Contributors

cambro avatar cannonlock avatar davenquinn avatar jczaplew avatar springmeyer avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

lance-weaver jjhi11 cannonlock

tileserver's Issues

Speed up CI builds

It's absolutely ridiculous for images like this one to take 30 minutes to build, for several reasons:

  • It keeps you waiting a long time for during debugging (important especially when you are not running the code locally)
  • It requires you to keep an eye on completion which makes moving on to other tasks harder
  • Coupled with a lack of auto-deployment, it means that several touch points are needed to complete a deployment task

This is all bad for productivity. And this won't be the only repository with a somewhat complex dependency chain.

I've taken the first step to improve the situation, by implementing layer caching in the Docker build process. We should also investigate auto-deployment and self-hosted runners which can run on our (quite powerful) infrastructure.

Create a Dockerized version that doesn't build Mapnik

The production version of Macrostrat's tileserver builds Mapnik to allow the creation of image tiles from Macrostrat's vector layers. Mapnik is heavy software that is difficult to compile (e.g., it requires boost which must be built from source). It breaks relatively easily and can be considered a "legacy" capability.

A simpler installation of the tileserver could be created by omitting this dependency (the Python code should gracefully degrade thanks to this commit). Maybe we can create a Dockerfile.minimal and go from there?

I suspect that having this minimal build will be helpful for testing down the line as well.

Break management utilities into separate package

This package consists both of the tileserver itself, which runs based on a GIS software stack, and a few management commands that have a much thinner dependency stack. We'd like these management commands to be accessible from the macrostrat command-line application. However, the GIS software stack is rather weighty and might conflict with dependencies needed for other Macrostrat activities. We should potentially break this into multiple sub-packages to allow more flexibility.

Paleogeography tile layers

Paleogeography will be a compelling new feature of Macrostrat v2.

The tileserver is key to supporting visualization of these new capabilities (UW-Macrostrat/web#101). To get there, we need to solve several issues around performance and caching. This is a tracking issue to organize these features.

Raster tile server responds badly to non-existent tile

It appears that when we request a raster tile that is outside the bounds of the queried raster image, we return a 502 status code, when it should be a 404 (probably)?

https://tileserver.development.svc.macrostrat.org/cog/tiles/10/193/388.png?url=https://s3.macrostrat.chtc.io/map-datastore/Provo30x60_OFR-586DM.tif

This wreaks havoc with certain clients, especially the Mapboard app, which goes into a cascade of request retries.

We should also store raster bounding boxes to do a better job of tiling together imagery. We may be getting 502s because the tileserver only knows that it can't read the requested file, not that it actually doesn't exist.

Basic metric reporting

Create a page to display basic stats:

  • Number of hits
  • Number of hits by app
  • Percent of cache hits (disk and redis)
  • Number of requests by zoom level

Report all of the above by

  • Past 24hrs
  • Last week
  • Last month
  • Last year
  • All time

Split caching database from data source

There is no special reason that the PostgreSQL tile cache needs to be in the same database as the primary Macrostrat database. We should make it configurable to a separate database, which will help with load balancing. We could also consider using a separate database connection for cache reads and writes for extra performance.

Logging

Make sure all routes use the logger and that it works as expected

Write tests

  • Make sure all layers function as expected
  • Make sure proper protocol buffers and pngs are returned
  • Make sure cache is warmed
  • Make sure logging works as expected
  • Validate functioning of caches
  • Validate functioning of headers

Incorrect cache function signature for raster tiles

tileserver/macrostrat_tileserver/image_tiles/core.py

Line 83 in 7737813

content, timer, cache_hit=True, media_type="image/png"

Raster image tiles still use an old signature for TileResponse that leads to a 500 error for Carto/Carto-Slim image layers (e.g., https://tileserver.macrostrat-dev.chtc.io/carto/3/3/4.png). This should be an easy adjustment.

Traceback:

INFO:     128.105.82.82:0 - "GET /carto/2/0/1.png HTTP/1.0" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 373, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 271, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 118, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.9/site-packages/starlette_cramjam/middleware.py", line 112, in __call__
    await responder(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette_cramjam/middleware.py", line 142, in __call__
    await self.app(scope, receive, self.send_with_compression)
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 706, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
  File "/app/./macrostrat_tileserver/image_tiles/__init__.py", line 34, in tile
    return await image_tiler.handle_tile_request(request, background_tasks, tile)
  File "/app/./macrostrat_tileserver/image_tiles/core.py", line 94, in handle_tile_request
    return TileResponse(content, timer, cache_hit=False, media_type="image/png")
  File "/app/./macrostrat_tileserver/utils.py", line 37, in TileResponse
    return Response(content, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'cache_hit'
INFO:     128.105.82.82:0 - "GET /carto/2/2/1.png HTTP/1.0" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 373, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 271, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 118, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.9/site-packages/starlette_cramjam/middleware.py", line 112, in __call__
    await responder(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette_cramjam/middleware.py", line 142, in __call__
    await self.app(scope, receive, self.send_with_compression)
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 706, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
  File "/app/./macrostrat_tileserver/image_tiles/__init__.py", line 34, in tile
    return await image_tiler.handle_tile_request(request, background_tasks, tile)
  File "/app/./macrostrat_tileserver/image_tiles/core.py", line 94, in handle_tile_request
    return TileResponse(content, timer, cache_hit=False, media_type="image/png")
  File "/app/./macrostrat_tileserver/utils.py", line 37, in TileResponse
    return Response(content, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'cache_hit'

Add a health check endpoint

In order to support Kubernetes liveness probes we need a stable way to evaluate service health. @brianaydemir is working on the infrastructure side of this.

The ideal approach would be /status or /healthcheck endpoint that returns 200 if the service is live. I'm not sure — should this hit the database? Perhaps it could return a count of cached tiles for different layers, if we wanted the endpoint to double as a "useful" view. But maybe it's better just to successfully return with no extra database work, especially if we intend this to be hit often.

Undefined table error 

asyncpg.exceptions.UndefinedTableError: relation "tile_layers.map_lines" does not exist
Resetting connection with an active transaction <buildpg.asyncpg.BuildPgConnection object at 0x7f075d1df820>
Query: SELECT tile_layers.map($1, $2, $3, $4::text::json), Params: [46, 93, 8, '{"source_id": "279"}']
INFO:     128.105.82.82:0 - "GET /map/8/46/93?source_id=279 HTTP/1.0" 500 Internal Server Error
ERROR:    Exception in ASGI application

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.