Giter VIP home page Giter VIP logo

hastile's Introduction

hastile

A Haskell tile server that produces GeoJSON or MVT (Mapbox Vector Tiles) from a PostGIS database.

Build Status

Getting Started

Running

  • ./hastile starter --dbConnection "host=localhost port=5432 user=dba password=password dbname=mapdata" --port 8080 --host "http://localhost" --cfgFile hastile-config.json

Building

All

Install Haskell Stack:

Linux

  • sudo apt-get install libgmp-dev postgresql gdal-bin postgis libgeos-dev libpq-dev

MacOS

  • brew install gmp postgresql gdal postgis geos libpq

Build

Point the server at a PostgreSQL (9.6+) database with the PostGIS extension enabled. It will automatically render tables with a "wkb_geometry" column and serve them as layers.

For example:

  • stack build
  • stack exec -- hastile starter --dbConnection "host=localhost port=5432 user=dba password=password dbname=mapdata" --port 8080 --host "http://localhost" --cfgFile hastile-config.json

Tiles will be available at: http://localhost:8080/table_name/z/x/y.mvt

A configuration file will be created automatically although not used (see running in Server mode below).

RESTful API

GET    /                          (application/json)                   - Returns the current configuration.
POST   /                          (application/json)                   - Add/overwite many layer configurations.
POST   /layername                 (application/json)                   - Add/overwrite a single layer configuration.
GET    /layername[.json]          (application/json)                   - Return TileJSON for a tile layer.
GET    /layername/Z/X/Y.<mvt|pbf> (application/vnd.mapbox-vector-tile) - Return Mapnik Vector Tile for given layername, Zoom, (X,Y).
GET    /layername/Z/X/Y.json      (application/json)                   - Return GeoJSON for given layername, Zoom, (X,Y).
GET    /token                     (application/json)                   - Returns tokens and authorised layers.
GET    /token/tokenid             (application/json)                   - Returns the authorised layers for the given token.
POST   /token                     (application/json)                   - A token and authorised layers to upsert the token database.
DELETE /token/tokenid             (application/json)                   - Delete the given token from the token database.

Layer API

The POST / with multiple layer configuration or POST /layername with a layer configuration allows you to change the layers that Hastile serves up and will save the configuration file to disk.

To create a new layer:

  • curl -d '{ "layer_name": { "table-name": "...", "format": "geojson", "quantize": 2, "simplify": {} } }' -H "Content-Type: application/json" -X POST http://localhost:8080/

To modify an existing layer:

  • curl -d '{ "table-name": "...", "format": "geojson", "quantize": 2, "simplify": {} }' -H "Content-Type: application/json" -X POST http://localhost:8080/layer_name

Payload

Example:

    "rail_network": {
      "security": "public",
      "table-name": "suburbanrail_hastile",
      "format": "wkb-properties",
      "minzoom": 0,
      "maxzoom": 20,
      "bounds": [ 144.94932174682617, -37.82449737924566, 144.97901916503906, -37.84788365473107 ],
      "format": "wkb-properties",
      "last-modified": "2018-10-31 09:09:39 +1000",
      "quantize": 4,
      "simplify": {
        "12": "douglas-peucker"
      }
    }
Attributes
layer-name
string
The unique key to the layer configuration

Layer Configuration

Attributes
security
string
"public" - allow anyone to request
"private" - require a valid token (using ?token=xxxx).
table-name
string
The unique key to the layer.
format
string
"source" - a normal table
"wkb-properties" - combination of JSON hash and wkb_geometry columns
"geojson" - combination of GeoJSON, JSON hash of properties and wkb geomtry columns.
last-modified
date
Last time the layer has been update, used to return the Last-Modified HTTP header.
quantize
integer
Positive integer, amount to round and remove duplicates (10 is probably the most, 1 is typical).
simplify
map
A map of zoom levels to simplification settings.
minzoom
int
0..30 Minimum zoom level to render, otherwise returns 404. Adds setting to TileJSON.
maxzoom
int
0..30 Maximum zoom level to render, otherwise returns 404. Adds setting to TileJSON.
bounds
array float
Bounds are represented in WGS:84 latitude and longitude values. Adds settings to TileJSON.

Layer Simplification Settings

Attributes
zoom-level
string
The zoom level (>=) to apply the layer simplification algorithm.

Layer Simplification Algorithm

Attributes
algorithm-name
string
"douglas-peucker" - apply the Ramer-Douglas-Peucker algorithm (epsilon of 1.0).

Building

Building:

  • stack build
  • stack test

Server Mode

Server mode makes more features available:

  • Token based security (private layers),
  • Metrics (prometheus)

Token Based Security

A token table is required for token security. This is required for "private" layers.

Setup the Token Database

  • Create a postgres database to store the tokens table: createdb -O dba db_name If you don't have the createdb utility then use the migration tool :
    ./db/migration createdb db_name dba
  • Initialize the DB :
    ./db/migration init "postgresql://db_user:password@db_server:db_port/db_name"
  • Run the migrations :
    ./db/migration migrate "postgresql://db_user:password@db_server:db_port/db_name"
  • Use the migration tool for migrations :
    ./db/migration --help

Token API

To insert or update a token:

  • curl -d '{ "token": "abcd", "layers": ["layer1", "layer2"] }' -H "Content-Type: application/json" -X POST http://localhost:8080/token

To delete a token:

  • curl -H "Content-Type: application/json" -X DELETE http://localhost:8080/token/abcd

Running in Server Mode

To run Hastile in server mode you must use a configuration file:

  • stack exec -- hastile server --configFile hastile-config.json

Configuration

The file contains settings for the database connection and layer configuration, for example:

{
  "db-connection": "host=example.com port=5432 user=tiler password=123abc dbname=notoracle"
  "layers": {
    "layer1": { 
      "table_name": "layer1_table",
      "format": "wkb-properties",
      "last-modified": "2017-01-15T23:49:36Z"
    },
    "layer2": {
      "table_name": "layer2_table",
      "format": "geojson",
      "last-modified": "2017-01-15T23:49:36Z"
    }
  }
}

Where, db-connection is a PostgreSQL connection string.

To construct a table with a GeoJSON feature with all properties containing arbitrary columns from a table, create a materialized view like:

CREATE MATERIALIZED VIEW layer1_table as SELECT jsonb_build_object(
    'type',      'Feature',
    'id',         ogc_fid,
    'geometry',   ST_AsGeoJSON(wkb_geometry)::jsonb,
    'properties', to_jsonb(row) - 'ogc_fid' - 'wkb_geometry'
)::json as geojson, (to_jsonb(row) - 'wkb_geometry') :: JSON as properties,row.wkb_geometry as wkb_geometry FROM (SELECT * FROM source_layer1_table) row;

This will create the two columns required: geojson (a GeoJSON feature in JSON format) and the geometry column.

You can configure other database, mapnik and HTTP port settings too:

{
  "db-pool-size": 10,
  "db-timeout": 5,
  "port": 8080
}

If you want to combine multiple tables into a single layer you can use UNION and MATERIALIZED VIEWS and then query it directly:

CREATE MATERIALIZED VIEW layers AS
  SELECT geojson FROM layer1_table
  UNION
  SELECT geojson FROM layer2_table

Changing the configuration to:

  "layers": {
    "layer": {
      ...
    }  
  }

Running

To start the server: ./hastile server --configFile FILEPATH

To run with GHC Metrics: ./hastile server --configFile FILEPATH +RTS -T

Documentation

Hastile docs are generated using mkdocs.

Preview docs locally by using mkdocs serve or mkdocs build. Then deploy to https://indicatrix.github.io/hastile using mkdocs gh-deploy. This command will deploy to the gh-pages branch as outlined here.

Projections

We assume tiles are requested in the spherical mercator (EPSG 3857 AKA EPSG 900913 AKA Webby McWebcator). Furthermore, map data is assumed to be stored in EPSG 4326.

Helpful links

hastile's People

Contributors

ajmcmiddlin avatar dendrei avatar jamiecook avatar ljnicol avatar newmana avatar ozmat avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

hastile's Issues

Let cabal build mapnik-vector-tile FFI

Automate this so users don't have to care. Might need users to point us to a checkout of mapnik-vector-tile, although maybe we should just do that as part of the setup.

Source Layer type

Based on #63 we need to:

Create a layer type than just uses a SQL Table with the following properties:

  • ogc_fid OR id column,
  • wkb_geometry column (geometry type),
  • Numeric types: integer, bigint, and real/float4,
  • Character types: varchar.

The task is really to take this table and produce a (hs-geojson) GeoJSON that zellige can use.

Cluster geojson point data

Build into serving point data - automatically create clusters of points. For example, at a high zoom level you see 10 points, zoomed out they cluster to 1 point with the number 10 on it.

Handle SIGTERM gracefully

Currently have no idea what happens when we SIGTERM. Given we have a database connection pool and config updates in play, we should make sure we exit cleanly on SIGTERM.

Add buffer to bounding box for features in a tile

Features being rendered at the edge of a tile get clipped when icon-allow-overlap is enabled. To overcome the clipping, features at the edge of a tile boundary need to be rendered by the tiles on either side of the boundary. This way, when the marker is clipped when rendered on one tile, the other tile is also aware of the feature and renders the part that appears in its tile.

To achieve this, a buffer is added to vector tiles, so features not strictly in a tile, but near the boundary, are known and rendered by both tiles. It looks like mapnik has a default buffer of 128px for its vector tiles, but hastile does no buffering in its query, so the features aren't present in the data to be included in the tile buffer.

Some relevant issues:

Empty layers find tables

If the configuration doesn't have a layers configuration or it's empty then search the database for tables with wkb_geometry columns. Create an in memory configuration that represents using the sensible defaults in #64

From #63

Load configuration via HTTP

Instead of being on a local file system - load from remote location.

Maybe have it get a URI - either file:// or http://.

A file would still accept POSTs from the HTTP API, whereas HTTP would be read only and POSTs would be an error.

Stop leaking information

This has been a bit of a POC up until now, but if hastile is going to get use we need to clean up a few things.

  • Turn config route off, or at least put behind a debug flag
  • Stop returning detailed errors in HTTP responses
    • Log detailed info and return a generic, user-friendly error

Add logging

Need some insight into what hastile is doing when running - especially when we start cleaning up error reponses (see #35)

Command line startup

Currently, we can only startup the server with a configuration file. Add an alternative where we can give the minimum set of parameters for starting hastile via command line with no configuration file:

  • Host and Port
  • Database connection

For example:
hastile --host http://localhost --port 8080 --db-connection "host=localhost port=5432 user=user password=password dbname=database"

Make Mapnik working

This needs investigation. Tilesplash writes some Geo JSON containing the features it pulls from PostGIS, and Mapnik picks that up and produces a tile.

I currently have no idea how to Mapnik, or how to hs-mapnik. However, my brief scanning of the hs-mapnik code suggests we'll probably need to add MVT support to the bindings as a first step.

Create a last modified hash and control via API

At the moment, to update the last modified you restart the server. This updates the Last-Modified for all layers. Instead - record the Last-Modified and POST to an API with a new date (ISO 8601 string).

For example:
POST /zone
{
'Last-Modified': '2017-01-15T23:49:36Z'
}

At the moment it could probably write back to the config file (and also be defined in the config file to override the initial setting of since the server started).

Get layer and query from user config

Users need to be able to configure the layers and queries associated with those layers. In addition, we need to allow users to specify where the bounding box should appear in their query.

Tilesplash does this by escaping variables in queries. bbox4326 in Tilesplash is what we want, so we'll make it compatible for now and replace !bbox4326! with the SQL for the bounding box.

Hastile failing to render tiles

Error message:

libc++abi.dylib: terminating with uncaught exception of type mapnik::datasource_exception: geojson_datasource: Failed parse GeoJSON file from in-memory string
libc++abi.dylib: [1]    1711 abort      stack exec -- hastile --configFile ../sitewisely-web/config/hastile.config

Use correct MIME type for `.mvt`

The spec says that the return type of an mvt should be application/vnd.mapbox-vector-tile.

Currently this is not a trivial change given that .json and .mvt share a route, however I think we should aim to comply with the spec.

Sensible Defaults for Configuration

Based on #63 we need to:

Have a JSON configuration file to override the "sensible defaults".

The "sensible defaults" for Layer Configuration are:

  • Table Name - the name of the layer - the table in the database to use.
  • Security - public (instead of the current default private)
  • Format - source - use only a single table for the source data (no materialised view)
  • Last Modified - when the server starts
  • Quantize - 1
  • No Simplification.

For Security

  • If all layers are public layers don't look for the tokens table (or try and cache it).
  • Token API would also not work.

Keep or remove SQL for layer definition

We moved to storing GeoJSON feature and geometry only per table.

If we're going to commit to having the user store their data like this, it seems like we could just go the whole hog and get rid of the query config altogether. The onus is then on the user to present a table/view that has geojson and wkb_geometry columns, and just give us the name of the relation.

Possible negatives:

  • Prevent adding other columns and filtering - many layers from one table,
  • Prevent query strings from being used as dynamic filters - WHERE foo = ?foo - from a URL http:..123.mvt?foo=something
  • Prevent token based security - joining on entry in query string.

All of these things could be implemented other ways too but would have to consider this in the design/solution.

Add JSON config file

We're going to need to configure the layers and queries to use, so stick 'em in JSON for now. Hopefully Aeson will make that easy.

Upgrade to 8.0.2

This is already on a branch - just putting it here in case there's a problem somewhere (like CI)

Split out SphericalMercator to library?

Before realising a lot of the translating between SRSs could be offloaded to PostGIS, I wrote some projection code to deal with EPSG3857. Might as well track that somewhere - but it doesn't belong in hastile. hastile might depend on that library for the stuff it needs though.

Memory Leak

I think we can confirm this now.

syslog.1:May 23 07:37:22 ip-172-31-30-125 kernel: [2514035.054075] [26637] 0 26637 2097114 1880989 3942 11 0 0 hastile
syslog.1:May 23 07:37:22 ip-172-31-30-125 kernel: [2514035.054129] Out of memory: Kill process 26637 (hastile) score 894 or sacrifice child
syslog.1:May 23 07:37:22 ip-172-31-30-125 kernel: [2514035.059148] Killed process 26637 (hastile) total-vm:8388456kB, anon-rss:7522816kB, file-rss:1120kB
s

Snapshot of two weeks running hastile:
https://scoutapp.com/dashboards/share/CQ6EjnD5vlK9h5aQNXdF5w

Serve tileJSON for layers

Rather than using the list of tile urls in the styles, Hastile could generate a tileJSON, compliant with here.

This would mean we could handle bounds, min and max zoom, attribution etc directly from Hastile, rather than having to add this data later.

Add min/max zoom and bounds for TileJSON

Add producing for both Starter and Server for tilejson production:

  • bounds,
    • Depends on the layer format:
      • source - query whole table,
      • geojson and wkb-properties - given by configuration.
  • min and max zoom
    • 1..20 for starter, and optional config for server.

Add layer capture to routes

Routes should be layer specific, only returning values when a layer of the same name has been configured.

Zoom dependant queries

Change the configuration file to create a pair of settings:

  • Query
  • Zoom level

The idea is given a zoom level call a specific query.

For example:

    "layer1": { 
      "queries": {
        "query": {
          "zoom": 1,
          "sql": "SELECT ST_AsGeoJSON(wkb_geometry), hstore(layer1_table)-'wkb_geometry'::text FROM layer1_table WHERE ST_Intersects(wkb_geometry, !bbox_4326!)"
        },
        "query": {
          "zoom": 7,
          "sql": "SELECT ST_AsGeoJSON(wkb_geometry), hstore(layer1_table)-'wkb_geometry'::text FROM layer1_table WHERE ST_Intersects(wkb_geometry, !bbox_4326!)"
        }
      }
      "last-modified": "2017-01-15T23:49:36Z"
    },

Request Queue

Create a queue that when a configurable size is reached it rejects requests with a 503.

If connection is broken have Hastile stop trying to render the tile

Configure Hastile to store cached tiles.

Instead of using just using the web to cache tiles from the client side, have Hastile handle it's own tile sets and serve them from there. The caches can include:

  • S3 Bucket
  • Local hard drive.

It must allow invalidation of individual tiles.

Use the format described here:
https://www.azavea.com/blog/2018/09/06/tilegarden-serverless-tile-rendering-with-aws-lambda/

https://medium.com/@alexrolek/the-serverless-vector-map-stack-lives-22835d341d7d
https://medium.com/@mojodna/tapalcatl-2-updates-8eb827d969c1
https://medium.com/@mojodna/tapalcatl-cloud-optimized-tile-archives-1db8d4577d92

OSM meta tiles:

Do input validation on request parameters

We currently don't do any input validation on request parameters. For example, we could check and provide nice errors when any of the following aren't met:

  • z, x, y are all natural
  • z is below some max zoom level
  • x and y are within range given zoom level
  • layer has configuration
  • sql is good and DB query is successful (depends on Hasql error reporting)

Finish Polygon Testing and Functionality

  • Test winding order for polygons.
  • Clipped polygons - if clipped still return.
  • Lines must be at least two points.
  • Empty geometry tests.
  • Get new polygon clipping working or remove.

Create a pure haskell MVT rendering

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.