mapbox / tilelive-mapnik Goto Github PK

mapnik renderer backend for tilelive

License: BSD 3-Clause "New" or "Revised" License

Makefile 0.51% JavaScript 95.36% Batchfile 4.13%

tilelive-mapnik's Introduction

tilelive-mapnik

Renderer backend for tilelive.js that uses node-mapnik to render tiles and grids from a Mapnik XML file. tilelive-mapnik implements the Tilesource API.

Installation

npm install tilelive-mapnik

Though tilelive is not a dependency of tilelive-mapnik you will want to install it to actually make use of tilelive-mapnik through a reasonable API.

Usage

var tilelive = require('tilelive');
require('tilelive-mapnik').registerProtocols(tilelive);

tilelive.load('mapnik:///path/to/file.xml', function(err, source) {
    if (err) throw err;

    // Interface is in XYZ/Google coordinates.
    // Use `y = (1 << z) - 1 - y` to flip TMS coordinates.
    source.getTile(0, 0, 0, function(err, tile, headers) {
        // `err` is an error object when generation failed, otherwise null.
        // `tile` contains the compressed image file as a Buffer
        // `headers` is a hash with HTTP headers for the image.
    });

    // The `.getGrid` is implemented accordingly.
});

Note that grid generation will only work when there's metadata inside a <Parameters> object in the Mapnik XML.

The key fields are interactivity_layer and interactivity_fields. See an example in the tests. These Parameters are normally added by the application that creates the XML, in this case CartoCSS

tilelive-mapnik's People

Contributors

Stargazers

Watchers

tilelive-mapnik's Issues

Add a way to specify tile image format

I couldn't find a way to specify output image format, and MapnikSource.prototype._renderMetatile indeed only seems to be considering 'utf' and 'jpeg' as values, assuming 'png' for anything else.

Allowing extended formats would allow for this: CartoDB/Windshaft#52

Generate a different stylesheet per zoom level

...that only contains styles relevant to that zoom level.

remove map.clear() in pool.destroy()

This map.clear() forcefully requests that mapnik delete the layers and styles inside a mapnik map. This usage is a throwback to when mapnik did not cache references to memory mapped shapefiles leading quickly, particularly on osx, to the hitting of ulimit: too many open files.

We should remove this call. i doubt it ever worked for the purpose at the time and it is unnecessary now as the styles and layers will get properly released once the actual mapnik::Map is released.

Error: Tileset has no interactivity

I followed a test request path and it seems to me as it's doing what documented in tilelive-mapnik README file.

This is what gets passed to tilelive.load(uri, callback) as uri:

{ query: 
   { base: 'windshaft_test:test_table:png::::1341413151605',
     metatile: 1,
     bufferSize: 64 },
  protocol: 'mapnik:',
  slashes: true,
  xml: '...',
  mml: 
   { interactivity: { layer: 'test_table', fields: [Object] },
     format: 'png' } }

But then, in MapnikSource.getGrid, this._info is:

this._info on MapnikSource.getGrid:
{ id: 'Windshaft',
  name: 'Windshaft',
  minzoom: 0,
  maxzoom: 22,
  bounds: [ -180, -85.05112877980659, 180, 85.05112877980659 ],
  center: [ 0, 0, 2 ] }

and the final effect is the Error: Tileset has no interactivity error.

This is with tilelive-mapnik-0.4.1 and tilelive-4.3.0

Unclear licensing

Hello,
I'm packaging tilelive-mapnik for Debian, and the source tree is missing a licensing statement. The only reference can be found in package.json, where there's "BSD" -- but that really means nothing :)

Please provide clear licensing terms somewhere: README.md is fine, a separate LICENSE/COPYING file would be best.

Thank you,
David

Empty grids not interpreted as such by tilelive

Need to update solid key for new UTFGrid key handling in mapnik. Currently blank grids are assigned a negative integer by mapnik but the check in tilelive.js is for the '0' key:

https://github.com/mapbox/tilelive.js/blob/master/lib/copytask.js#L185-L187

Next actions:

Huddle with @springmeyer, confirm whether grid keys are now always integers
Consider normalizing <= 0 to '0' here: https://github.com/mapbox/tilelive-mapnik/blob/master/lib/render.js#L88

add tif output support

Mapnik now has experimental tif output support. tilelive-mapnik, maybe after the node-v6 push, could add support for tifs: mapnik/mapnik#967

Provide a NEWS file

There's no file in the repository containing notable changes between versions, could be useful to have one.

metatile diameter/circumference constants wrong

In render.js the diameter and circumference calculations were accidentally switched (in terms of which should be * by 2 vs PI). Ultimately they are multiplied such the MAX_RES value was correct. The DIAMETER constant used in the calculateMetatile was also correct, but does not actually represent the diameter, but rather the earth's circumference/2, aka the origin shift.

pool is draining and cannot accept work

I suspect there's an issue with pool management.
I'm doing my best to call .close() on a TileLive object only when I'm very sure nobody is going to call any method on it anymore, but still I'm getting the "pool is draining and cannot accept work" exception.

This is UNCONFIRMED so please bear with me while I further debug the issue and consider this only a placeholder.

When metatiling 0/0/0.png is blank

need to look into the cause after changes to support metatiling are fully pushed upstream in tilelive.js

convert tests to mocha

13 test failures with 0.4.2 and mapnik-2.0.0

13 tests fail with the following setup:

[email protected] /home/src/cartodb/tilelive-mapnik
├── [email protected] 
├── [email protected] 
├── [email protected] 
├── [email protected] <--- shrinkwrapped, package.json contains ~0.7.4
├── [email protected] 
├── [email protected] 
└── [email protected]

Here's the full npm test output:

[strk@gnash:/usr/src/cartodb/tilelive-mapnik(rel-0.4.2)] npm test

> [email protected] pretest /home/src/cartodb/tilelive-mapnik
> which expresso || npm install --dev

/home/src/cartodb/tilelive-mapnik/node_modules/.bin/expresso

> [email protected] test /home/src/cartodb/tilelive-mapnik
> which expresso | sh


   uncaught undefined: Error: Unknown child node in 'Map': 'Parameters'

   uncaught undefined: Error: Unknown child node in 'Map': 'Parameters'


   uncaught undefined: Error: Unknown child node in 'Map': 'Parameters'


   uncaught undefined: Error: Unknown child node in 'Map': 'Parameters'


   uncaught undefined: Error: Unknown child node in 'Map': 'Parameters'


   uncaught undefined: AssertionError: false == true
    at /home/src/cartodb/tilelive-mapnik/test/render.test.js:124:16
    at MapnikSource.<anonymous> (/home/src/cartodb/tilelive-mapnik/lib/mapnik_backend.js:41:13)
    at MapnikSource.g (events.js:156:14)
    at MapnikSource.emit (events.js:67:17)
    at Array.4 (/home/src/cartodb/tilelive-mapnik/lib/mapnik_backend.js:101:20)
    at EventEmitter._tickCallback (node.js:190:38)


   grid.test.js getGrid(): AssertionError: {} deepEqual {"grid_0_0_0":true,"grid_1_0_0":true,"grid_1_0_1":true,"grid_1_1_0":true,"grid_1_1_1":true,"grid_2_0_0":true,"grid_2_0_1":true,"
    at /home/src/cartodb/tilelive-mapnik/test/grid.test.js:46:16
    at Test.<anonymous> (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:961:13)
    at Test.emit (events.js:64:17)
    at EventEmitter.<anonymous> (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:892:18)
    at EventEmitter.<anonymous> (events.js:88:20)
    at EventEmitter.emit (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:1053:10)
    at report (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:992:13)
    at EventEmitter.emit (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:1050:9)

   grid.test.js getGrid() with invalid layer: AssertionError: false == true
    at /home/src/cartodb/tilelive-mapnik/test/grid.test.js:85:16
    at Test.<anonymous> (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:961:13)
    at Test.emit (events.js:64:17)
    at EventEmitter.<anonymous> (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:892:18)
    at EventEmitter.<anonymous> (events.js:88:20)
    at EventEmitter.emit (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:1053:10)
    at report (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:992:13)
    at EventEmitter.emit (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:1050:9)


   info.test.js getInfo(): AssertionError: false == true 
    at /home/src/cartodb/tilelive-mapnik/test/info.test.js:31:16
    at Test.<anonymous> (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:961:13)
    at Test.emit (events.js:64:17)
    at EventEmitter.<anonymous> (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:892:18)
    at EventEmitter.<anonymous> (events.js:88:20)
    at EventEmitter.emit (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:1053:10)
    at report (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:992:13)
    at EventEmitter.emit (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:1050:9)


   info.test.js getInfo() with XML string: AssertionError: false == true
    at /home/src/cartodb/tilelive-mapnik/test/info.test.js:67:16
    at Test.<anonymous> (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:961:13)
    at Test.emit (events.js:64:17)
    at EventEmitter.<anonymous> (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:892:18)
    at EventEmitter.<anonymous> (events.js:88:20)
    at EventEmitter.emit (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:1053:10)
    at report (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:992:13)
    at EventEmitter.emit (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:1050:9)


   info.test.js getInfo() with template: AssertionError: false == true
    at /home/src/cartodb/tilelive-mapnik/test/info.test.js:96:16
    at Test.<anonymous> (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:961:13)
    at Test.emit (events.js:64:17)
    at EventEmitter.<anonymous> (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:892:18)
    at EventEmitter.<anonymous> (events.js:88:20)
    at EventEmitter.emit (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:1053:10)
    at report (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:992:13)
    at EventEmitter.emit (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:1050:9)


   render.test.js getTile(): AssertionError: {} deepEqual {"tile_0_0_0":true,"tile_1_0_0":true,"tile_1_0_1":true,"tile_1_1_0":true,"tile_1_1_1":true,"tile_2_0_0":true,"tile_2_0_1":true,"
    at /home/src/cartodb/tilelive-mapnik/test/render.test.js:56:16
    at Test.<anonymous> (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:961:13)
    at Test.emit (events.js:64:17)
    at EventEmitter.<anonymous> (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:892:18)
    at EventEmitter.<anonymous> (events.js:88:20)
    at EventEmitter.emit (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:1053:10)
    at report (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:992:13)
    at EventEmitter.emit (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:1050:9)


   render.test.js getTile() with XML string: AssertionError: {} deepEqual {"tile_0_0_0":true,"tile_1_0_0":true,"tile_1_0_1":true,"tile_1_1_0":true,"tile_1_1_1":true,"tile_2_0_0":true,"tile_2_0_1":true,"
    at /home/src/cartodb/tilelive-mapnik/test/render.test.js:89:16
    at Test.<anonymous> (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:961:13)
    at Test.emit (events.js:64:17)
    at EventEmitter.<anonymous> (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:892:18)
    at EventEmitter.<anonymous> (events.js:88:20)
    at EventEmitter.emit (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:1053:10)
    at report (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:992:13)
    at EventEmitter.emit (/home/src/cartodb/tilelive-mapnik/node_modules/expresso/bin/expresso:1050:9)


   Failures: 13

Mapnik version is 2.0.0, node-mapnik 0.7.5 does support it.

move to map.fromStringSync()

It appears that the async loading of a map from a string is prompting: mapnik/mapnik#951 (comment)

Moving to sync fixes the issue.

I also had to do this to be able to load the BIL format rasters for the FOSS4G shootout - so this is the second case I have encountered were race conditions (in mapnik) can occur do the the async method.

Assigning to myself to research further.

Test JPEG/PDF output

(refactoring branch)

Enhanced feature inspection API

@kkaefer had ideas for this.

Dependents

Updates to MML

Let's coordinate on revamping MML to make tilelive-mapnik breath a little easier. Discussing with @springmeyer and @tmcw it looks like we have two options

Expand the mml format to allow for additional information (with sensible defaults).
Create a new format that "wraps" mml that tilelive-mapnik can use.

My thought is to expand MML to contain the following tilejson keys:

// existing MML keys
srs
Layer
Stylesheet

// tilejson keys
name
description
version
attribution
legend
minzoom
maxzoom
bounds
center
formatter
scheme

We also need to figure out what to do with two key pieces of information

format (png, jpeg, etc.)
interactivity (layer, keyname)

I think format is basically a tilelive-mapnik specific key. interactivity on the other hand I think should make its way back into Mapnik XML -- maybe as a property/flag of the <Layer> structure or so.

flexible tile sizes

The recent metatiling work introduced a few hardcoded 256 sizes. Need to factor these out to make sure we're flexible for tile sizes like 64x64 for mobile tiles.

Remove _mml

Task is to remove the need for the _mml object by moving the various properties used there to Mapnik XML metadata tags.

use a pool of images to reduce re-allocation overhead

For export jobs that attempt to maximize machine resources in order to render a tile cache as fast as possible, the allocation and cleanup of mapnik.Image objects shows up in profiling traces. While image allocation seems like it should be of trivial cost compared to data access and rendering, the sheer number of images allocated (per metatile) + the nature of v8 garbage deferred collection means that we may be able to speed up renders by pooling images and re-using them.

Issues to tackle:

_DONE, YES_: is there a clear benefit to pooling images and re-using images (after clearing their data) to reduce the number of allocations per render?
_DONE, YES_ can both images and mapnik.Grid objects still be treated the same here?
_TODO:_ is there a benefit to making allocation / cleanup / clearing of images operate in the threadpool (be async)?
_TODO:_ is using V8::AdjustAmountOfExternalAllocatedMemory in the main thread creating a bottleneck?
_TODO:_ a reasonable api for releasing, automatically, or via api call, the image pool

image encoding errors go unhandled

image encoding errors that are thrown by the constructor in node-mapnik (invalid args) currently go uncaught.

This is fixed in the image-pool branch: 132e230. Once merged, I'll close this issue. The merge is waiting on #50

lockingcache and socket is not writable

@kkaefer I'm doing some load testing against TileMill, listening on socket instead of port (not sure this is important, though). What I've run into is this error, which kills the tile server. One clue is that if I increase the lifetime of objects in the locking cache to a long time or so, like 1 hour, the error stops occurring.

My load testing is against one map, using siege w/ a urls file in internet mode, so it randomly chooses urls to hit from that file.


node.js:134
        throw e; // process.nextTick error, or 'error' event on first tick
        ^
Error: Socket is not writable
    at Socket._writeOut (net.js:391:11)
    at Socket.write (net.js:377:17)
    at ServerResponse._writeRaw (http.js:392:28)
    at ServerResponse._send (http.js:372:15)
    at ServerResponse.write (http.js:622:16)
    at ServerResponse.end (http.js:682:16)
    at ServerResponse.send (/home/ianward/www/tilemill/node_modules/bones/node_modules/express/lib/response.js:108:8)
    at /home/ianward/www/tilemill/servers/Tile.bones:83:17
    at Function.<anonymous> (/home/ianward/www/tilemill/node_modules/tilelive-mapnik/lib/mapnik.js:313:9)
    at /home/ianward/www/tilemill/node_modules/tilelive-mapnik/lib/lockingcache.js:60:26

Make internal, global cache of MapnikSource objects optional

Ideally we should remove this cache and provide recommendations on why/how to cache MapnikSource objects outside of tilelive-mapnik, which can be important for avoiding the overhead of initialization/mapnik map xml loading. More info at mapbox/tilemill#1893.

However, to ease this transition I think we should make it possible to disable the cache.

Test different render modes

(local XML file, remote XML file, Carto stylesheet, literal string)

Expose MapnikSource cache and clear when we close

Calling applications may want to clear the cache and when the source is closed we should remove its entry from the cache to avoid leaks.

Potential performance research: variable format rendering

JPEG images can be encoded faster and smaller. But they don't support alpha. For rendering scenarios that only need alpha support for a small number of tiles (like at edges of maps of aerial imagery) variable encoding may prove fruitful. The idea would be that tilelive-mapnik could support a new format called auto and behind the scenes it could decide which image format might be optimal based on whether any alpha values less than 255 are encountered in a quick pass over the image pixels.

Fix safe64

.replace() only fixes the first instance.

MapnikSource.prototype.close is calling callback too early

As per https://github.com/coopernurse/node-pool the generic pool's "drain" function is an async call, and the author suggests to use as this:

 pool.drain(function() {
    pool.destroyAllNow();
});

Instead MapnikSource.prototype._close (which is called by MapnikSource.prototype.close)
use it like this:

        this._pool.drain();
        this._pool.destroyAllNow();

Advertise proper support for node 0.8.0

I just tested 0.4.1 with node-0.8.0 and worked fine:

[strk@gnash:/usr/src/cartodb/tilelive-mapnik(rel-0.4.1)] npm test

> [email protected] pretest /home/src/cartodb/tilelive-mapnik
> which expresso || npm install --dev

/home/src/cartodb/tilelive-mapnik/node_modules/.bin/expresso

> [email protected] test /home/src/cartodb/tilelive-mapnik
> which expresso | sh


   100% 13 tests

[strk@gnash:/usr/src/cartodb/tilelive-mapnik(rel-0.4.1)] node --version
v0.8.0

But when trying to list it as a dependency of some other package then npm install complains that it doesn't support node-0.8.0:

npm http GET https://registry.npmjs.org/tilelive-mapnik/0.4.1
npm http 304 https://registry.npmjs.org/tilelive-mapnik/0.4.1
npm ERR! notsup Unsupported
npm ERR! notsup Not compatible with your version of node/npm: [email protected]
npm ERR! notsup Required: {"node":"~0.6.x"}
npm ERR! notsup Actual:   {"npm":"1.1.32","node":"0.8.0"}

I belive the "engines" entry of package.json should be altered to include 0.8.0.
I didn't test with 0.7.x.

Limit total amount of tiles kept in LockingCache

Advertise proper support for node 0.8.0 in 0.3 branch

I'm having troubles in using 0.4.x (see #38) while 0.3.2 works just fine. The only problem with 0.3.2 is it doesn't advertise support for node-0.8 so npm install refuses to install it as a dependency.

npm test gives me 100% 13 tests result with node-0.8.2 so I think it would be safe to advertise support in package.json and publish a 0.3.3, what do you think ?

eio usage breaks in node v0.6


~/projects/tilemill[node-v6]$ ./index.js 

node.js:201
        throw e; // process.nextTick error, or 'error' event on first tick
              ^
TypeError: Expected number of threads
    at Object.<anonymous> (/Users/dane/projects/tilemill/node_modules/tilelive-mapnik/lib/mapnik.js:16:16)
    at Module._compile (module.js:432:26)
    at Object..js (module.js:450:10)
    at Module.load (module.js:351:31)
    at Function._load (module.js:310:12)
    at Module.require (module.js:357:17)
    at require (module.js:368:17)
    at Object.<anonymous> (/Users/dane/projects/tilemill/index.js:9:1)
    at Module._compile (module.js:432:26)
    at Object..js (module.js:450:10)

skipblank memory usage and main loop blocking

I'm starting to be more convinced that the skipblank branch uses memory too aggressively. Partially related, it also blocks the main loop, leading to the eio threadpool being underutilized.

E.g. I am unable to detect any memory leaks at the v8/node/mapnik level (e.g. or deeper problems). Even if v8 garbage collection is forced during skipping (using while (!mapnik.gc()) {} it is still very easy to run out of memory when many tiles are being skipped.

This is a ticket to discuss possible ways to work around this.

Ideas:

move parts of code to c++ (threadpool)
move existing c++ calls to async (isSolid, getPixel, others ??)
more agressive use of process.nextTick (to help reduce blocking)
use child process nodes to handle key tasks

Tests for lib/resolve.js

Need tests for resolve.js.

remove call to map.zoomAll() during stylesheet load

I can see any reason we'd want the map.zoomAll() usage in place, and especially not the silencing of valid errors:

https://github.com/mapbox/tilelive-mapnik/blob/master/lib/mapnik.js#L208-214 in place.

If errors are happening at this stage and should not be fatal this is not the right way to fix things. Rather fixes should be made to mapnik upstream if things are throwing that should not.

I will take a look at removing unless someone speaks up about the usefulness of this.

Support passing parameters to LockingCache

Without doing this, you can't use tilelive-mapnik for anything faster than minutely updates (which is what I'm trying to do)

key solidCache on image format

The image solidCache in tilelive-mapnik uses a rgba string for keys. This works fine as long as the image format does not change. Because the solidCache is global, and there is no way for calling applications to clear it, an application like TileMill may request png tiles but still get back jpeg encoded data for images already in the cache.

The solution I think is to also key by image format. Or expose an API to clear the cache.

Entries in tiles cache released too early

I've been debugging a big slowdown for 2 weeks now and finally found the culprit being the GC hitting hard after a couple hundred requests to Windshaft.

But Windshaft is using a single TileLive object, backed by TileLive-mapnik, so I tought it was using the internal cache.

Well, the thing is that as soon as a tile is added the the LockingCache, all queued callbacks for that entry are invoked and then the cache entry is removed ! See LockingCache.prototype.trigger, calling this.del(id).

I don't understand why the code does this, especially since each cache item is attached a Timer which expires in 60 seconds in order to clear the entry. So why clearing it immediately after the call rather than on its "natural" expiration ?

I would drop the quick deletion in the trigger function and eventually expose the timeout to the caller.
If you agree I'll produce a pull request for this.

More info (with numbers): CartoDB/Windshaft#32

Error: No compatible version found: mapnik@'>=0.7.1- <0.8.0-'

The latest tag (v0.4.0) fails on the "npm install" step with the following error:

Error: No compatible version found: mapnik@'>=0.7.1- <0.8.0-'
["0.1.0","0.1.1","0.1.2","0.2.3","0.2.5","0.2.6","0.2.7","0.2.8","0.2.9","0.2.10","0.2.11","0.2.12","0.2.13","0.3.0","0.3.1","0.4.0","0.4.1","0.5.0","0.5.1","0.5.2","0.5.3","0.5.4","0.5.6","0.5.7","0.5.8","0.5.9","0.5.10","0.5.5","0.5.11","0.5.12","0.5.13","0.5.14","0.5.15","0.5.16","0.5.17"]

Error reporting for invalid input

Invalid/unreadable mml file
Carto fails to parse
Pass through of node-mapnik error

lib/render.js gets EARTH_DIAMETER wrong

var EARTH_RADIUS = 6378137;
var EARTH_DIAMETER = EARTH_RADIUS * Math.PI; <---- WRONG, it's EARTH_RADIUS * 2
var EARTH_CIRCUMFERENCE = EARTH_DIAMETER * 2; <--- WRONG, it's EARTH_DIAMETER * Math.PI
var MAX_RES = EARTH_CIRCUMFERENCE / 256;

The EARTH_CIRCUMFERENCE value is correct anyway but EARTH_DIAMETER is not.
Later on, ``EARTH_DIAMETERis used for further calculations (incalculateMetatile`) which may or may not be correct.

Assuming all calculus are correct I think it's still worth fixing the names of the variables, to reduce confusion.

Move (back) to synchronous map loading

Due to mapnik/mapnik#1459 it is unsafe to load maps asynchronously.

We should no longer default to this behavior, to avoid possible crashes. Normal usage in TileMill usually is not impacted by this problem (though I've seen it frequently when running from the command line) - likely because other moving parts slow down map initialization and lead to the singleton being initialized in time. But, this is not safe going forward. Minor optimizations or other changes like #49 that would lead to more maps being loading more quickly can immediately trigger crashes.

Build failed: -> task failed (err #1):

npm install tilelive-mapnik on Ubuntu 10.04


npm ERR! error installing [email protected] Error: [email protected] install: `node-waf -v configure build`
npm ERR! error installing [email protected] `sh "-c" "node-waf -v configure build"` failed with 1
npm ERR! error installing [email protected]     at ChildProcess.<anonymous> (/root/local/node/lib/node_modules/npm/lib/utils/exec.js:49:20)
npm ERR! error installing [email protected]     at ChildProcess.emit (events.js:67:17)
npm ERR! error installing [email protected]     at ChildProcess.onexit (child_process.js:192:12)
npm ERR! error installing [email protected] Error: [email protected] install: `node-waf -v configure build`
npm ERR! error installing [email protected] `sh "-c" "node-waf -v configure build"` failed with 1
npm ERR! error installing [email protected]     at ChildProcess.<anonymous> (/root/local/node/lib/node_modules/npm/lib/utils/exec.js:49:20)
npm ERR! error installing [email protected]     at ChildProcess.emit (events.js:67:17)
npm ERR! error installing [email protected]     at ChildProcess.onexit (child_process.js:192:12)
npm ERR! [email protected] install: `node-waf -v configure build`
npm ERR! `sh "-c" "node-waf -v configure build"` failed with 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the mapnik package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-waf -v configure build
npm ERR! You can get their info via:
npm ERR!     npm owner ls mapnik
npm ERR! There is likely additional logging output above.
npm ERR! 
npm ERR! System Linux 2.6.18-274.el5.028stab093.2PAE
npm ERR! command "node" "/root/local/node/bin/npm" "install" "tilelive-mapnik"
npm ERR! cwd /usr/lib/python2.6
npm ERR! node -v v0.4.9
npm ERR! npm -v 1.0.30
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /usr/lib/python2.6/npm-debug.log
npm not ok

contents of npm-debug.log (i cleared it before running install)

https://gist.github.com/e817b9e757c43c470ca5

test failures on master


~/projects/tilelive-mapnik[master]$ npm test

> [email protected] pretest /Users/dane/projects/tilelive-mapnik
> which expresso || npm install --dev

/Users/dane/projects/tilelive-mapnik/node_modules/.bin/expresso

> [email protected] test /Users/dane/projects/tilelive-mapnik
> which expresso | sh


   uncaught: AssertionError: {"name":"world","id":"world","minzoom":0,"maxzoom":22,"center":[0,4.317819745709997,2],"bounds":[-180,-79.11799791776475,180,87.75363740918475]} deepEqual {"name":"world","minzoom":0,"maxzoom":22,"id":"world","bounds":[0,0,-0.000008983152841195214,-0.000008983152840993819],"center":[-0.000004491576420597607,-0.000004491576420496909,2]}
    at /Users/dane/projects/tilelive-mapnik/test/info.test.js:45:20
    at MapnikSource.getInfo (/Users/dane/projects/tilelive-mapnik/lib/mapnik.js:340:21)
    at /Users/dane/projects/tilelive-mapnik/test/info.test.js:42:16
    at MapnikSource.<anonymous> (/Users/dane/projects/tilelive-mapnik/lib/mapnik.js:39:13)
    at MapnikSource.g (events.js:143:14)
    at MapnikSource.emit (events.js:67:17)
    at Function.<anonymous> (/Users/dane/projects/tilelive-mapnik/lib/mapnik.js:125:16)
    at next (/Users/dane/projects/tilelive-mapnik/node_modules/step/lib/step.js:51:23)
    at /Users/dane/projects/tilelive-mapnik/lib/mapnik.js:259:9
    at adjustCallback (/Users/dane/projects/tilelive-mapnik/node_modules/generic-pool/lib/generic-pool.js:187:7)


   uncaught: AssertionError: {"name":"test","id":"test","minzoom":0,"maxzoom":22,"center":[1.054687500000007,29.53522956294847,2],"bounds":[-180,-79.11799791776475,180,87.75363740918475]} deepEqual {"name":"test","minzoom":0,"maxzoom":22,"id":"test","center":[1.054687500000007,29.53522956294847,2],"bounds":[0,0,-0.000008983152841195214,-0.000008983152840993819]}
    at /Users/dane/projects/tilelive-mapnik/test/info.test.js:69:20
    at MapnikSource.getInfo (/Users/dane/projects/tilelive-mapnik/lib/mapnik.js:340:21)
    at /Users/dane/projects/tilelive-mapnik/test/info.test.js:66:16
    at MapnikSource.<anonymous> (/Users/dane/projects/tilelive-mapnik/lib/mapnik.js:39:13)
    at MapnikSource.g (events.js:143:14)
    at MapnikSource.emit (events.js:81:20)
    at Function.<anonymous> (/Users/dane/projects/tilelive-mapnik/lib/mapnik.js:125:16)
    at next (/Users/dane/projects/tilelive-mapnik/node_modules/step/lib/step.js:51:23)
    at /Users/dane/projects/tilelive-mapnik/lib/mapnik.js:259:9
    at adjustCallback (/Users/dane/projects/tilelive-mapnik/node_modules/generic-pool/lib/generic-pool.js:187:7)


   uncaught: AssertionError: {"name":"world","id":"world","minzoom":0,"maxzoom":22,"center":[0,4.317819745709997,2],"bounds":[-180,-79.11799791776475,180,87.75363740918475]} deepEqual {"name":"world","minzoom":0,"maxzoom":22,"id":"world","bounds":[0,0,-0.000008983152841195214,-0.000008983152840993819],"center":[-0.000004491576420597607,-0.000004491576420496909,2]}
    at /Users/dane/projects/tilelive-mapnik/test/info.test.js:14:20
    at MapnikSource.getInfo (/Users/dane/projects/tilelive-mapnik/lib/mapnik.js:340:21)
    at /Users/dane/projects/tilelive-mapnik/test/info.test.js:11:16
    at MapnikSource.<anonymous> (/Users/dane/projects/tilelive-mapnik/lib/mapnik.js:39:13)
    at MapnikSource.g (events.js:143:14)
    at MapnikSource.emit (events.js:81:20)
    at Function.<anonymous> (/Users/dane/projects/tilelive-mapnik/lib/mapnik.js:125:16)
    at next (/Users/dane/projects/tilelive-mapnik/node_modules/step/lib/step.js:51:23)
    at /Users/dane/projects/tilelive-mapnik/lib/mapnik.js:259:9
    at adjustCallback (/Users/dane/projects/tilelive-mapnik/node_modules/generic-pool/lib/generic-pool.js:187:7)


   uncaught: Error: Images not equal(96.2956): /tmp/5235849495511502    /Users/dane/projects/tilelive-mapnik/test/fixture/tiles/transparent_2_2_2.png
    at /Users/dane/projects/tilelive-mapnik/test/support/assert.js:26:27
    at ChildProcess.exithandler (child_process.js:100:7)
    at ChildProcess.emit (events.js:67:17)
    at Socket.<anonymous> (child_process.js:172:12)
    at Socket.emit (events.js:64:17)
    at Array.<anonymous> (net.js:831:12)
    at EventEmitter._tickCallback (node.js:126:26)


   uncaught: Error: Images not equal(102.316): /tmp/3448255138937384    /Users/dane/projects/tilelive-mapnik/test/fixture/tiles/transparent_2_0_1.png
    at /Users/dane/projects/tilelive-mapnik/test/support/assert.js:26:27
    at ChildProcess.exithandler (child_process.js:100:7)
    at ChildProcess.emit (events.js:67:17)
    at Socket.<anonymous> (child_process.js:172:12)
    at Socket.emit (events.js:64:17)
    at Array.<anonymous> (net.js:831:12)
    at EventEmitter._tickCallback (node.js:126:26)


   uncaught: Error: Images not equal(95.3265): /tmp/2778729395940900    /Users/dane/projects/tilelive-mapnik/test/fixture/tiles/transparent_2_3_2.png
    at /Users/dane/projects/tilelive-mapnik/test/support/assert.js:26:27
    at ChildProcess.exithandler (child_process.js:100:7)
    at ChildProcess.emit (events.js:67:17)
    at Socket.<anonymous> (child_process.js:172:12)
    at Socket.emit (events.js:64:17)
    at Array.<anonymous> (net.js:831:12)
    at EventEmitter._tickCallback (node.js:126:26)


   uncaught: Error: Images not equal(99.3059): /tmp/7789509759750217    /Users/dane/projects/tilelive-mapnik/test/fixture/tiles/transparent_2_3_1.png
    at /Users/dane/projects/tilelive-mapnik/test/support/assert.js:26:27
    at ChildProcess.exithandler (child_process.js:100:7)
    at ChildProcess.emit (events.js:67:17)
    at Socket.<anonymous> (child_process.js:172:12)
    at Socket.emit (events.js:64:17)
    at Array.<anonymous> (net.js:831:12)
    at EventEmitter._tickCallback (node.js:126:26)


   uncaught: Error: Images not equal(90.0117): /tmp/3670433207880705.5    /Users/dane/projects/tilelive-mapnik/test/fixture/tiles/transparent_2_2_1.png
    at /Users/dane/projects/tilelive-mapnik/test/support/assert.js:26:27
    at ChildProcess.exithandler (child_process.js:100:7)
    at ChildProcess.emit (events.js:67:17)
    at Socket.<anonymous> (child_process.js:172:12)
    at Socket.emit (events.js:64:17)
    at Array.<anonymous> (net.js:831:12)
    at EventEmitter._tickCallback (node.js:126:26)


   uncaught: Error: Images not equal(90.275): /tmp/990615754853934    /Users/dane/projects/tilelive-mapnik/test/fixture/tiles/transparent_2_1_1.png
    at /Users/dane/projects/tilelive-mapnik/test/support/assert.js:26:27
    at ChildProcess.exithandler (child_process.js:100:7)
    at ChildProcess.emit (events.js:67:17)
    at Socket.<anonymous> (child_process.js:172:12)
    at Socket.emit (events.js:64:17)
    at Array.<anonymous> (net.js:831:12)
    at EventEmitter._tickCallback (node.js:126:26)


   render.test.js getTile(): AssertionError: {"tile_0_0_0":true,"tile_1_0_0":true,"tile_1_0_1":true,"tile_1_1_0":true,"tile_1_1_1":true,"tile_2_0_0":true,"tile_2_0_1":true,"tile_2_0_2":true,"tile_2_0_3":true,"tile_2_1_0":true,"tile_2_1_1":true,"tile_2_1_2":true,"tile_2_1_3":true,"tile_2_2_0":true,"tile_2_2_1":true,"tile_2_2_2":true,"tile_2_2_3":true,"tile_2_3_0":true,"tile_2_3_1":true,"tile_2_3_2":true,"tile_2_3_3":true} deepEqual {}
    at /Users/dane/projects/tilelive-mapnik/test/render.test.js:56:16
    at EventEmitter.<anonymous> (/Users/dane/projects/tilelive-mapnik/node_modules/expresso/bin/expresso:847:37)
    at EventEmitter.<anonymous> (events.js:81:20)
    at EventEmitter.emit (/Users/dane/projects/tilelive-mapnik/node_modules/expresso/bin/expresso:925:10)
    at report (/Users/dane/projects/tilelive-mapnik/node_modules/expresso/bin/expresso:871:13)
    at EventEmitter.emit (/Users/dane/projects/tilelive-mapnik/node_modules/expresso/bin/expresso:922:9)


   Failures: 10


npm ERR! [email protected] test: `which expresso | sh`
npm ERR! `sh "-c" "which expresso | sh"` failed with 10
npm ERR! 
npm ERR! Failed at the [email protected] test script.
npm ERR! This is most likely a problem with the tilelive-mapnik package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     which expresso | sh
npm ERR! You can get their info via:
npm ERR!     npm owner ls tilelive-mapnik
npm ERR! There is likely additional logging output above.
npm ERR! 
npm ERR! System Darwin 11.3.0
npm ERR! command "node" "/usr/local/bin/npm" "test"
npm ERR! cwd /Users/dane/projects/tilelive-mapnik
npm ERR! node -v v0.4.12
npm ERR! npm -v 1.0.106
npm ERR! code ELIFECYCLE
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /Users/dane/projects/tilelive-mapnik/npm-debug.log
npm not ok

Add metatiling support

(refactoring branch)

Allow cache purging to be configurable in LockingCache

As per this discussion: 9b8f7ac#commitcomment-563034 it may be useful in a live tiling situation to be able to require the LockingCache to retain cached tiles until they are timed out.

discard in memory map string after loading?

The tilelive map object carries the entire XML string in memory to then pass to mapnik via node-mapnik's fromString.

With large maps they can end up taking many MB of memory when loaded by mapnik so perhaps we could save a bit on memory by then discarding the XML string.

Generate formatter from mml file

(refactoring branch)

Broken UTF grid encoding when caches exist

See CartoDB/Windshaft-cartodb#67 and particularly
this comment:
CartoDB/Windshaft-cartodb#67 (comment)

I haven't tried with latest version because it changes the way interactivity is specified, but I saw that when a cache is present and the "image" is "solid" the cache is still returned, no matter if field names changed since last call (field names are not part of the cache key maybe ?)

Tests broken

@kkaefer @springmeyer might know better, but tests are currently broken here, possibly just because of progress in mapnik - images no longer match the test fixtures.

mapbox / tilelive-mapnik Goto Github PK

tilelive-mapnik's Introduction

tilelive-mapnik

Installation

Usage

tilelive-mapnik's People

Contributors

Stargazers

Watchers

Forkers

tilelive-mapnik's Issues

Dependents

Recommend Projects

Recommend Topics

Recommend Org