Giter VIP home page Giter VIP logo

node-mbtiles's Introduction

mbtiles

Node.js utilities and tilelive integration for the MBTiles format.

Build Status Build status

Installation

npm install @mapbox/mbtiles
var MBTiles = require('@mapbox/mbtiles');

API

Constructor

All MBTiles instances need to be constructed before any of the methods become available. NOTE: All methods described below assume you've taken this step.

new MBTiles('./path/to/file.mbtiles?mode={ro, rw, rwc}', function(err, mbtiles) {
  console.log(mbtiles) // mbtiles object with methods listed below
});

The mode query parameter is a opening flag of mbtiles. It is optional, default as rwc. Available flags are:

  • ro: readonly mode, will throw error if the mbtiles does not exist.
  • rw: read and write mode, will throw error if the mbtiles does not exist.
  • rwc: read, write and create mode, will create a new mbtiles if the mbtiles does not exist.

Reading

getTile(z, x, y, callback)

Get an individual tile from the MBTiles table. This can be a raster or gzipped vector tile. Also returns headers that are important for serving over HTTP.

mbtiles.getTile(z, x, y, function(err, data, headers) {
  // `data` is your gzipped buffer - use zlib to gunzip or inflate
});

getInfo(callback)

Get info of an MBTiles file, which is stored in the metadata table. Includes information like zoom levels, bounds, vector_layers, that were created during generation. This performs fallback queries if certain keys like bounds, minzoom, or maxzoom have not been provided.

mbtiles.getInfo(function(err, info) {
  console.log(info); // info
});

getGrid(z, x, y, callback)

Get a UTFGrid tile from the MBTiles table.

mbtiles.getGrid(z, x, y, function(err, data) {
  // continue onwards
});

Writing

startWriting AND stopWriting

In order to write a new (or currently existing) MBTiles file you need to "start" and "stop" writing. First, construct the MBTiles object.

mbtiles.startWriting(function(err) {
  // start writing with mbtiles methods (putTile, putInfo, etc)
  mbtiles.stopWriting(function(err) {
    // stop writing to your mbtiles object
  });
});

putTile(z, x, y, buffer, callback)

Add a new tile buffer to a specific ZXY. This can be a raster tile or a gzipped vector tile (we suggest using require('zlib') to gzip your tiles).

var zlib = require('zlib');

zlib.gzip(fs.readFileSync('./path/to/file.mvt'), function(err, buffer) {
  mbtiles.putTile(0, 0, 0, buffer, function(err) {
    // continue onward
  });
});

putInfo(data, callback)

Put an information object into the metadata table. Any nested JSON will be stringified and stored in the "json" row of the metadata table. This will replace any matching key/value fields in the table.

var exampleInfo = {
  "name": "hello-world",
  "description":"the world in vector tiles",
  "format":"pbf",
  "version": 2,
  "minzoom": 0,
  "maxzoom": 4,
  "center": "0,0,1",
  "bounds": "-180.000000,-85.051129,180.000000,85.051129",
  "type": "overlay",
  "json": `{"vector_layers": [ { "id": "${layername}", "description": "", "minzoom": 0, "maxzoom": 4, "fields": {} } ] }`
};

mbtiles.putInfo(exampleInfo, function(err) {
  // continue onward
});

putGrid(z, x, y, grid, callback)

Inserts a UTFGrid tile into the MBTiles store. Grids are in JSON format.

var fs = require('fs');
var grid = JSON.parse(fs.readFileSync('./path/to/grid.json', 'utf8'));
mbtiles.putGrid(0, 0, 0, grid, function(err) {
  // continue onward
});

Hook up to tilelive

When working at scale, node-mbtiles is meant to be used within a Tilelive ecosystem. For example, you could set up an MBTiles file as a "source" and an S3 destination as a "sink" (using tilelive-s3). Assuming you have a system set up with an mbtiles:// protocol that points to a specific file and authorized to write to the s3 bucket:

var tilelive = require('@mapbox/tilelive');
var MBTiles = require('@mapbox/mbtiles');
var s3 = require('@mapbox/tilelive-s3');

s3.registerProtocols(tilelive);
MBTiles.registerProtocols(tilelive);

var sourceUri = 'mbtiles:///User/hello/path/to/file.mbtiles';
var sinkUri = 's3://my-bucket/tiles/{z}/{x}/{y}';

// load the mbtiles source
tilelive.load(sourceUri, function(err, src) {
  // load the s3 sink
  tilelive.load(sinkUri, function(err, dest) {

    var options = {}; // prepare options for tilelive copy
    options.listScheme = src.createZXYStream(); // create ZXY stream from mbtiles

    // now copy all tiles to the destination
    tilelive.copy(src, dst, options, function(err) {
      console.log('tiles are now on s3!');
    });
  });
});

Test

npm test

node-mbtiles's People

Contributors

apendleton avatar dependabot[bot] avatar dnomadb avatar flippmoke avatar hannesj avatar ianshward avatar jfirebaugh avatar jingsam avatar kaibot3000 avatar kkaefer avatar mapsam avatar mattficke avatar nyurik avatar rclark avatar sbma44 avatar springmeyer avatar suhasdeshpande avatar tmcw avatar vicb avatar willwhite avatar wrynearson avatar yhahn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

node-mbtiles's Issues

unable to install on ubuntu

npm ERR! error installing [email protected] Error: No compatible version found: sqlite3@'>=2.0.17- <2.1.0-' npm ERR! error installing [email protected] Valid install targets: npm ERR! error installing [email protected] ["2.0.0","2.0.1","2.0.2","2.0.3","2.0.4","2.0.5","2.0.6","2.0.7","2.0.8","2.0.9","2.0.10","2.0.11","2.0.12","2.0.13","2.0.14","2.0.15","2.1.0","2.1.1"] npm ERR! error installing [email protected] at installTargetsError (/usr/lib/node_modules/npm/lib/cache.js:424:10) npm ERR! error installing [email protected] at /usr/lib/node_modules/npm/lib/cache.js:406:17 npm ERR! error installing [email protected] at saved (/usr/lib/node_modules/npm/lib/utils/npm-registry-client/get.js:136:7) npm ERR! error installing [email protected] at Object.cb [as oncomplete] (/usr/lib/node_modules/npm/node_modules/graceful-fs/graceful-fs.js:36:9) npm ERR! Error: No compatible version found: sqlite3@'>=2.0.17- <2.1.0-' npm ERR! Valid install targets: npm ERR! ["2.0.0","2.0.1","2.0.2","2.0.3","2.0.4","2.0.5","2.0.6","2.0.7","2.0.8","2.0.9","2.0.10","2.0.11","2.0.12","2.0.13","2.0.14","2.0.15","2.1.0","2.1.1"] npm ERR! at installTargetsError (/usr/lib/node_modules/npm/lib/cache.js:424:10) npm ERR! at /usr/lib/node_modules/npm/lib/cache.js:406:17 npm ERR! at saved (/usr/lib/node_modules/npm/lib/utils/npm-registry-client/get.js:136:7) npm ERR! at Object.cb [as oncomplete] (/usr/lib/node_modules/npm/node_modules/graceful-fs/graceful-fs.js:36:9) npm ERR! Report this *entire* log at: npm ERR! npm ERR! or email it to: npm ERR! [email protected] npm ERR! npm ERR! System Linux 3.0.0-12-generic npm ERR! command "node" "/usr/bin/npm" "install" "-g" "mbtiles" npm ERR! cwd /home/ubuntu/Downloads npm ERR! node -v v0.6.2 npm ERR! npm -v 1.0.106 npm ERR! npm ERR! Additional logging details can be found in: npm ERR! /home/ubuntu/Downloads/npm-debug.log npm not ok

Promises API

Hello!

Any chances to introduce a promise-compatible API?

mbcheck Metadata: Cannot call method 'toString' of null

Hi, I'm getting this kind of error with mbtiles file generated by tilemill:

root@swamp:/mnt/tmp# mbcheck vys_myto_m_volby.mbtiles
Metadata

/usr/lib/node_modules/mbtiles/node_modules/step/lib/step.js:39
throw arguments[0];
^
TypeError: Cannot call method 'toString' of null
at /usr/lib/node_modules/mbtiles/lib/utils.js:10:29
at Array.map (native)
at /usr/lib/node_modules/mbtiles/lib/utils.js:8:44
at Array.map (native)
at Object.utils.table (/usr/lib/node_modules/mbtiles/lib/utils.js:7:29)
at Function. (/usr/lib/node_modules/mbtiles/bin/mbcheck:52:15)
at next (/usr/lib/node_modules/mbtiles/node_modules/step/lib/step.js:51:23)
at next.parallel (/usr/lib/node_modules/mbtiles/node_modules/step/lib/step.js:83:14)
at Statement.MBTiles._metadata (/usr/lib/node_modules/mbtiles/lib/mbtiles.js:253:25)

It looks like there's no field[] array. Since this error emerged mbpipe doesn't work too as it generates mbtiles with missing tiles.

Not compatible to [email protected] ?

Just tried to install [email protected] and got a message that it is not compatible to [email protected], install log is as following:

root@scale:/home/mab# npm list
/home/mab
โ””โ”€โ”€ [email protected]

root@scale:/home/mab# npm install mbtiles
npm http GET https://registry.npmjs.org/mbtiles
npm http 304 https://registry.npmjs.org/mbtiles
npm http GET https://registry.npmjs.org/optimist
npm http GET https://registry.npmjs.org/sqlite3
npm http GET https://registry.npmjs.org/step
npm http GET https://registry.npmjs.org/sphericalmercator
npm http GET https://registry.npmjs.org/underscore
npm http GET https://registry.npmjs.org/zlib
npm http 304 https://registry.npmjs.org/sqlite3
npm http 304 https://registry.npmjs.org/optimist
npm http 304 https://registry.npmjs.org/step
npm http 304 https://registry.npmjs.org/underscore
npm http 304 https://registry.npmjs.org/sphericalmercator
npm ERR! error installing [email protected]
npm http GET https://registry.npmjs.org/sphericalmercator/-/sphericalmercator-1.0.1.tgz
npm http GET https://registry.npmjs.org/optimist/-/optimist-0.2.8.tgz
npm http GET https://registry.npmjs.org/underscore/-/underscore-1.1.7.tgz

npm ERR! Error: No compatible version found: sqlite3@'>=2.0.17- <2.1.0-'
npm ERR! Valid install targets:
npm ERR! ["2.0.0","2.0.1","2.0.2","2.0.3","2.0.4","2.0.5","2.0.6","2.0.7","2.0.8","2.0.9","2.0.10","2.0.11","2.0.12","2.0.13","2.0.14","2.0.15","2.1.0","2.1.1"]
npm ERR! at installTargetsError (/usr/local/lib/node_modules/npm/lib/cache.js:488:10)
npm ERR! at next_ (/usr/local/lib/node_modules/npm/lib/cache.js:438:17)
npm ERR! at next (/usr/local/lib/node_modules/npm/lib/cache.js:415:44)
npm ERR! at /usr/local/lib/node_modules/npm/lib/cache.js:408:5
npm ERR! at saved (/usr/local/lib/node_modules/npm/lib/utils/npm-registry-client/get.js:147:7)
npm ERR! at Object.oncomplete (/usr/local/lib/node_modules/npm/node_modules/graceful-fs/graceful-fs.js:231:7)
npm ERR! You may report this log at:
npm ERR! http://github.com/isaacs/npm/issues
npm ERR! or email it to:
npm ERR! [email protected]
npm ERR!
npm ERR! System Linux 2.6.32-5-amd64
npm ERR! command "node" "/usr/local/bin/npm" "install" "mbtiles"
npm ERR! cwd /home/mab
npm ERR! node -v v0.6.11
npm ERR! npm -v 1.1.2
npm ERR! message No compatible version found: sqlite3@'>=2.0.17- <2.1.0-'
npm ERR! message Valid install targets:
npm ERR! message ["2.0.0","2.0.1","2.0.2","2.0.3","2.0.4","2.0.5","2.0.6","2.0.7","2.0.8","2.0.9","2.0.10","2.0.11","2.0.12","2.0.13","2.0.14","2.0.15","2.1.0","2.1.1"]
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! /home/mab/npm-debug.log
npm not ok

non-tilelive

can this be used to access mbtiles outside of tilelive? As it looks like it could in theory, but i've been having no luck attempting to figure out how.

test failures on master

I'm seeing a recursive test failure on master (with node v0.4.12):

> [email protected] test /Users/dane/projects/node-mbtiles
> expresso


   uncaught undefined: Error: EMFILE, Too many open files '/Users/dane/projects/node-mbtiles/test/fixtures/grids/'
    at Object.readdirSync (fs.js:376:18)
    at Statement.<anonymous> (/Users/dane/projects/node-mbtiles/test/write_grids.test.js:23:16)
--> in Database#run('PRAGMA synchronous=OFF', [Function])
    at MBTiles.startWriting (/Users/dane/projects/node-mbtiles/lib/mbtiles.js:399:18)
    at /Users/dane/projects/node-mbtiles/test/write_grids.test.js:19:17
    at Function.<anonymous> (/Users/dane/projects/node-mbtiles/lib/mbtiles.js:68:9)
    at next (/Users/dane/projects/node-mbtiles/node_modules/step/lib/step.js:51:23)


   uncaught undefined: Error: EMFILE, Too many open files '/Users/dane/projects/node-mbtiles/test/fixtures/images/plain_1_7_13_4.png'


   uncaught undefined: Error: EMFILE, Too many open files '/Users/dane/projects/node-mbtiles/test/fixtures/images/plain_1_7_14_4.png'


   uncaught undefined: Error: EMFILE, Too many open files '/Users/dane/projects/node-mbtiles/test/fixtures/images/plain_1_7_15_4.png'


   uncaught undefined: Error: EMFILE, Too many open files '/Users/dane/projects/node-mbtiles/test/fixtures/images/plain_1_7_1_3.png'


   uncaught undefined: Error: EMFILE, Too many open files '/Users/dane/projects/node-mbtiles/test/fixtures/images/plain_1_7_2_3.png'

Support for node.js 4.2.1?

Hello,

The npm install mbtiles is failing with node.js v4.2.1. (Actually it's the sqlite package that is failing). When will the new node version be supported?

stopWriting not reentrant?

I'm trying to move tiles from a source to a sink in 'parallel' and flushing to disk as soon as each is ready for committing, but getting errors while doing so.

The tilelive API says that start- and stopWriting should be reentrant.

// This function must be reentrant: Write mode may only be ended after the
// same number of calls to .stopWriting(). Use a counter to keep track of
// how often write mode was started.

But I don't think this is the case with the mbtiles implementation - it appears to only use a flag rather than a counter

mbtiles._isWritable = false;

I've mocked up a little script to demonstrate the problem - see https://gist.github.com/gravitystorm/9691382 - I get the following:

andy@eiger:~/temp/mbtiles-reentrant$ node index.js 

/home/andy/temp/mbtiles-reentrant/index.js:40
          if (err) throw err;
                         ^
Error: MBTiles not in write mode
    at MBTiles.putTile (/home/andy/temp/mbtiles-reentrant/node_modules/mbtiles/lib/mbtiles.js:454:44)
    at null._onTimeout (/home/andy/temp/mbtiles-reentrant/index.js:39:14)
    at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)

Let me know if this is a bug, or a misunderstanding on my side!

mbpipe bug?

I wanted to create some post-processing on a map using mbpipe: Running this command:

mbpipe 'convert -size 256x256 xc:white png:- -alpha Off -compose Copy_Opacity -composite -monochrome -alpha On png:-' layer.mbtiles

The status meter ran pretty fast for 18mb file. It declared 'Pipe complete' but it seemingly made no change to the layer.

Error: write EPIPE at errnoException

Via support:

I am trying to reduce the size of my mbtiles file. Via MBPipe I run
mbpipe 'advpng -z -4' file.mbtiles On a 8 MB file this works as expected, but as soon as the mbtiles file exceed something about 30 MB the process fails with.

Error: write EPIPE at errnoException (net.js:776:11) at Object.afterWrite (net.js:594:19)

Any ideas on this?

listTiles

It would be nice to have an API which listed zxy tiles in the MBTiles, without that you can't know which specific zxy values to pass to getTile without querying the whole globe.

Vector mbtiles schema

Hello.
When a user makes .mbtiles by the tippecanoe tool, it creates just one table - tiles.
But I can't stream any data into this file because of the putTile method uses other tables (map and images), and also can't create a new file with this specification by mbtiles library.
I guess we need to wait for 2.0 spec?

Indexing in mbtiles

Hi!

I am using a tilelive server to serve tiles for my mapbox gl front end. I am not sure if this is the right place to ask, but is it possible to insert custom indexes in my mbtiles, for faster access? Or are there any other methods for faster access of features in mbtiles?

A little background on what I am trying to do :
I am trying to have a filter on my mapbox gl side, and since I have too many features, it is not very efficient.

Thanks! Any help will be very much appreciated.

MBTiles, TMS, and Flipping Y

Hi, I'm trying to display MBTiles, served with tilelive+node-mbtiles, using modestmaps.js but result appears to be mixed up vertically.

I read somewhere that MBTiles uses TMS spec which requires 'flipping' y coordinate before doing the math.

Is node-mbtiles already handling the flipping internally or is it expected to be done by the caller?

Add tests with bad input

Test for

  • invalid schema
  • invalid data
  • unreadable locations
  • locked files
  • non-existant files
  • incomplete files

Silent crash on large putTiles statement

I'm investigating the limitations of this mbtiles package.

When I add 640,000 tiles using mbtiles.putTile, the command understandably takes a long time, but it then exits (after 16 minutes) with no error.

I've tried outputting node-sqlite3 sql log using the debug on trace callback, and the last thing output in my cli is COMMIT, then sits blankly for most of that 16 min, suggesting to me that perhaps the sqlite3 binary might be doing something.

install issue, am I doing something wrong?

[tedx@ ~]$ sudo npm install @mapbox/mbtiles
npm ERR! git clone [email protected]:mapbox/mbtiles Initialized empty Git repository in /root/.npm/_git-remotes/git-github-com-mapbox-mbtiles-44905b0c/
npm ERR! git clone [email protected]:mapbox/mbtiles
npm ERR! git clone [email protected]:mapbox/mbtiles FIPS integrity verification test failed.
npm ERR! git clone [email protected]:mapbox/mbtiles Permission denied (publickey).
npm ERR! git clone [email protected]:mapbox/mbtiles fatal: The remote end hung up unexpectedly
npm ERR! addLocal Could not install mapbox/mbtiles
npm ERR! Error: ENOENT, stat 'mapbox/mbtiles'
npm ERR! If you need help, you may report this log at:
npm ERR! http://github.com/isaacs/npm/issues
npm ERR! or email it to:
npm ERR! [email protected]

npm ERR! System Linux 2.6.32-642.6.2.el6.x86_64
npm ERR! command "node" "/usr/bin/npm" "install" "@mapbox/mbtiles"
npm ERR! cwd /home/tedx
npm ERR! node -v v0.10.48
npm ERR! npm -v 1.3.6
npm ERR! path mapbox/mbtiles
npm ERR! code ENOENT
npm ERR! errno 34
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! /home/tedx/npm-debug.log
npm ERR! not ok code 0

Update bin/*

Looks like some API changes broke the utils in bin/*.

Is it possible to extract MBTiles using a POLY file?

Hi and thanks for this project. It's extremely helpful and saves tones of time so there's no need to generate MBTiles by regions, and it can be done by higher levels, and then we can cut them to generate smaller MBTiles. To do this, I'm using a bounding box, and the following query:

tilelive-copy --minzoom=0 --maxzoom=14 --bounds="5.9559,45.818,10.4921,47.8084" planet.mbtiles my-extract-for-switzerland.mbtiles

This is already super helpful and it is super great. However, I'm wondering if it's possible to use POLY files, instead of a bounding box, to cut the planet.mbtiles (or whatever mbtiles file)?

This would allow creating even smaller MBTiles. At the moment, the only possibility I see to do this is to use the POLY file, with osmosis (for example), to generate a smaller osm.pbf file, and then use that file to create the MBTiles. However, the process to generate MBTiles it's slow and cutting the MBTiles using the above (bounding boxed) is super fast. If it would be possible to use a POLY file instead, it would be just perfect.

Thanks in advance!

[mbcompact] error : <MBTiles> has no method '_insert'

Hello,

First I must say, that I am a total noob regarding node or js so my problem may be stupid. Here it is :

I have installed node-mbtiles (branch node-v6) and I have the following error when using the mbcompact tool :

-------------------------------------------------- TRACE -----------------------------
> ./mbcompact test.mbtiles

The "sys" module is now called "util". It should have a similar interface.
00 -------------- 50 -------------- 100

/home/ubuntu/src/node-mbtiles/node_modules/step/lib/step.js:39
        throw arguments[0];
                       ^
TypeError: Object #<MBTiles> has no method '_insert'
    at Function.<anonymous> (/home/ubuntu/src/node-mbtiles/bin/mbcompact:95:37)
    at next (/home/ubuntu/src/node-mbtiles/node_modules/step/lib/step.js:51:23)
    at Step (/home/ubuntu/src/node-mbtiles/node_modules/step/lib/step.js:122:3)
    at Statement.<anonymous> (/home/ubuntu/src/node-mbtiles/bin/mbcompact:93:21)
--------------------------------------------------

Can you help me please ?

Thanks!
Alex

Install fails on node 10.14.2

$ n
node/10.14.2
Steves-MacBook-Pro-3 ~$ npm install @mapbox/mbtiles
npm ERR! asyncWrite is not a function
npm ERR! pna.nextTick is not a function

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/stevebennett/.npm/_logs/2018-12-18T21_40_16_075Z-debug.log

This is just a heads-up. Installing on node 10.0.0 worked.

More info:

2 info using [email protected]
3 info using [email protected]
4 verbose npm-session 0f98ffcb0835d2b9
5 silly install loadCurrentTree
6 silly install readLocalPackageData
7 verbose stack TypeError: asyncWrite is not a function
7 verbose stack     at onwrite (/usr/local/lib/node_modules/npm/node_modules/readable-stream/lib/_stream_writable.js:480:7)
7 verbose stack     at WritableState.onwrite (/usr/local/lib/node_modules/npm/node_modules/readable-stream/lib/_stream_writable.js:180:5)
7 verbose stack     at WriteStream.to [as _worker] (/usr/local/lib/node_modules/npm/node_modules/pacote/node_modules/make-fetch-happen/cache.js:154:13)
7 verbose stack     at WriteStream._write (/usr/local/lib/node_modules/npm/node_modules/mississippi/node_modules/flush-write-stream/index.js:35:13)
7 verbose stack     at doWrite (/usr/local/lib/node_modules/npm/node_modules/readable-stream/lib/_stream_writable.js:428:64)
7 verbose stack     at writeOrBuffer (/usr/local/lib/node_modules/npm/node_modules/readable-stream/lib/_stream_writable.js:417:5)
7 verbose stack     at WriteStream.Writable.write (/usr/local/lib/node_modules/npm/node_modules/readable-stream/lib/_stream_writable.js:334:11)
7 verbose stack     at WriteStream.to [as _worker] (/usr/local/lib/node_modules/npm/node_modules/pacote/node_modules/make-fetch-happen/cache.js:171:25)
7 verbose stack     at WriteStream._write (/usr/local/lib/node_modules/npm/node_modules/mississippi/node_modules/flush-write-stream/index.js:35:13)
7 verbose stack     at doWrite (/usr/local/lib/node_modules/npm/node_modules/readable-stream/lib/_stream_writable.js:428:64)
7 verbose stack     at writeOrBuffer (/usr/local/lib/node_modules/npm/node_modules/readable-stream/lib/_stream_writable.js:417:5)
7 verbose stack     at WriteStream.Writable.write (/usr/local/lib/node_modules/npm/node_modules/readable-stream/lib/_stream_writable.js:334:11)
7 verbose stack     at WriteStream.to [as _worker] (/usr/local/lib/node_modules/npm/node_modules/pacote/node_modules/make-fetch-happen/cache.js:182:19)
7 verbose stack     at WriteStream._write (/usr/local/lib/node_modules/npm/node_modules/mississippi/node_modules/flush-write-stream/index.js:35:13)
7 verbose stack     at doWrite (/usr/local/lib/node_modules/npm/node_modules/readable-stream/lib/_stream_writable.js:428:64)
7 verbose stack     at writeOrBuffer (/usr/local/lib/node_modules/npm/node_modules/readable-stream/lib/_stream_writable.js:417:5)
8 verbose cwd /Users/stevebennett/[...]
9 verbose Darwin 16.6.0

insert slow

Why is batchSize set to 10000 but the insertion is very slow and nearly 100 seconds to complete
thanks

README?

This readme needs ... something. Let's do it. At least some examples of how to get info/tiles and plug into tilelive.

cc @mapbox/core-tech @yuletide

Issue while running using tilelive-copy and and MBTiles

Hi,

Previously, I had a machine and I didn't have any troubles with node-mbtiles, however, now that I'm trying in a new machine I'm not getting through it successfully.

I've installed tilelive and node-mbtiles by running:
sudo npm install tilelive
npm install @mapbox/mbtiles

This is the output of both commands:

tilelive

sudo npm install tilelive
npm WARN deprecated [email protected]: This module has moved: please install @mapbox/tilelive instead
npm WARN deprecated [email protected]: This module is now under the @mapbox namespace: install @mapbox/sphericalmercator instead
npm WARN deprecated [email protected]: renamed to d3-queue
npm WARN saveError ENOENT: no such file or directory, open '/home/carlos/workspace/osm-pbf-generator/package.json'
npm WARN enoent ENOENT: no such file or directory, open '/home/carlos/workspace/osm-pbf-generator/package.json'
npm WARN osm-pbf-generator No description
npm WARN osm-pbf-generator No repository field.
npm WARN osm-pbf-generator No README data
npm WARN osm-pbf-generator No license field.

+ [email protected]
added 13 packages from 20 contributors and audited 861 packages in 1.401s
found 0 vulnerabilities

mapbox/mbtiles

npm install @mapbox/mbtiles

> [email protected] install /home/carlos/workspace/osm-pbf-generator/node_modules/sqlite3
> node-pre-gyp install --fallback-to-build

node-pre-gyp WARN Using request for node-pre-gyp https download 
[sqlite3] Success: "/home/carlos/workspace/osm-pbf-generator/node_modules/sqlite3/lib/binding/node-v57-linux-x64/node_sqlite3.node" is installed via remote
npm WARN saveError ENOENT: no such file or directory, open '/home/carlos/workspace/osm-pbf-generator/package.json'
npm WARN enoent ENOENT: no such file or directory, open '/home/carlos/workspace/osm-pbf-generator/package.json'
npm WARN osm-pbf-generator No description
npm WARN osm-pbf-generator No repository field.
npm WARN osm-pbf-generator No README data
npm WARN osm-pbf-generator No license field.

+ @mapbox/[email protected]
added 117 packages from 113 contributors and audited 164 packages in 2.508s
found 0 vulnerabilities

I understand that the warnings aren't an issue here. So I've continued to use an script it worked fine in my previous setup. It's this one:

tilelive-copy --minzoom=0 --maxzoom=14 --bounds="$(perl poly2bb.pl countries/ES/348981.poly)" countries/ES/ES.mbtiles countries/ES/348981.mbtiles

And it throws an error which sounds like tilelive isn't aware of the recently installed node-mbtiles package:

/usr/local/lib/node_modules/tilelive/bin/tilelive-copy:100
        if (err) throw err;
                 ^

Error: Invalid tilesource protocol: mbtiles:
    at Object.tilelive.load (/usr/local/lib/node_modules/tilelive/lib/tilelive.js:95:25)
    at /usr/local/lib/node_modules/tilelive/lib/tilelive.js:336:18
    at pop (/usr/local/lib/node_modules/tilelive/node_modules/queue-async/queue.js:24:14)
    at Object.defer (/usr/local/lib/node_modules/tilelive/node_modules/queue-async/queue.js:55:11)
    at Object.tilelive.copy (/usr/local/lib/node_modules/tilelive/lib/tilelive.js:334:36)
    at copy (/usr/local/lib/node_modules/tilelive/bin/tilelive-copy:99:14)
    at Object.<anonymous> (/usr/local/lib/node_modules/tilelive/bin/tilelive-copy:69:1)
    at Module._compile (module.js:653:30)
    at Object.Module._extensions..js (module.js:664:10)
    at Module.load (module.js:566:32)

Could anyone help? I'm definitely lost and haven't found information on how to fix this.

Update mbcheck

Currently fails with:

willwhite ~/Inbox: mbcheck test.mbtiles 
Metadata
name         
type         
description  
version      
format       
bounds       

/Users/diggersf/.nvm/v0.4.9/lib/node_modules/mbtiles/node_modules/step/lib/step.js:38
        throw arguments[0];
                       ^
TypeError: Cannot call method 'prepare' of undefined
    at Function.<anonymous> (/Users/diggersf/.nvm/v0.4.9/lib/node_modules/mbtiles/bin/mbcheck:67:40)
    at next (/Users/diggersf/.nvm/v0.4.9/lib/node_modules/mbtiles/node_modules/step/lib/step.js:51:23)
    at Step (/Users/diggersf/.nvm/v0.4.9/lib/node_modules/mbtiles/node_modules/step/lib/step.js:124:3)
    at Function.<anonymous> (/Users/diggersf/.nvm/v0.4.9/lib/node_modules/mbtiles/bin/mbcheck:64:9)
    at next (/Users/diggersf/.nvm/v0.4.9/lib/node_modules/mbtiles/node_modules/step/lib/step.js:51:23)
    at Function.<anonymous> (/Users/diggersf/.nvm/v0.4.9/lib/node_modules/mbtiles/bin/mbcheck:60:9)
    at next (/Users/diggersf/.nvm/v0.4.9/lib/node_modules/mbtiles/node_modules/step/lib/step.js:51:23)
    at next (/Users/diggersf/.nvm/v0.4.9/lib/node_modules/mbtiles/node_modules/step/lib/step.js:54:7)
    at next (/Users/diggersf/.nvm/v0.4.9/lib/node_modules/mbtiles/node_modules/step/lib/step.js:54:7)
    at next (/Users/diggersf/.nvm/v0.4.9/lib

Allow in-memory sqlite db for testing

It would be great to make use of the special ':memory:' sqlite location to create databases in memory. The package currently expects there to be a file existing to report various stats.

MBtiles caching

Hello.
I host mbtiles with tilelive + mbtiles. When I update .mbtiles file, server returns an old tiles, even if I delete .mbtiles file. I can't find any info, how to prevent caching.
In the docs: file will be cached only if we use new sqlite3.cached.Database(), but you use new sqlite3.Database() (https://github.com/mapbox/node-mbtiles/blob/v0.9.0/lib/mbtiles.js#L59). So I can't understand, why the file is cached and how to prevent it.

Missing licensing info

Hello,
I'm packaging the whole stack leading up to tilemill for Debian.

While packaging mbtiles, I noticed it's missing any information about the license it's released under. There's only a "BSD" in package.json which is practically useless :)

It would be best if you put a LICENSE or COPYING file in the repository (and the released tarball).

In any case, it would be (temporarily) ok if you state the full license in a reply to this bugreport, so that I can continue my work in Debian, without waiting for a new mbtiles release.

Thank you very much,
David Paleino

new tag

would be great to have a new tag to allow tilemill to pull in the new tag of node-sqlite (with the changes to make packaging easier)

data.key passed as number leads to sqlite type affinity problems

data.key can be passed as a number by tilelive at https://github.com/mapbox/node-mbtiles/blob/master/lib/mbtiles.js#L552.

This is the cause of https://github.com/mapbox/tilemill/issues/1368.

This can lead to floating point numbers being pushed into sqlite for the rows that reference duplicate tiles, leading to broken lookups:

sqlite> SELECT i.tile_id FROM images i WHERE NOT EXISTS (SELECT m.tile_id FROM map m WHERE m.tile_id = i.tile_id);
-3101615871
-4294967295
sqlite> SELECT i.tile_id FROM images i WHERE NOT EXISTS (SELECT m.tile_id FROM map m WHERE CAST(m.tile_id as string) = CAST(i.tile_id as string));
sqlite> 

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.