project-osrm / node-osrm Goto Github PK

Currently OSRM just prints to std::cerr and calls exit if data, like the .hsgr, is not found as referenced in the server.ini (in typedefs.h -> ERR). This makes it impossible for the node module to recover from an error like this.

Ideally OSRM could throw an exception in QueryObjectsStorage.cpp:36.

The current batch of binaries are hosted under my personal aws account, but these should be migrated to a Mapbox controlled account so other committers on this project can have admin access to the bucket.

To do this I will:

• - Upgrade node-osrm master to node-pre-gyp 0.5.x - 6b5b6d3
• - Copy all existing binaries from https://node-osrm.s3.amazonaws.com to https://mapbox-node-binary.s3.amazonaws.com/osrm/ (note the osrm prefix)
• - Point package.json binary/host value at mapbox-node-binary - 495eff5
• - Re-encode the node-pre-gyp publishing keys in .travis.yml with new keys that work to publish to this new location - f48cfa4
• - In a few months once a new node-osrm tag is out and most people have upgraded I'll delete the node-osrm bucket.

Boost libs sometimes have -mt at the end to signify they are the multithreaded versions. After Project-OSRM/osrm-backend@4a3db7e node-osrm now needs to link all dependent libraries so we need to know their exact name.

So, builds against homebrew based boost will start failing at 1c51388 until we come up with a solution.

Possibilities are:

• depend on OSRM to provide a pkg-config that provides all library names
• come up with some custom node-osrm script to find and detect boost library names

I think pkg-config is ideal, just need to start testing this post Project-OSRM/osrm-backend@8383222

Create network data on the fly before tests are run using a small osm extract

@springmeyer and I discussed this -- they are not used and inflate the Travis build times.

Work started in b8457fc, just need to actually post binaries for both OS X and Linux. This ticket is to track when this work is done.

Ideally we could store a very small set of data for the tests to make it dead easy to test node-osrm on different platforms quickly (without having to download and wait to process data with osrm-extract and osrm-prepare). Is the data portable across architectures or will we need to generate copies for both 32 bit and 64 bit systems? Is the data fairly portable across node-osrm versions?

More JSy API:

• No separate Options or Query objects. JS idiom is just to pass options hashes directly.
• new osrm.Engine({ini: <path>, sharedMemory: <boolean>}) (better would be to eliminate the assumption of having a file on disk with options and pass them directly -- don't know enough about OSRM to gauge if that's feasible)
• Engine#route(options, callback)
• Engine#locate(latLon, callback)
• Engine#nearest(latLon, callback)
• Do we even need sync versions of these?
• Engine functions return JS objects to callback rather than a JSON string.

If I have the shared memory flag on, and have the .ini file pointing to .osrm files, does it still try to load those files? I tried to have the location of the files pointed to in the .ini files point to invalid locations, but it just throws an error that the files don't exist.

node-osrm latest now builds and runs against the latest develop code from OSRM with shared memory support. But current defaults are in place that do not turn on shared memory usage. This ticket stands to track actually testing it from node.js

Looks like we need to catch up with Project-OSRM/osrm-backend@cabaad4#diff-ce251cb4263f3c33dfb996c10931b6f2

In file included from ../src/node_osrm.cpp:15:
../src/engine.hpp:106:55: error: no member named 'c_str' in 'std::vector<std::basic_string<char>, std::allocator<std::basic_string<char> > >'
    return scope.Close(String::New(osrm_reply.content.c_str()));
                                   ~~~~~~~~~~~~~~~~~~ ^
../src/engine.hpp:165:25: error: no viable overloaded '='
        closure->result = osrm_reply.content;
        ~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~
/usr/include/c++/4.2.1/bits/basic_string.h:499:7: note: candidate function not viable: no known conversion from 'std::vector<std::string>' to
      'const std::basic_string<char>' for 1st argument
      operator=(const basic_string& __str) 

Installation is currently non-standard and currently depends on a clone of this repo being next to a built Project-OSRM directory. What would need to happen in order to it possible to install by simply running npm install node-osrm?

As a sanity check, if Travis is building a tag for release, it should error out early on if the tag does not match the version in package.json.

This will catch an error I made on my first try, when I forgot to include the package.json commit in the v0.2.5 tag.

node-osrm should be able to report the git hash of OSRM it is built against.

Possible tests:

• - invalid Port, IP, and Threads values in server.ini.
• - a single missing data reference: e.g. all data references are valid except edgesData points to a corrupt file.

The existing shared memory test never actually tested shared memory, and is now disabled. It will requrie some pre-test setup to get working.

Hitting this when using shared memory: https://gist.github.com/springmeyer/daa7705b5e81b8af95bb

Hoping that the problem is some mixing of boost headers/libs across versions: Project-OSRM/osrm-backend@9e75595#commitcomment-4529676

Ideally, for node-osrm, there could be a way to disable this OSRM output so that meaningful logging could be controlled by the application calling into node-osrm

It should be possible to run 'nearest' and 'locate' queries in addition to 'viaroute'. Something like this for nearest:

var query = new osrm.Query({
    service: "nearest",
    coordinates: [[52.4224,13.333086]]
});
var sync_result = engine.run(query);

or this for locate:

query = new osrm.Query({
    service: "locate",
    coordinates: [[52.4224,13.333086]]
});
sync_result = engine.run(query);

When I load node-osrm with data files that correspond to the entire planet, I get this error:

var engine = new osrm.Engine(opts);
^
Error: std::bad_alloc
at Object. (/home/ubuntu/routing_osrm_nodejs/server.js:10:14)
at Module._compile (module.js:449:26)
at Object.Module._extensions..js (module.js:467:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.runMain (module.js:492:10)
at process.startup.processNextTick.process._tickCallback (node.js:245:9)

Currently the paths appear to be interpreted as relative to the cwd of the calling process, which feels not ideal. The paths should be relative to the server.ini location since that is the file that defines the paths.

OSRM builds with clang and libc++ on Mavericks. This needs to be reflected in bindings.gyp. Otherwise, we get undefined references.

Berlin extract was helpful for early testing, but since OSRM core has extensive tests, I think we can scale back the testing extract size in node-osrm. A smaller size will help devs get set up faster and will ensure we don't hit travis timeouts (saw these today).

Goal:

• Put heavy load on node-osrm.Engine.run() to ensure that the async approach to running the route query holds up (does not hang or crash) and then produce initial profiling results to ensure no bottlenecks exist in the bindings themselves.

Tasks:

• Write a small http server on top of the node-osrm
• Devise a set of random route requests against the berlin test dataset
• Consider adapting the https://github.com/mapbox/bench tool for making the requests and timing responses.
• Run these tests on various hardware

Is it possible to have node-osrm use the same shared memory with an osrm-routed instance? This would also be extremely helpful!

@springmeyer @jfirebaugh

So far, libOSRM returns a JSON formatted return string. Would it make more sense in terms of efficiency, and the way NodeJS would like to have things, to return an object that has a certain interface that NodeJS can use.

The idea is to expose some object with a well-known interface/structure that can be used in any subsequent transformation of the result. I'd like to save some time on (costly) string generation/duplication.

With the move to tbb of upstream OSRM we should discuss dropping static linking for this one node plugin.

Cc: @springmeyer @jfirebaugh

Does the IP address and Port in the .ini file point to the address and port of the OSRM server that node-osrm will be connecting to? Does this indicate then that node-osrm is sending and receiving to the engine through a socket?

Travis v0.8.x tests are all passing but seeing and Illegal instruction failure with v0.10.x: https://travis-ci.org/DennisOSRM/node-OSRM/jobs/15018896#L740.

If I use node-osrm as my bindings to OSRM, does this allow me to have multiple simultaneous connections to OSRM?

Linking fails on OS X with home-brew installed boost. It comes with the -mt suffix. Leads to:

ld: library not found for -lboost_thread
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Release/osrm.node] Error 1
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/local/Cellar/node/0.10.22/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:267:23)
gyp ERR! stack     at ChildProcess.EventEmitter.emit (events.js:98:17)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (child_process.js:789:12)
gyp ERR! System Darwin 13.0.0
gyp ERR! command "node" "/usr/local/Cellar/node/0.10.22/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/dennisluxen/Coding/node-osrm
gyp ERR! node -v v0.10.22
gyp ERR! node-gyp -v v0.11.0
gyp ERR! not ok 

Is it possible to get the nearest road point through node-osrm? I've only seen routing requests calculated. I was hoping to have the APIs that are available through the OSRM http API available through node-osrm.

We could get a slightly simpler configuration if we were able to pass a hash into the Options constructor in addition to an ini file path.

new osrm.Options({
    threads: 8,
    hsgrData: 'path/to/data'
    ...
}, true);

parsing of server.ini and command line options will change with the shared_memory branch of OSRM. This needs to be implemented/pulled in here as well.

@DennisOSRM - I added travis config in 454fe6e.

Could you enable when you have a chance?

It appears that g_GIT_DESCRIPTION is not available in libOSRM.dylib and therefore ./Util/GitDescription.cpp may need to be added to libOSRM. The result is that when running node-osrm on OS X (at least) we hit:

Error: dlopen(/Users/dane/projects/Project-OSRM/node-osrm/lib/_osrm.node, 1): Symbol not found: _g_GIT_DESCRIPTION
  Referenced from: /Users/dane/projects/Project-OSRM/node-osrm/lib/_osrm.node
  Expected in: dynamic lookup

When I try to build or install node-osrm on RedHat, I get the following error:

module.js:485
  process.dlopen(filename, module.exports);
          ^
Error: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /home/mobius/node-osrm-master-3/lib/binding/osrm.node)
    at Object.Module._extensions..node (module.js:485:11)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:362:17)
    at require (module.js:378:17)
    at Object.<anonymous> (/home/mobius/node-osrm-master-3/lib/osrm.js:6:15)
    at Module._compile (module.js:449:26)
    at Object.Module._extensions..js (module.js:467:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)

What can I do to resolve this issue?

I'm using OSRM-develop (ce784e0491) and doing this to load the shared memory:

$ osrm-datastore /absolute/path/to/data.osrm

I can see the shared memory with ipcs -m. osrm-routed is able to access this shared memory.

Then I'm using node-osrm like this, where data.ini contains same absolute paths to data:

var opts = new osrm.Options("./data.ini", true /*used shared memory*/);
var engine = new osrm.Engine(opts);

node-osrm works, but it's not using the shared memory. I can manually free the shared memory segment with ipcrm -m {shmid} and my node app still does routing, for example.

Any ideas? Is it because I'm not using the latest version of OSRM, perhaps?

Plan to look into this such that npm publish automatically happens on git push --tags but after new binaries are published.

On the C++ side an OSRM engine has non-memory resources that are deallocated when it's destructed right? I think we need to expose that explicitly on the JS side so it doesn't leak if JS garbage isn't immediately collected.

What I'm seeing running tests with shared memory that create an Engine per test case is that after a few tests, Engine creation starts failing with "Error: Too many open files".

Need to enable the two skipped/pending (blue tests) once upstream issues are fixed:

• Project-OSRM/osrm-backend#692
• Project-OSRM/osrm-backend#693

@DennisOSRM - can you let me know if those issues are ones you'd like to jump on or would prefer me to take a look at?

Once solved these tests can be enabled by doing:

diff --git a/test/osrm.test.js b/test/osrm.test.js
index e03ba19..1664415 100644
--- a/test/osrm.test.js
+++ b/test/osrm.test.js
@@ -16,7 +16,7 @@ describe('osrm', function() {
     });

     // @TODO not safe yet to test: https://github.com/DennisOSRM/Project-OSRM/issues/692
-    it.skip('should throw if ini file is blank', function(done) {
+    it('should throw if ini file is blank', function(done) {
         assert.throws(function() { new osrm.Engine("./test/data/bogus.ini"); },
         /@TODO/);
         done();
@@ -38,7 +38,7 @@ describe('osrm', function() {


     // @TODO not safe yet to test: https://github.com/DennisOSRM/Project-OSRM/issues/693
-    it.skip('should throw if ini references corrupt files', function(done) {
+    it('should throw if ini references corrupt files', function(done) {
         assert.throws(function() { new osrm.Engine("./test/data/references-corrupt-files.ini"); },
         /hsgr not found/);
         done();

@springmeyer

https://github.com/DennisOSRM/node-OSRM/blob/master/src/query.hpp#L94-L95 seems to only add origin and destination but not any intermediate stops.

Is there some way to install my locally-built copy of node-osrm as a dependency to my own project, and also get the lib/binding/ folder included?

I use the following to build node-osrm and install it into my local, global npm repo:

$ cd node-osrm-master
$ npm install -g --build-from-source

Later when I try to install node-osrm as a dependency for my own project, it fails to copy the binding folder:

$ cd my_project
$ npm install osrm
...
$ ls ./node_modules/osrm/lib/
index.js  osrm.js

This means that I have to manually copy the binding folder from my global repo:

$ cp -r `npm root -g`/lib/node_modules/osrm/lib/binding ./node_modules/osrm/lib/

When I build now, I get this error:

node-pre-gyp ERR! build error
node-pre-gyp ERR! stack Error: Failed to execute 'node-gyp rebuild --fallback-to-build' (1)
node-pre-gyp ERR! stack at ChildProcess.module.exports.run_gyp (/home/mobius/node-osrm-master-3/node_modules/node-pre-gyp/lib/util/compile.js:34:29)
node-pre-gyp ERR! stack at ChildProcess.EventEmitter.emit (events.js:99:17)
node-pre-gyp ERR! stack at maybeClose (child_process.js:640:16)
node-pre-gyp ERR! stack at Process._handle.onexit (child_process.js:682:5)
node-pre-gyp ERR! System Linux 2.6.32-358.el6.x86_64
node-pre-gyp ERR! command "node" "/home/mobius/node-osrm-master-3/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR! cwd /home/mobius/node-osrm-master-3
node-pre-gyp ERR! node -v v0.8.23
node-pre-gyp ERR! node-pre-gyp -v v0.4.2
node-pre-gyp ERR! not ok
npm ERR! [email protected] install: node-pre-gyp install --fallback-to-build
npm ERR! sh "-c" "node-pre-gyp install --fallback-to-build" failed with 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the osrm package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-pre-gyp install --fallback-to-build
npm ERR! You can get their info via:
npm ERR! npm owner ls osrm
npm ERR! There is likely additional logging output above.

npm ERR! System Linux 2.6.32-358.el6.x86_64
npm ERR! command "/opt/node-js/bin/node" "/opt/node-js/bin/npm" "install" "--build-from-source"
npm ERR! cwd /home/mobius/node-osrm-master-3
npm ERR! node -v v0.8.23
npm ERR! npm -v 1.2.18
npm ERR! code ELIFECYCLE
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! /home/mobius/node-osrm-master-3/npm-debug.log
npm ERR! not ok code 0

Not sure why, but I'm seeing fatal: depth 0 is not a positive number in the logs which looks quite odd: https://travis-ci.org/DennisOSRM/node-osrm/jobs/19606691

Currently the Travis.ci build has a static_build target that pulls down OSRM, builds and links it statically to the node-osrm binary for both node v0.8.x and v0.10.x.

There are two ways we can improve this:

• Avoid actually compiling OSRM and instead just pull down a pre-built SDK with all headers and pre-built libs ready to build against. This will speed up builds immensely.
• Open up a switch to make it easy to decide what git hash of OSRM to build against (likely best to have the SDK built by the OSRM core .travis and posted publicly to make it easy to switch between versions if needed)

Until the above is done: the OSRM version used in the node-osrm binaries is determined by the hash used in the source build which is done by mapnik-packaging scripts here.

Node-osrm should use the shared memory facility of OSRM. From my point of view it is fair to also deprecate the ini files in the node bindings.

@springmeyer how much work do you think this is?

Naming convention in npm seems to be all lowercase.