tilemill-project / tilemill-builder-osx Goto Github PK

node v6 handles paths differently: http://nodejs.org/api/modules.html#modules_cycles, and so does npm.

For now the global modules check is disabled. This needs more testing to get working again.

We likely either need to globally install modules that are only needed for testing: like jshint or expresso, each build, so that they are present in the global folder: /Volumes/Flex/<build data>/lib/node_modules/, or install them once on the machine in a custom spot and then set NODE_PATH to point to them.

npm stores a big cache in ~/.npm/. We should watch out for getting burned by this cache, and perhaps should clear it before each build. The worry here is that sometimes versions of TileMill deps point to master tarballs during development (that do not increment their version in package.json). In rare cases I could see npm pulling from this catch and not grabbing the right/new versions if we get lazy upstream and forget to increment.

The flip side is that a local cache likely speeds up builds quite a bit, so if we don't get ever burned we should absolutely leverage it.

currently broken:

/Users/dane/Desktop/TileMill 2.app/Contents/Resources/node_modules/bones/node_modules/jquery/node_modules/jsdom/node_modules/contextify/node_modules/bindings/bindings.js:91
  throw err
        ^
Error: Could not locate the bindings file. Tried:
 → /Users/dane/Desktop/TileMill 2.app/Contents/Resources/node_modules/bones/node_modules/jquery/node_modules/jsdom/node_modules/contextify/build/contextify.node
 → /Users/dane/Desktop/TileMill 2.app/Contents/Resources/node_modules/bones/node_modules/jquery/node_modules/jsdom/node_modules/contextify/build/Debug/contextify.node
 → /Users/dane/Desktop/TileMill 2.app/Contents/Resources/node_modules/bones/node_modules/jquery/node_modules/jsdom/node_modules/contextify/build/Release/contextify.node
 → /Users/dane/Desktop/TileMill 2.app/Contents/Resources/node_modules/bones/node_modules/jquery/node_modules/jsdom/node_modules/contextify/out/Debug/contextify.node
 → /Users/dane/Desktop/TileMill 2.app/Contents/Resources/node_modules/bones/node_modules/jquery/node_modules/jsdom/node_modules/contextify/Debug/contextify.node
 → /Users/dane/Desktop/TileMill 2.app/Contents/Resources/node_modules/bones/node_modules/jquery/node_modules/jsdom/node_modules/contextify/out/Release/contextify.node
 → /Users/dane/Desktop/TileMill 2.app/Contents/Resources/node_modules/bones/node_modules/jquery/node_modules/jsdom/node_modules/contextify/Release/contextify.node
 → /Users/dane/Desktop/TileMill 2.app/Contents/Resources/node_modules/bones/node_modules/jquery/node_modules/jsdom/node_modules/contextify/build/default/contextify.node
 → /Users/dane/Desktop/TileMill 2.app/Contents/Resources/node_modules/bones/node_modules/jquery/node_modules/jsdom/node_modules/contextify/compiled/0.10.15/darwin/x64/contextify.node
    at bindings (/Users/dane/Desktop/TileMill 2.app/Contents/Resources/node_modules/bones/node_modules/jquery/node_modules/jsdom/node_modules/contextify/node_modules/bindings/bindings.js:88:9)
    at Object.<anonymous> (/Users/dane/Desktop/TileMill 2.app/Contents/Resources/node_modules/bones/node_modules/jquery/node_modules/jsdom/node_modules/contextify/lib/contextify.js:1:96)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/Users/dane/Desktop/TileMill 2.app/Contents/Resources/node_modules/bones/node_modules/jquery/node_modules/jsdom/lib/jsdom/browser/index.js:14:21)
    at Module._compile (module.js:456:26)

Any ideas here? Current builds are failing with this test: https://github.com/mapbox/tilemill-builder-osx/blob/master/build.sh#L335-340

for i in `find . -name '*.node'`; do
  if [ -n "`otool -L $i | grep version | sed -e 's/^[^\/@]*//' | grep -v ^\/usr/lib | grep -v '@loader_path/libmapnik.dylib'`" ] || [ -n "`otool -L $i | grep local`" ]; then
    echo "Improper linking for $i"
    #exit 1
  fi
done

Results are:

Checking Mapnik module linking & architecture…
Improper linking for ./node_modules/bones/node_modules/jquery/node_modules/htmlparser/libxmljs.node
Improper linking for ./node_modules/bones/node_modules/jquery/node_modules/jsdom/node_modules/contextify/build/Release/contextify.node

Also, why not exit 1?

All mapnik datasource plugin compiles are optional depending on whether their dependencies are found (for those that have dependencies). So, we need to add a check for the presences of all key datasources like 'ogr.input', 'gdal.input', 'postgis.input'. At the same time we might as well enforce the presence of 'shape.input' and 'csv.input' as well.

It looks like some xcode or other upgrade may have changed the libxml/iconv versions such that the gdal binary mapnik builds against is too stale. Error is https://gist.github.com/springmeyer/f8325c00105995cb32c3.

Will require updating the old mapnik SDK on this machine with a newer gdal.

Currently libgdal is used in two places in TileMill - in the Mapnik GDAL/OGR plugins and by node-srs. In each case the relevant code is statically linked - so each bit pulls in just what it needs of GDAL. But, we should consider distributing a single libgdal. One piece of making this work is tweaking the node-srs build scripts to work against this shared gdal, which is now done as per mapbox/node-srs#23

Ideally sparkle.sh could be updated to detect a local build rather than having to download from github.

• key needs added to build machine to run it locally.
• need to look into this error:

./sparkle.sh: line 13: [: !=: unary operator expected

Not sure why, but it is preventing tilemill from starting. When I ran a build manually it also failed to install, but then doing npm install sqlite3@~2.1.5 manually worked.

Appears the || and && logic is not working right.

Is there a strong reason why this repository isn't public?

We've had occasional issues with rolling release builds based on our automated dev builds. @willwhite can you elaborate on the process and we can adjust the build script to account for them? Maybe a second script, or a flag on the main script, that changes the appcast URL and whatever else needs to happen.

second half of https://github.com/mapbox/tilemill/commit/35fdd6ade2ff83a7239f50c184410f1cc1459db6

Would it make sense to name the final zip files using the output of git describe --tags?

Trying to build on build machine with 0.4.12 and getting this:

npm verb installOne cb [email protected]
npm ERR! Unsupported
npm ERR! Not compatible with your version of node/npm: [email protected]
npm ERR! Required: {"node":"~0.6.x"}
npm ERR! Actual: {"npm":"1.0.105","node":"0.4.12"}

Are we building wholesale on 0.6 or is this a fluke?

/cc @springmeyer @willwhite @miccolis

http://developer.github.com/v3/repos/downloads/

node is currently getting installed globally, but ideally should be installed in some sandbox and then $PATH modified to point to it.

Currently hitting

  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang '-D_DARWIN_USE_64_BIT_INODE=1' '-D_LARGEFILE_SOURCE' '-D_FILE_OFFSET_BITS=64' '-D_GNU_SOURCE' '-DHAVE_CONFIG_H' '-DCARES_STATICLIB' -I/Volumes/Flex/mapnik-packaging/osx/out/build-libc++-x86_64/include -I../deps/cares/include -I../deps/cares/src -I../deps/cares/config/darwin  -Os -gdwarf-2 -mmacosx-version-min=10.5 -arch x86_64 -Wall -Wendif-labels -W -Wno-unused-parameter -fno-strict-aliasing -MMD -MF /Volumes/Flex/mapnik-packaging/osx/out/packages/node-v0.10.17/out/Release/.deps//Volumes/Flex/mapnik-packaging/osx/out/packages/node-v0.10.17/out/Release/obj.target/cares/deps/cares/src/ares__close_sockets.o.d.raw -I/Volumes/Flex/mapnik-packaging/osx/out/build-libc++-x86_64/include -DNDEBUG -O3 -arch x86_64 -D_FILE_OFFSET_BITS=64 -mmacosx-version-min=10.7 -isysroot /SDKs/MacOSX10.7.sdk -c -o /Volumes/Flex/mapnik-packaging/osx/out/packages/node-v0.10.17/out/Release/obj.target/cares/deps/cares/src/ares__close_sockets.o ../deps/cares/src/ares__close_sockets.c
In file included from ../deps/cares/src/ares__close_sockets.c:17:
../deps/cares/src/ares_setup.h:79:10: fatal error: 'errno.h' file not found

I've tested a TileMill build on my own machine with 4.3, but I want to upgrade the build server as well. Anyone in the midst of anything that should have this wait and stay on 4.2.1 for now?

/cc @springmeyer @tmcw @willwhite

This will cut down on unnecessary builds as well as duplicate chat log messages.

Like #27, we have another failure from too aggressively clearing out known folder names that do not (you would think) contain any runtime code:

module.js:340
    throw err;
          ^
Error: Cannot find module 'nwmatcher/src/nwmatcher-noqsa'
    at Function.Module._resolveFilename (module.js:338:15)
    at Function.Module._load (module.js:280:25)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/Users/dane/TileMill.app/Contents/Resources/node_modules/bones/node_modules/jquery/node_modules/jsdom/lib/jsdom/selectors/index.js:1:79)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)

@willwhite, you and I had talked about this. It would be nice to know which version of Mapnik a build was based on, either hidden someplace, recorded in our log, and/or available in the about box. Maybe something like what Xcode does, i.e. both a version number (our TileMill hash) and a build number (our Mapnik info) as far as placement in the about box?

This will probably fall in with #4 as well. Release 0.7.0 went up with a 3051 build number, so there were Sparkle problems. Will need to fix this for production builds. /cc @willwhite

@springmeyer, what's the latest on this? I had a note to ping you about it in a while, but I don't remember the status. Maybe that you've got things building on v6 now, and I wanted to update this script for that?

We need to figure out if and by what system we would auto-update a dev Sparkle feed for testers. Currently this URL is defined in the first dev build of 0.7.0 I posted as http://mapbox.com/tilemill/platforms/osx/appcast-dev.xml, which doesn't exist yet.

While we might auto-upload nightly builds automatically, we probably don't want to push every one of these to Sparkle or that would be turbo-annoying.

Maybe something like tabs with a b in them, like 0.7.0b1?

/cc @willwhite

Just a placeholder ticket for this. What's the status, @springmeyer? If perchance we can do this on the sooner side, I could upgrade the build server to 10.8 next week when I'm in DC...

I frequently find I can only get like 10-15 kb/s via the vpn connection to the build server to be able to download builds. This of course takes ages and the problem is magnified by then having to upload and download again to make a dev channel build.

Perhaps we could try to hit the github api and upload builds automatically, signing locally, in one shot: http://developer.github.com/v3/repos/downloads/