Ability to build standalone binaries against GDAL SDK about node-gdal HOT 26 CLOSED

@brandonreavis @brianreavis
I've built gdal with expat, geos and proj on Windows.
What is it exactly you would need?
Just gdal.lib or anything else?

from node-gdal.

@BergWerkGIS Epic! Just gdal.lib to link node-gdal to. We need to get libpq in eventually (#20), but I haven't had a chance to get it configured in gyp... but maybe that's irrelevant for the windows setup.

from node-gdal.

DL here (libpq included):
https://mapnik.s3.amazonaws.com/dist/dev/node-gdal_win32_deps.7z

Note:
To get a test gdal.lib as fast as possible, I've not used gyp, but the windows build scripts that are included with the sources of each library.

As next step I think it would be good to use your gyp files to have the same system for all plattforms?
With that approach we also have to keep in mind that gdal-config is only available on Unix systems.
So, for Windows, we will have to create a simple BAT that spits out the libs.

To test locally I would have to replace deps/libgdal/libgdal.gyp:libgdal with <pathto>/gdal.lib?

Correct?

from node-gdal.

I've never done it before, but it looks like it needs to go in "libraries" (like so).

from node-gdal.

@springmeyer today I took a stab at your second bullet point. It doesn't look like the mapnik sdk bundle includes header files for gdal. From poking around in node-mapnik's build_from_sdk script, it looks to me like the shared libgdal will need gdal-config, headers, and the compiled lib.

In your vision does this mean another TARGET in mapnik-packaging specifically for building gdal?

from node-gdal.

@brianreavis @springmeyer
For creating a libgdal.lib to download, I switched from my approach to using your repo.
I think this makes more sense: only one repo and one batch of build scripts to maintain.

Probably the gyp script could be tweaked to check if there already is a libgdal.lib (has been downloaded before npm install is called) and only if there is none, to compile everything.

I managed to compile geos on Windows. Just added a few additional includes.

With the latest geos snapshots this is not necessary.
Would it be possible to upgrade geos to a recent snapshot?

I'm not able to compile node-gdal.
Error on AppVeyor
Line in source

 gdal_rasterband.cpp
..\src\gdal_rasterband.cpp(210): error C2664: 'CPLErrorHandler CPLSetErrorHandler(CPLErrorHandler)' : cannot convert argument 1 from 'void (__cdecl *)(CPLErr,int,const char *)' to 'CPLErrorHandler' [C:\projects\node-gdal\build\gdal.vcxproj]
None of the functions with this name in scope match the target type

I think to this could be due to the fact that I'm using VS2013/MSBuild 12 (look for Microsoft Visual Studio 12.0 in the AppVeyor output) and you are using MSBuild 10 (look for Microsoft Visual Studio 10.0).

BTW:
Which branch should I be working on?
master or pre-gyp-publish?
I've used pre-gyp-publish.

from node-gdal.

With my latest commit local build is successfull.

In src/gdal_rasterband.cpp I changed
void statisticsErrorHandler(CPLErr eErrClass, int err_no, const char *msg)
to
void CPL_STDCALL statisticsErrorHandler(CPLErr eErrClass, int err_no, const char *msg).

AppVeyor is still running: https://ci.appveyor.com/project/BergWerkGIS/node-gdal/build/1.0.2

from node-gdal.

We just merged pre-gyp-publish into master, so probably master (or a new branch that can be merged in)? I just added you as a collaborator, so you should have full push access and such.

I think @springmeyer would know more about the best approach on where to put the build scripts. At one point we were talking about one repository for each dependency, so in the future there could be multiple node bindings without the need for nearly-identical build scripts in each. cc: @yhahn @rclark

For right now though, I think it's totally fine to have in the repo here. In the future we might want to try splitting them out.

Finally, thanks for all the work on this!

from node-gdal.

@BergWerkGIS I agree that its nice to keep all the build scripts together, however that's what runs us into the original problem: these builds timeout on AppVeyor after working for 30 min.

As I understand, this is why we actually need to split the building of gdal, etc. from building the bindings for this library. @springmeyer definitely is the goto on the vision for how this might look overall.

In the short-term, and for a first / alpha release, we could definitely use @BergWerkGIS's work in this repository to build, package, and pre-gyp publish for Windows, just not automatically using AppVeyor. Thoughts on going this route?

from node-gdal.

I'm thrilled to see the movement here. @rclark - right: in my experience with node-mapnik, I've found it is critical to have fairly quick (~5 min) build times for node-mapnik for easy per-commit testing and also smooth packaging. So yeah, ensuring as much work as possible goes into the SDK of headers and libgdal is good. Will be reviewing more as I catch up this week.

from node-gdal.

Let me explain what I meant in more detail:

• one repo
• one binding.gyp
• appveyor.yml downloads libgdal.lib before npm install is called
• binding.gyp checks if there is a build\release\libgdal.lib
  • if there is, then just node-gdal get's built (Aimed for building on AppVeyor).
  • if there is none, deps and node-gdal get built (Aimed for general local development and preparation of libgdal.lib as a download).

Does this sound reasonable?

from node-gdal.

On unix systems we can do npm install gdal --shared_gdal and the shared_gdal gets set to something other than false:

from node-gdal.

btw, I realize the name --shared_gdal is terrible. But the idea holds: by default builds use bundled gdal, but a command line option can be passed to trigger building against an external gdal. I like the idea of the external GDAL being able to be anywhere rather than just in build\release\libgdal.lib. What do you think?

from node-gdal.

I'll have a look at passing parameters to npm install (still on the way making myself familiar with gyp).
Of course, custom path for the lib is what we should aim for.
build\release\libgdal.lib was just an example, because the current script puts it there.

from node-gdal.

@brianreavis @springmeyer

YEAH!

Local build with prepared libs was successful (npm test: 98 passing, 9 pending).

AppVeyor is still running at the moment https://ci.appveyor.com/project/BergWerkGIS/node-gdal/build/1.0.6

I'm passing the directory containg the libs via --shared_gdal=c:/projects/node-gdal/deplibs/.

My changes were these:
https://github.com/BergWerkGIS/node-gdal/compare/pre-gyp-publish?expand=1

from node-gdal.

@BergWerkGIS Epic! That looks clean too. How/where is https://mapnik.s3.amazonaws.com/dist/dev/libgdal_win_$env:PLATFORM.7z being built? Is that a manual process based on build-gdal?

from node-gdal.

@brianreavis
I've abandoned build-gdal for this task.
I think it is better to have one build process (this repo) so that the prebuilt libs match exactly what node-gdal is expecting.

This is how I did it:

• local build with npm install --build-from-source --msvs_version=2013
• package build\release directory into 7z. Including msys find to get the list of libs in a format that works with binding.gyp (The Windows find utility cannot produce appropriate output).
• upload 7z to S3
• adjust appveyor.yml that it downloads and extracts the 7z and calls npm install --build-from-source --msvs_version=2013 --shared_gdal=c:/projects/node-gdal/deplibs/

from node-gdal.

I think it is better to have one build process (this repo) so that the prebuilt libs match exactly what node-gdal is expecting.

Interesting. I trust your judgement on this and would be interested to hear more about the tradeoffs you are seeing by using external scripts and whether/how we might learn to adapt your findings to the mapnik/node-mapnik packaging process for windows.

from node-gdal.

But don't let my curiosity slow you down - just flagging my interest. Getting a first windows tag out of node-gdal is still the priority - thanks for making it possible @BergWerkGIS!

from node-gdal.

My statement was based on:

• priority to get a working Windows node-gdal
• at the moment it seems to me that mapnik/node-mapnik vs node-gdal have slightly different needs
• I was not able to get node-gdal working with the libs created by build-gdal at frist try

Of course I think the final target should be to have one package/SDK/repo containing the dependencies that are needed by several projects.
On the other hand it might be overkill to download a big SDK if just one lib is needed.

Looking forward to a chat to connect the dots and get an overview of the overall direction/needs/involved libs and projects.
Of course only after the 🔥 has cooled down and you are safe.

from node-gdal.

@BergWerkGIS this is great and positions us to make a first release, which is 👍 👍. A couple questions that come to mind:

• the 7z archive is dropped into /deplibs/gdal, but binding.gyp includes headers from /deps/gdal. Could these get out-of-sync with each other?
• any thoughts on how we could take the "local" out of the process of building the 7z file?
• Looks like the appveyor build didn't finish on node 0.8.x
• want to merge #44 and work in this repo instead of your fork?

from node-gdal.

Thanks 😄

• Yes, they could get out of sync. Each time new gdal headers are introduced to node-gdal, the libs would have to be rebuilt
• One way to get "local" out of the build process would be to upgrade the AppVeyor account. When I exchanged some emails with Feodor Fitsner (the AppVeyor guy), he offered custom plans. I'm pretty sure we could work out a deal with more CPU or longer build time. AppVeyor is based on Azure so the possible configuration would be one of those: http://msdn.microsoft.com/en-us/library/azure/dn197896.aspx
• node 0.8: oh yeah, forgot about that: the version of node-gyp that's bundled with 0.8 doesn't know about --msvs_version=2013 yet. I've already solved that somewhere else, I think with node-mapnik (will take a look)
• ok, will work in this repo from now own

from node-gdal.

@rclark
Concerning "local":
The preferable solution would of course be a dedicated repo for creating/uploading the deps/headers.
Like outlined by Dane: mapbox/windows-cpp-collab/issues/50
Current solution is based on priority to get a working Windows release and the fact, that I was not able to get node-gdal working with the libs created by build-gdal, will investigate.

from node-gdal.

Build with node 0.8 is successful.
Trick is, to do a npm install node-gyp, so that the old node-gyp that is bundled with 0.8 is not used.
See AppVeyor

from node-gdal.

The 'mapbox' appveyor account is already on a somewhat custom plan. I worked with Feodor to get a concurrency of 5. I forgot to ask about longer build times. I agree that priority here is node-gdal tag so local build for the SDK is fine, but in the coming weeks an appveyor build would be pretty sweet.

On Jul 17, 2014, at 3:21 PM, Wilhelm Berg [email protected] wrote:

Thanks

Yes, they could get out of sync. Each time new gdal headers are introduced to node-gdal, the libs would have to be rebuilt
One way to get "local" out of the build process would be to upgrade the AppVeyor account. When I exchanged some emails with Feodor Fitsner (the AppVeyor guy), he offered custom plans. I'm pretty sure we could work out a deal with more CPU or longer build time. AppVeyor is based on Azure so the possible configuration would be one of those: http://msdn.microsoft.com/en-us/library/azure/dn197896.aspx
node 0.8: oh yeah, forgot about that: the version of node-gyp that's bundled with 0.8 doesn't know about --msvs_version=2013 yet. I've already solved that somewhere else, I think with node-mapnik (will take a look)
ok, will work in this repo from now own
—
Reply to this email directly or view it on GitHub.

from node-gdal.

closing this as I think we're mostly done. I have longer term plans to make it easier to create the standalone SDK's and this will surface over at https://github.com/mapbox/mason once I start on this project.

from node-gdal.