Use Docker containers on TravisCI about gmt HOT 8 CLOSED

I have a functioning docker container which currently has an in-container build for testing purposes.However, it can easily be adapted to serve as a base image for CI/CD

You can simply use the first part of the Dockerfile to configure the environment. Second part fetches GSSHG and DCW data, third part builds gmt using a simplistic build script.

Dependencies will have to be updated for 5.x builds but that was on my road map as well.

https://github.com/nc5ng/gmt-docker/blob/master/Dockerfile

## Dockerfile for Base GMT Image nc5ng/gmt for 6.0.0 branch
##     (Currently master)
##
## Approach: Perform development build in situ
##
## Construct and delete the build environment.
## This reduces image size at expense of longer build.
## Also fetch DCW/GSSHG data first, this extends layer cache
## between subsequent builds (less to re-download)
##
FROM ubuntu:16.04
LABEL maintainer="[email protected]"

## Part 1: Base Image and Run Dependencies

ARG BIN_DEPS="wget libnetcdf11 libgdal1i libfftw3-3 libpcre3 liblapack3 graphicsmagick liblas3"
ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update					 &&\
    apt-get install -y					   \
    	    software-properties-common 			   \
	    python-software-properties 			   \
	    $BIN_DEPS					 &&\
    add-apt-repository universe        			 &&\
    apt-get purge -y                   		   \
            software-properties-common 			   \
	    python-software-properties 			 &&\
    apt-get autoremove -y 				 &&\
    rm -rf /var/lib/apt/lists/*          

## Part 2: GSHHG and DCW Base Data

ARG GSHHG_VERSION=2.3.7
ARG DCW_VERSION=1.1.4
ENV GMT_DCW_FTP=ftp://ftp.soest.hawaii.edu/dcw/dcw-gmt-$DCW_VERSION.tar.gz \
    GMT_GSHHG_FTP=ftp://ftp.soest.hawaii.edu/gshhg/gshhg-gmt-$GSHHG_VERSION.tar.gz

RUN   mkdir -p /opt/gmt					 &&\
      cd /opt/gmt				 	 &&\
      wget $GMT_DCW_FTP					 &&\
      wget $GMT_GSHHG_FTP		 		 &&\
      tar -xzf gshhg-gmt-$GSHHG_VERSION.tar.gz 		 &&\
      tar -xzf dcw-gmt-$DCW_VERSION.tar.gz 		 &&\
      rm -f *.tar.gz                       		   \
    

## Part 3: Download and Build GMT

ARG GMT_BRANCH=trunk
ARG GMT_CMAKE_ARGS=""	    
ARG BUILD_DEPS="subversion build-essential cmake libcurl4-gnutls-dev \
	    libnetcdf-dev libgdal1-dev libfftw3-dev libpcre3-dev \
	    liblapack-dev libblas-dev "

COPY install.sh /opt/gmt/install.sh

RUN apt-get update 					&&\
    apt-get install -y $BUILD_DEPS     			&&\
    (							  \
        cd /opt/gmt 					&&\	
    	./install.sh fetch_src 				&&\
	./install.sh build  				&&\
	./install.sh clean_src 				  \
    ) 		     	       				&&\
    apt-get purge  -y $BUILD_DEPS			&&\
    apt-get autoremove -y 				&&\
    rm -rf /var/lib/apt/lists/*

WORKDIR /workspace

ENTRYPOINT ["/usr/local/bin/gmt"]

P.S. I am also happy to transition the image hierarchy to the GMT organization

from gmt.

Hi @akshmakov thanks for the help! That sounds like what we needed. Our current scripts are in the ci folder and are called from .travis.yml. Do you think you could modify them to use your container instead?

Would you like to try doing this yourself? I know you're familiar with Github but I can help with anything you need. We're always glad to have more people involved 😃

We can think about moving the container to a GMT organization later.

from gmt.

@leouieda Sounds good, Im working on adapting my existing image for CI. Should have a PR to review in the next day or two after I run some tests.

from gmt.

@leouieda is there any chance ftp.soest.hawaii.edu has some rate limiting set up for Travis IP's? I am having ftp download fails that only happen on travis builds and not consistently, but never seen on my build farm or dockerhub directly.

Is there a mirror that I can try for GSHHG and DCW datasets?

from gmt.

Should have a PR to review in the next day or two after I run some tests.

Perfect!

Is there a mirror that I can try for GSHHG and DCW datasets?

@akshmakov yeah, there is something wrong with the ftp downloads. I'm using http to download from the source sites instead of the gmt site. See https://github.com/GenericMappingTools/gmt/blob/master/ci/travis-setup.sh

from gmt.

I think we should at least provide following docker images:

1. images for each release (e.g. 5.4.5, with GMT 5.4.5 installed)
2. images for latest GMT5 and GMT6
3. image for normal travis builds (with GMT dependencies installed and gshhg, dcw downloaded)
4. image for cron travis builds (with GMT dependencies, gshhg, dcw, python/sphinx, maybe latex)

from gmt.

Leaving this for @leouieda to comment on (he is on travel) as I don't know about docker images...

from gmt.

@seisman I've kind of changed my mind of the whole Docker thing:

1. There is no advantage to use a docker image on Travis under Linux. The main bottle neck is downloading the coastlines and compiling GMT. Both of these would still be needed (when downloading the docker image).
2. Using docker on Mac defeats the purpose of testing on Mac.

It might be useful to provide GMT enabled Docker images for end users who might want them. But might be too much work to maintain a version for master. Either way, probably better for another issue since this one is specifically for Travis.

from gmt.