rocker-org / ml Goto Github PK

View Code? Open in Web Editor NEW

50.0 50.0 13.0 163 KB

experimental machine learning container

License: GNU General Public License v2.0

Makefile 1.74% Dockerfile 29.79% Shell 64.03% R 4.43%

ml's People

Contributors

Stargazers

Watchers

Forkers

restonslacker adolfoeliazat noamross seabbs markedmondson1234 akoukoullis zojeda drroad mattwarkentin smwindecker d-bohn nagdevamruthnath fvd

ml's Issues

can we use --gpu in docker directly ie without Nvidia-docker?

is nvidia-docker is necessary anymore. With 19.03 and older, can you do (after installing nvidia-docker-cli and nvidia-docker-toolkit) pass --gpus= parameter to docker ?

Switching between python virtualevs (e.g. support greta / multiple tensorflow versions)

We had set the env var RETICULATE_PYTHON_ENV in the global environmental variable so that any user will have access to a system-wide python virtual environment.

Sometimes users will want a separate virtual environment, e.g. to use packages that require older / specific versions of tensorflow, such as greta (v0.3.1).

## set up required greta (0.3.1) env
library(reticulate)
virtualenv_install(envname="/opt/venv/greta", 
  packages = c("tensorflow==1.14.0", 
                         "tensorflow_probability==0.7.0") )

use_virtualenv("/opt/venv/greta")

## test this is 1.14.0 and not 2.0:
tensorflow::tf_version()

Unfortunately, this doesn't change the virtualenv, because it's been locked by the env var, requiring the user to override that setting, e..g by using a local .Renviron that sets RETICULATE_PYTHON_ENV=/opt/venv/greta.

A better solution is for us to set WORKON_HOME env var (e.g. to /opt/venv). This allows reticulate() to "discover" the default reticulate virtualenv we create in /opt/venv/reticulate, so we have an out-of-the-box environment, while the above commands can be run from a fresh R session to set up a new environment without having to restart the R session and mess with Renviron.

Note: when we don't force RETICULATE_PYTHON_ENV, reticulate;:py_config() harasses us to install miniconda instead of using the system Python:

reticulate::py_config()
No non-system installation of Python could be found.
Would you like to download and install Miniconda?

What does reticulate have against the system-installation of python? I get that reticulate() emphasizes user-level $HOME installs of everything over system-wide defaults, but this message seems a little too pushy given that it's already detected a perfectly viable system installation...

The h5py Python package is required to save and load models

So I am having an issue. I am trying to load in a model created in python, but I simply cannot get it to work. The error tells me I need to install the h5py package, but no matter how I do it (or try it) I cannot get it to work. Below is the code that leads to my problem:

On mac:
First I fire up the docker image
docker run --rm -d --name deeplearnr -p 8787 -e PASSWORD=learning_is_deep rocker/ml:3.6.0

Then I log into it using the default rstudio user.

In R:

> system("git clone https://github.com/jleinonen/ikea-names")
Cloning into 'ikea-names'...

> library(keras)
> my_model <- load_model_hdf5("ikea-names/models/model_default.h5")
Error in load_model_hdf5("ikea-names/models/model_default.h5") : 
  The h5py Python package is required to save and load models

Notes on upgrading cuda version on host

After upgrading the nvidia drivers on the host (e.g. with apt-get upgrade), nvidia tasks will fail to run due to driver mistmatch, e.g.:

 $ nvidia-smi
Failed to initialize NVML: Driver/library version mismatch

Rebooting the machine resolves this, but if that is not a convenient option for a server, this can be done manually: stop any running Xorg instances (e.g. sudo service stop gdm3) and then sudo rmmod nvidia. The latter may fail by listing submodules that are still running, so stop these as well, sudo rmmod nvidia-uvm. Then restart the nvidia drivers with sudo nvidia-smi to confirm GPU is back and running.

see:

Running nvidia-docker instances, e.g. with docker run --gpus all ... should now work again as before. Should add this to the user docs when we get to writing down more stuff about CUDA images...

rsession-ld-library-path

I find I need to write to the rsession-ld-library-path parameter in rserver.conf for a couple of packages, and I can see if you do that installing more stuff on top of Rocker images you could overwrite it. Do you think the rstudio image/script could include some run-time command to write LIBRARY_LD_PATH to this in a way that's careful, so that one doesn't have to worry about this at all?

gpg --keyserver fails because of blocked ports

My employer blocks a number of outbound ports so while working on #2, I had to make another modification to the Dockerfile as follows:

    && gpg --keyserver keyserver.ubuntu.com --recv-key E084DAB9 \

becomes

    && gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-key E084DAB9 \

this based on the solution here: https://superuser.com/questions/64922/how-to-work-around-blocked-outbound-hkp-port-for-apt-keys#222083

I'm happy to submit a PR, but wanted to wait until #2 is resolved and wanted to see how you feel about this modification.

Using rocker/ml-gpu image through Singularity on a HPC

Hi,

First off, Rocker is amazing and I am very thankful for the work thats been done to support R in Docker.

I have used several of your Docker images in the past, but in particular I am interested in using the rocker/ml-gpu image on my research institutes HPC in order to go GPU-based deep learning in R using tensorflow and keras.

Singularity provides the ability to run containers on a HPC, and supports Docker images for containerization. However, I am also aware that GPU-based Dockers require nvidia-docker. Do you know if the requirement for nvidia-docker runtime prohibits using rocker/ml-gpu or other GPU images through singularity on a HPC.

For context, the HPC runs CentOS 7 and Singularity 3.1.0. I am fairly new to GPU-based images and deep learning, so hopefully this question is clear.

How DO you add RStudio?

Hi, I've tried to add RStudio to the base R + TF + keras but have failed :) May I ask to take a look at my attempt? There is something about the $R_HOME folder that I don't understand - the pertinent bit is:

&& mkdir -p /etc/R \
  && mkdir -p /usr/local/lib/etc/R/ \
  ## Write config files in $R_HOME/etc
  && echo '\n\
    \n# Configure httr to perform out-of-band authentication if HTTR_LOCALHOST \
    \n# is not set since a redirect to localhost may not work depending upon \
    \n# where this Docker container is running. \
    \nif(is.na(Sys.getenv("HTTR_LOCALHOST", unset=NA))) { \
    \n  options(httr_oob_default = TRUE) \
    \n}' >> /usr/local/lib/etc/R/Rprofile.site \
  && echo "PATH=${PATH}" >> /usr/local/lib/etc/R/Renviron \

I added the && mkdir -p /usr/local/lib/etc/R/ \ as it was not finding the folder with the error:
/bin/sh: 1: cannot create /usr/local/lib/R/etc/Rprofile.site: Directory nonexistent

And I changed /usr/local/lib/R/etc/ to /usr/local/lib/etc/R/ as it seemed a typo, but trying to change it back now.

so it at least finishes building, but when I start the container I can't connect to RStudio, the logs say:

rsession: no process found

.. which I guess is related.

Dockerfile below with comments

# I updated to most recent version, not got far enough in the build to see if this is a problem yet
FROM nvidia/cuda:9.0-cudnn7-runtime

## This handle reaches Carl and Dirk
MAINTAINER "Carl Boettiger and Dirk Eddelbuettel" [email protected]

## Prepare R installation from 
RUN sh -c 'echo "deb http://cloud.r-project.org/bin/linux/ubuntu xenial/" >> /etc/apt/sources.list' \
    && gpg --keyserver keyserver.ubuntu.com --recv-key E084DAB9 \
    && gpg -a --export E084DAB9 | apt-key add -

RUN apt-get update \
    && apt-get upgrade -y -q \
    && apt-get install -y --no-install-recommends \
           r-base \
           r-base-dev \
           r-cran-littler \
           libxml2-dev \
           libxt-dev \
           libssl-dev \
           libcurl4-openssl-dev \
           imagemagick \
           python-pip \
           libpython2.7 \
    && python -m pip install --upgrade pip \
    && pip install virtualenv \
    && echo 'options(repos = c(CRAN = "https://cloud.r-project.org"))' >> /etc/R/Rprofile.site \
    && /usr/lib/R/site-library/littler/examples/install.r tensorflow keras \
    && r -e "keras::install_keras(tensorflow = '1.4.1-gpu')"

# copied from rstudio dockerfile https://github.com/rocker-org/rocker-versioned/tree/master/rstudio/3.5.2
ARG RSTUDIO_VERSION
#ENV RSTUDIO_VERSION=${RSTUDIO_VERSION:0.1.463}
ARG S6_VERSION
ENV S6_VERSION=${S6_VERSION:-v1.21.7.0}
ENV S6_BEHAVIOUR_IF_STAGE2_FAILS=2
ENV PATH=/usr/lib/rstudio-server/bin:$PATH

## Download and install RStudio server & dependencies
## Attempts to get detect latest version, otherwise falls back to version given in $VER
## Symlink pandoc, pandoc-citeproc so they are available system-wide
RUN apt-get update \
  && apt-get install -y --no-install-recommends \
    file \
    git \
    libapparmor1 \
    libcurl4-openssl-dev \
    libedit2 \
    libssl-dev \
    lsb-release \
    psmisc \
    procps \
    python-setuptools \
    sudo \
    wget \
  && wget -O libssl1.0.0.deb http://ftp.debian.org/debian/pool/main/o/openssl/libssl1.0.0_1.0.1t-1+deb8u8_amd64.deb \
  && dpkg -i libssl1.0.0.deb \
  && rm libssl1.0.0.deb \
  && RSTUDIO_LATEST=$(wget --no-check-certificate -qO- https://s3.amazonaws.com/rstudio-server/current.ver) \
  && [ -z "$RSTUDIO_VERSION" ] && RSTUDIO_VERSION=$RSTUDIO_LATEST || true \
  && wget -q http://download2.rstudio.org/rstudio-server-${RSTUDIO_VERSION}-amd64.deb \
  && dpkg -i rstudio-server-${RSTUDIO_VERSION}-amd64.deb \
  && rm rstudio-server-*-amd64.deb \
  ## Symlink pandoc & standard pandoc templates for use system-wide
  && ln -s /usr/lib/rstudio-server/bin/pandoc/pandoc /usr/local/bin \
  && ln -s /usr/lib/rstudio-server/bin/pandoc/pandoc-citeproc /usr/local/bin \
  && git clone https://github.com/jgm/pandoc-templates \
  && mkdir -p /opt/pandoc/templates \
  && cp -r pandoc-templates*/* /opt/pandoc/templates && rm -rf pandoc-templates* \
  && mkdir /root/.pandoc && ln -s /opt/pandoc/templates /root/.pandoc/templates \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/ \
  ## RStudio wants an /etc/R, will populate from $R_HOME/etc
  && mkdir -p /etc/R \
  && mkdir -p /usr/local/lib/etc/R/ \
  ## Write config files in $R_HOME/etc
  && echo '\n\
    \n# Configure httr to perform out-of-band authentication if HTTR_LOCALHOST \
    \n# is not set since a redirect to localhost may not work depending upon \
    \n# where this Docker container is running. \
    \nif(is.na(Sys.getenv("HTTR_LOCALHOST", unset=NA))) { \
    \n  options(httr_oob_default = TRUE) \
    \n}' >> /usr/local/lib/etc/R/Rprofile.site \
  && echo "PATH=${PATH}" >> /usr/local/lib/etc/R/Renviron \
  ## Need to configure non-root user for RStudio
  && useradd rstudio \
  && echo "rstudio:rstudio" | chpasswd \
	&& mkdir /home/rstudio \
	&& chown rstudio:rstudio /home/rstudio \
	&& addgroup rstudio staff \
  ## Prevent rstudio from deciding to use /usr/bin/R if a user apt-get installs a package
  &&  echo 'rsession-which-r=/usr/local/bin/R' >> /etc/rstudio/rserver.conf \
  ## use more robust file locking to avoid errors when using shared volumes:
  && echo 'lock-type=advisory' >> /etc/rstudio/file-locks \
  ## configure git not to request password each time
  && git config --system credential.helper 'cache --timeout=3600' \
  && git config --system push.default simple \
  ## Set up S6 init system
  && wget -P /tmp/ https://github.com/just-containers/s6-overlay/releases/download/${S6_VERSION}/s6-overlay-amd64.tar.gz \
  && tar xzf /tmp/s6-overlay-amd64.tar.gz -C / \
  && mkdir -p /etc/services.d/rstudio \
  && echo '#!/usr/bin/with-contenv bash \
          \n## load /etc/environment vars first: \
  		  \n for line in $( cat /etc/environment ) ; do export $line ; done \
          \n exec /usr/lib/rstudio-server/bin/rserver --server-daemonize 0' \
          > /etc/services.d/rstudio/run \
  && echo '#!/bin/bash \
          \n rstudio-server stop' \
          > /etc/services.d/rstudio/finish \
  && mkdir -p /home/rstudio/.rstudio/monitored/user-settings \
  && echo 'alwaysSaveHistory="0" \
          \nloadRData="0" \
          \nsaveAction="0"' \
          > /home/rstudio/.rstudio/monitored/user-settings/user-settings \
  && chown -R rstudio:rstudio /home/rstudio/.rstudio

COPY userconf.sh /etc/cont-init.d/userconf

## running with "-e ADD=shiny" adds shiny server
COPY add_shiny.sh /etc/cont-init.d/add
COPY disable_auth_rserver.conf /etc/rstudio/disable_auth_rserver.conf
COPY pam-helper.sh /usr/lib/rstudio-server/bin/pam-helper

EXPOSE 8787

## automatically link a shared volume for kitematic users
VOLUME /home/rstudio/kitematic

CMD ["/init"]

Update tensorflow to 2.0 and keras to 2.2.5

Not an Issue

Just a comment/Request
Tensorflow 2.0 and keras 2.2.5 has been released for a while. Is there a plan to update the versions? It seems fairly simple to update on dockerfile.

Default tensorflow install now requires CUDA 9

With the release of TF 1.5, the prebuilt binaries are built against CUDA 9 (https://github.com/tensorflow/tensorflow/releases/tag/v1.5.0) instead of CUDA 8. The Dockerfile should probably either derive from the 9.0-cudnn7-runtime container or keras should install version 1.4 (e.g. keras::install_tensorflow(version="1.4.0-gpu"). By default it appears that keras now installs 1.5.

reticulate python rstudio

Can we get rocker/ml and rocker/ml-gpu to work with reticulate on RStudio?

library(reticulate)
py_config()

python: /usr/local/bin/python
libpython: /usr/lib/python3.5/config-3.5m-x86_64-linux-gnu/libpython3.5.so
pythonhome: /usr:/usr
version: 3.5.3 (default, Sep 27 2018, 17:25:39) [GCC 6.3.0 20170516]
numpy: /usr/local/lib/python3.5/dist-packages/numpy
numpy_version: 1.16.3

python versions found:
/usr/local/bin/python
/usr/bin/python
/usr/bin/python3

py_install('pandas')

Error: Prerequisites for installing Python packages not available.

Please install the following Python packages before proceeding: pip, virtualenv

greta is not working

I was trying to play with greta using this container but I'm getting an error. This is what I am doing:

nvidia-docker run -it rocker/ml-gpu:latest bash

root@7dc3309926d4:/# nvidia-smi
Fri Apr 19 12:25:12 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.116                Driver Version: 390.116                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 106...  Off  | 00000000:01:00.0  On |                  N/A |
| 45%   42C    P0    27W / 120W |   1382MiB /  6076MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

root@7dc3309926d4:/# R

R version 3.5.2 (2018-12-20) -- "Eggshell Igloo"
Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> x <- iris$Petal.Length
> y <- iris$Sepal.Length
> library(greta)

Attaching package: 'greta'

The following objects are masked from 'package:stats':

    binomial, poisson

The following objects are masked from 'package:base':

    %*%, backsolve, beta, colMeans, colSums, diag, forwardsolve, gamma,
    rowMeans, rowSums, sweep

> int <- normal(0, 5)
> coef <- normal(0, 3)
> sd <- lognormal(0, 3)
> mean <- int + coef * x
> distribution(y) <- normal(mean, sd)
> m <- model(int, coef, sd)
> draws <- mcmc(m, n_samples = 1000)

/usr/local/lib/python3.5/dist-packages/numpy/lib/type_check.py:546: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  'a.item() instead', DeprecationWarning, stacklevel=1)
/usr/local/lib/python3.5/dist-packages/numpy/lib/type_check.py:546: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  'a.item() instead', DeprecationWarning, stacklevel=1)
/usr/local/lib/python3.5/dist-packages/numpy/lib/type_check.py:546: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  'a.item() instead', DeprecationWarning, stacklevel=1)
/usr/local/lib/python3.5/dist-packages/numpy/lib/type_check.py:546: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  'a.item() instead', DeprecationWarning, stacklevel=1)
/usr/local/lib/python3.5/dist-packages/numpy/lib/type_check.py:546: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  'a.item() instead', DeprecationWarning, stacklevel=1)
/usr/local/lib/python3.5/dist-packages/numpy/lib/type_check.py:546: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  'a.item() instead', DeprecationWarning, stacklevel=1)
/usr/local/lib/python3.5/dist-packages/numpy/lib/type_check.py:546: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  'a.item() instead', DeprecationWarning, stacklevel=1)
/usr/local/lib/python3.5/dist-packages/numpy/lib/type_check.py:546: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  'a.item() instead', DeprecationWarning, stacklevel=1)
/usr/local/lib/python3.5/dist-packages/numpy/lib/type_check.py:546: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  'a.item() instead', DeprecationWarning, stacklevel=1)
/usr/local/lib/python3.5/dist-packages/numpy/lib/type_check.py:546: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  'a.item() instead', DeprecationWarning, stacklevel=1)
/usr/local/lib/python3.5/dist-packages/numpy/lib/type_check.py:546: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  'a.item() instead', DeprecationWarning, stacklevel=1)
/usr/local/lib/python3.5/dist-packages/numpy/lib/type_check.py:546: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  'a.item() instead', DeprecationWarning, stacklevel=1)
Error: greta hit a tensorflow error:

Error in py_call_impl(callable, dots$args, dots$keywords): NotFoundError: ./libdevice.compute_30.10.bc not found
	 [[{{node cluster_0_1/xla_compile}} = _XlaCompile[Nresources=0, Targs=[DT_DOUBLE, DT_DOUBLE, DT_DOUBLE, DT_DOUBLE, DT_DOUBLE, DT_DOUBLE, DT_DOUBLE, DT_DOUBLE, DT_DOUBLE], Tconstants=[DT_INT32, DT_INT32, DT_INT32, DT_INT32, DT_INT32, DT_INT32], function=cluster_0[_XlaCompiledKernel=true, _XlaNumConstantArgs=6, _XlaNumResourceArgs=0], _device="/job:localhost/replica:0/task:0/device:GPU:0"](Const, Tile_3/multiples/1, Reshape/shape, strided_slice_3/stack, strided_slice_3/stack_1, Sum_1/reduction_indices, _arg_Placeholder_0_0/_3, _arg_Placeholder_1_0_1/_5, _arg_Placeholder_2_0_2/_7, _arg_Placeholder_3_0_3/_9, _arg_Placeholder_4_0_4/_11, _arg_Placeholder_5_0_5/_13, _arg_Placeholder_6_0_6/_15, _arg_Placeholder_7_0_7/_17, _arg_Placeholder_8_0_8/_19)]]
	 [[{{node cluster_0_1/xla_run/_1}} = _Recv[client_terminated=false, recv_device="/job:localh

link libnvblas?

libnvblas.so gets installed with the existing cuda libraries. Apparently this can be enabled as the drop-in BLAS library for R, and is smart enough to let openblas handle things and only take over when it can provide significant acceleration(?)

EDIT

Haven't found great documentation on setup or performance, but looks like this can be done as a one-off at runtime by setting LD_PRELOAD and configuring the fallback to openblas:

## create config file:
echo "NVBLAS_LOGFILE nvblas.log
NVBLAS_CPU_BLAS_LIB /usr/lib/libopenblas.so
NVBLAS_GPU_LIST ALL" > /etc/nvblas.conf

Run R with these env vars:

NVBLAS_CONFIG_FILE=/etc/nvblas.conf LD_PRELOAD=/usr/local/cuda/lib64/libnvblas.so.9.0 R

Will have to benchmark a bit, but maybe worth adding this into our cuda/base setup @noamross ?

> library(keras)
> 
> mnist <- dataset_mnist()
ImportError: No module named keras
Use the install_keras() function to install the core Keras library
Error: Error loading Python module keras

Upgrading pip in Dockerfile causes "cannot import name main" error

Using this Dockerfile from the gpu folder:

RUN apt-get update \
    && apt-get upgrade -y -q \
    && apt-get install -y --no-install-recommends \
           r-base \
           r-base-dev \
           r-cran-littler \
           libxml2-dev \
           libxt-dev \
           libssl-dev \
           libcurl4-openssl-dev \
           imagemagick \
           python-pip \
           libpython2.7 \
    && pip install --upgrade pip==9.0.3 \
    && pip install virtualenv \
    && echo 'options(repos = c(CRAN = "https://cloud.r-project.org"))' >> /etc/R/Rprofile.site \
    && /usr/lib/R/site-library/littler/examples/install.r tensorflow keras \
    && r -e "keras::install_keras(tensorflow = '1.4.1-gpu')"

Which resulted in this error:

...snip...
Collecting pip
  Downloading https://files.pythonhosted.org/packages/d7/41/34dd96bd33958e52cb4da2f1bf0818e396514fd4f4725a79199564cd0c20/pip-19.0.2-py2.py3-none-any.whl (1.4MB)
Installing collected packages: pip
  Found existing installation: pip 8.1.1
    Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed pip-19.0.2
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main
The command '/bin/sh -c apt-get update     && apt-get upgrade -y -q     && apt-get install -y --no-install-recommends            r-base            r-base-dev            r-cran-littler            libxml2-dev            libxt-dev            libssl-dev            libcurl4-openssl-dev            imagemagick            python-pip            libpython2.7     && pip install --upgrade pip     && pip install virtualenv     && echo 'options(repos = c(CRAN = "https://cloud.r-project.org"))' >> /etc/R/Rprofile.site     && /usr/lib/R/site-library/littler/examples/install.r tensorflow keras && r -e "keras::install_keras(tensorflow = '1.4.1-gpu')"' returned a non-zero code: 1
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: exit status 1

The solution as suggested here pypa/pip#5240 is to change the install line.

    && python -m pip install --upgrade pip \