netlify / build-image Goto Github PK

https://github.com/spf13/hugo/releases/tag/v0.17

I'm seeing this message in deploy details: Installing NPM modules using Yarn version 0.18.1. Is there any reason to use an old yarn version? Latest is 1.3.2.

Our current default version of hugo is quite old - almost halfway back in the version history (v0.17 vs current v0.30.2).

Could we do the thing we've done with node wherein we pin the old version for old sites and install a newer version to use by default when someone uses a build command hugo on a new site?

It would be a better user experience as very few new customers showing up today use old hugo.

This would need an associated migration in the database for existing repo settings, and some logic in the build image to respect it, but this seems like the sanest place to track all the work.

Request by @bep and his posse of thousands of hugo users :)

Hello,

Could you add to the readme how to clear build cache, as in your application.

Thanks

EDIT: I see that when we start again start-image.sh, the cache is cleared. How to keep it ? 💇‍♂️

right now you can set YARN_VERSION as desired but then you will almost certainly clash with the --ignore-optional we have as default to cope with problems in our current version.

When updating, make sure we do something conditional to preserve the behavior of older builds pinned at yarn 0.18.1

Any reason you're using such an old (2 years) version?

Hello,

When trying to build the image, I'm greeted with the following error:

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
The command '/bin/sh -c curl -sO http://www.vips.ecs.soton.ac.uk/supported/$LIBVIPS_VERSION_MAJOR.$LIBVIPS_VERSION_MINOR/vips-$LIBVIPS_VERSION.tar.gz &&   tar zvxf vips-$LIBVIPS_VERSION.tar.gz &&   cd vips-$LIBVIPS_VERSION &&   ./configure --enable-debug=no --enable-docs=no --without-python --without-orc --without-fftw --without-gsf $1 &&   make &&   make install &&   ldconfig' returned a non-zero code: 2

By replacing the ENV variables, I tried to manually download the file (http://www.vips.ecs.soton.ac.uk/supported/7.42/vips-7.42.3.tar.gz), and ended up with a 404.

By checking the github, it also seems this version is not available anymore and I'm quite blocked on how to move forward.

Thanks

Released on 🎄 day: https://www.ruby-lang.org/en/news/2017/12/25/ruby-2-5-0-released/

Inside a local docker machine, I do cd /opt/repo and then run my site's build script (using Grunt: grunt build). But is seems that the first build task, that tries to a temporary folder, fails on insufficient permissions:

Warning: Unable to delete "dist" file (EACCES, permission denied 'dist'). Use --force to continue.

I see that the user inside the machine is called buildbot, and all files in the repo have these permissions:

-rwxr-xr-x  1 1000 staff 2533 Jul 30 16:45 Gruntfile.js

Any ideas?

When:

• you use .nvmrc to set node version and have 6.11.1 in it
• and we save your dependencies
• future builds using those dependencies SOMETIMES fail with these errors:

Using version none of node
[...]
/opt/build/build.sh: line 120: npm: command not found

This is a link to a message in our (private) slack with a link to a customer's failed deploy example (customer's site name and URL's includes their business name so not publishing it here)

https://netlify.slack.com/archives/C0QFKKY2U/p1506701906000491

There is no steady repro case - clearing the cache helps "sometimes, for a while" - but the above linked build has verbose logging on so someone on Netlify's team can debug.

Hi! I would prefer to have fewer configuration files when possible. In the case of a Ruby / Jekyll site that already has a Gemfile, I'd rather not also add a .ruby-version to specify my desired Ruby version.

Heroku supports a declaration in the Gemfile that's similar to the gem … syntax.

ruby "~> 2.4.2"

I believe Travis CI also supports discovering this ruby version declaration in addition to a version in .travis.yml or in .ruby-version.

Maybe Netlify could support discovering a site's Ruby version in either place?

• .ruby-version
• Gemfile

❤️

Hello,
What is the exact command that will be used on netlify when there is a git webhook?

I want to ensure my build that I am testing will be done the same way on your end.

Not sure if this makes sense since the field can contain a range of versions rather than specifying an exact version, but maybe we could limit it to supporting only a definition for node that is a single version #?

This would be in addition to setting NODE_VERSION or .nvmrc since we obviously can't deprecate the existing methods without breaking a lot of builds.

what do you think @rybit ?

I think that there are extensive docs on the things engine could be used for - including yarn and npm version setting:

https://docs.npmjs.com/files/package.json#engines

since there is no way to NOT run bower install if there is a /bower.json - we should allow users to pass flags like we do for NPM and YARN

currently we re-run npm install in case the checksum of package.json changes:

https://github.com/netlify/build-image/blob/master/run-build-functions.sh#L114

We should do the same if package-lock.json exists and is changed, and someone who knows what they heck they're doing should decide if the presence of package-lock.json means that we don't even worry about changes to package.json at that point while this code gets written

@biilmann suggested this was reasonable

I suggest something like if [ ! .nvmrc ] ; then if [ .node-version ] use .node-version type processing, so we don't make a breaking change to anyone using nvmrc and happening to have a conflicting .node-version file

Hello,

I'm using the image to debug a deploy that is not working as expected (some of my dependencies are not correctly loaded, but this is irrelevant to this issue). Instead of pushing every test commit to Netlify, I'm running the docker image to see where it fails.

In my debugging process, I would also like to edit my scripts to pinpoint where the issue might lie. But I can't find any editor (vim, nano, emacs) available in the image for debugging. The $EDITOR variable is also empty.

Am I missing something or would you accept a PR to add vim?

Currently we only have a few locales, none non-US-english, installed in the build environment:

$ locale -a
C
C.UTF-8
en_US.utf8
POSIX

We have a request for French, but should probably just include all of the ones we can think of, since it's useful to people who aren't US-English speakers to be able to use the build environment in their native language and format things (like dates on blog posts) using system utilities.

Hello,

The current image uses the yarn v0.18.1, while the current stable version is v1.2.1. This specific old yarn version has an issue with the dependency tree (just like old npm versions) where some dependencies are not installed when doing a yarn install in specific configurations.

I'm experiencing this issue in my Netlify build (both on the hosted version and by running the image myself) that starting webpack will yield the following errors:

6:31:11 PM: + webpack --config webpack.prod.babel.js --progress
6:31:11 PM: module.js:491
    throw err;
    ^

Error: Cannot find module 'yargs'
    at Function.Module._resolveFilename (module.js:489:15)
    at Function.Module._load (module.js:439:25)
    at Module.require (module.js:517:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/opt/build/.yarn_cache/npm-webpack-3.5.5-3226f09fc8b3e435ff781e7af34f82b68b26996c/bin/webpack.js:16:13)
    at Module._compile (module.js:573:30)
    at Object.Module._extensions..js (module.js:584:10)
    at Module.load (module.js:507:32)
    at tryModuleLoad (module.js:470:12)
    at Function.Module._load (module.js:462:3)

Forcing yarn to v1.2.1 (by editing the run-build.sh) actually builds my website correctly, with all dependencies correctly installed.

I haven't found a way to set this value from netlify.toml, it seems I should be able to pass YARN_VERSION, but I can't find documentation on the syntax (should it be in [build]? in a specific [context]?). In any event, I think updating the default version of yarn can only be beneficial.

Hi,

I've been fighting a bit with getting GatsbyJS working on Netlify. The root issues is in gatsby: gatsbyjs/gatsby#2001 (comment)
Which is caused due to GatsbyJS using a global graphql. Meaning this isn't an error on your end.

As can be seen in the issue, the suggested solution is to use Yarn resolutions and force every yarn dependency to resolve to the same version.

However, I haven't been able to get Yarn 1.0 to run succesfully. Whenever I try to build it stumbles over a new missing dependency in the build flow:

UPDATE:
I've tried to reproduce it with a new starter gatsby project without luck. I'll keep digging on my end of things, but if you have any ideas I'm all ears

Can we add any ruby version that listen gem supports ?
Right now, I can't bundle any project with this gem.

Here is gem informations: https://rubygems.org/gems/listen

Thanks.

I've successfully used plain 'docker build' against the dockerfile in this repo in the past, but not sure if that is all you do to create the real build-image.

Could we get a line in README.md to explain how to do this?

this would solve issues like #40

Build command bundle install --without production fails to run, because Netlify uses its own variant. Build fails, because our production setup relies on pg gem, which is impossible to skip with Netlify.

Ported from the feature requests listing, seems to belong here instead:

Summary

We should support more php version the way we support node versions (with an easy config flag) and probably also some PHP dependency manager.

Details

We currently support an ancient (but stock for our ubuntu release) php version, 5.5. Via phpbrew we kiiind of support php 5.6. It's not something anyone would figure out how to use without some guidance, but it is at least possible

The current version of PHP is 7.2 and while this is a fairly recent release, it's also a major change (there was no production-quality php 6 release).

We should also support a dependency manager like Composer (think Ruby bundler for php)

Note: there are very few PHP users on our platform, so we are unlikely to pursue this - great opportunity for a contribution if it's something you need!

This repo is currently missing an editorconfig and thus has two tabs and spaces. Looking to get some discussion or push for an .editorconfig file to be added so that we're not mixing tabs/spaces 😄

https://github.com/yarnpkg/yarn

I feel like it would be simpler for me to do a test run by simply pulling the image:

docker pull netlify/build

Run a container, mounting my repo, and running the command I want to test:

docker run --rm -v $(pwd):/opt/repo -w /opt/repo netlify/build build yarn build-site

Currently, when I try to run this, I get the error:

/usr/local/bin/build: line 27: /usr/local/bin/run-build-functions.sh: No such file or directory  
Installing dependencies: node=6.10.2 ruby=2.1.2 yarn=0.18.1                                      
/usr/local/bin/build: line 34: install_dependencies: command not found                           
Installing missing commands                     
/usr/local/bin/build: line 37: install_missing_commands: command not found                       
Executing user command: yarn install            
/usr/local/bin/build: line 40: yarn: command not found                                           
Caching artifacts                               
/usr/local/bin/build: line 44: cache_artifacts: command not found   

I feel like this would be easily fixed by adding the following line here to the Dockerfile.

ADD run-build-functions.sh /usr/local/bin/run-build-functions.sh

Is there any reason why you chose to opt-out this critical file from the docker image?

Could you please add ruby 2.3.3 and 2.4.0 to the build image? Thanks!

At present it appears that we only "respect" NPM version in case:

• there is a package.json in the base dir (https://github.com/netlify/build-image/blob/master/run-build-functions.sh#L323)
• and don't run yarn (https://github.com/netlify/build-image/blob/master/run-build-functions.sh#L330)

Some customers would like to specify an NPM_VERSION flag and have it work when for instance they have a monorepo without a /package.json, and cd to another directory to use "npm run test" in a build shell script or makefile rather than as invoked from "installing automatically from package.json".

This is a rare use case, and a request rather than a demand, but at present I would like to track it to see how frequent it is, so it's marked needs-review until it bubbles up with some more requests and we can re-assess whether it is worth the effort to support directly.

Current workaround: intentionally install the npm version you need before running:

npm install npm@$NPM_VERSION -g && npm run test

requested by: https://app.intercom.io/a/apps/q245f50x/inbox/inbox/unassigned/conversations/14414749736

My node netlify build is hanging so I thought I'd try to run it in the docker image:

buildbot@b8dca7eb5755:/$ build npm install
/usr/local/bin/build: line 33: /.nvm/nvm.sh: No such file or directory

and trying something simpler:

buildbot@b8dca7eb5755:/$ build echo hai
/usr/local/bin/build: line 33: /.nvm/nvm.sh: No such file or directory

Any ideas what's going on?

build X does not work in the build-image:

$ docker run -it -v ${PWD}:/opt/repo netlify/build:latest /bin/bash
buildbot@a5628d0af185:/$ build hugo
Cloning into '/opt/buildhome/repo'...
done.
/usr/local/bin/build: line 27: /usr/local/bin/run-build-functions.sh: No such file or directory
Installing dependencies
/usr/local/bin/build: line 30: install_dependencies: command not found
Installing missing commands
/usr/local/bin/build: line 33: install_missing_commands: command not found
Executing user command: hugo
Caching artifacts
/usr/local/bin/build: line 40: cache_artifacts: command not found
buildbot@a5628d0af185:/$ find . -name run-build-functions.sh
find: `./etc/ssl/private': Permission denied
find: `./etc/polkit-1/localauthority': Permission denied
find: `./var/lib/php5': Permission denied
find: `./var/lib/polkit-1': Permission denied
find: `./var/cache/ldconfig': Permission denied
find: `./var/spool/cron/crontabs': Permission denied
find: `./var/spool/rsyslog': Permission denied
find: `./root': Permission denied
find: `./proc/tty/driver': Permission denied
buildbot@a5628d0af185:/$ exit

And, one cannot fix it:

$ cp ~/run-build-functions.sh .
$ git add run-build-functions.sh
$ git commit -m "add script"
[master 6782400] add script
 1 file changed, 2349 insertions(+)
 create mode 100644 run-build-functions.sh
$ docker run -it -v ${PWD}:/opt/repo netlify/build:latest /bin/bash
buildbot@624fc098b6cf:/$  cp /opt/repo/run-build-functions.sh /usr/local/bin
cp: cannot create regular file ‘/usr/local/bin/run-build-functions.sh’: Permission denied

speaking of this image:

netlify/build latest 525b8d552832 25 hours ago 3.27 GB

Preview deploys can take a long time to build during the "Deploying to CDN" phase, as they start from an empty site. In my case there are ~2500 files, and it takes >5mins even if only a few files are modified.

The image use for the build could be master image (or another one from a more recent commit on the build's branch), needing fewer files to be uploaded.

Also potentially, for some builds there are no files modified, and Netlify could skip the preview build publishing, because it is the same as a previous published build.

https://serverfault.com/questions/349460/how-to-move-files-between-two-s3-buckets-with-minimum-cost
(also much quicker)

This issue is pretty similar to #92, as I can't build the image using the Dockerfile in the repo. It is failing at the end of the file, when initiating phpbrew.

I have an error about curl not being available:

===> Configuring 5.6.31...


Use tail command to see what's going on:
   $ tail -F /opt/buildhome/.phpbrew/build/php-5.6.31/build.log


Error: Configure failed:
The last 5 lines in the log file:
checking whether to enable ctype functions... yes

checking for cURL support... yes

checking for cURL in default path... not found

configure: error: Please reinstall the libcurl distribution -

    easy.h should be in <curl-dir>/include/curl/

Please checkout the build log file for more details:
         tail /opt/buildhome/.phpbrew/build/php-5.6.31/build.log
The command '/bin/sh -c /bin/bash -c 'phpbrew init && source ~/.phpbrew/bashrc && phpbrew install 5.6 +default &&     phpbrew app get composer'' returned a non-zero code: 255

The full log (of the relevant part) is:

Step 44/45 : RUN /bin/bash -c 'phpbrew init && source ~/.phpbrew/bashrc && phpbrew install 5.6 +default &&     phpbrew app get composer'
 ---> Running in 1509c979022d
Using root: /opt/buildhome/.phpbrew
Initialization successfully finished!
<=====================================================>
Phpbrew environment is initialized, required directories are created under

    /opt/buildhome/.phpbrew

Paste the following line(s) to the end of your ~/.bashrc and start a
new shell, phpbrew should be up and fully functional from there:

    source /opt/buildhome/.phpbrew/bashrc

To enable PHP version info in your shell prompt, please set PHPBREW_SET_PROMPT=1
in your `~/.bashrc` before you source `~/.phpbrew/bashrc`

    export PHPBREW_SET_PROMPT=1

To enable .phpbrewrc file searching, please export the following variable:

    export PHPBREW_RC_ENABLE=1


For further instructions, simply run `phpbrew` to see the help message.

Enjoy phpbrew at $HOME!!

<=====================================================>
Downloading https://secure.php.net/releases/index.php?json&version=7&max=100 via php stream
Downloading https://secure.php.net/releases/index.php?json&version=5&max=100 via php stream
===> phpbrew will now build 5.6.31
===> Loading and resolving variants...
Downloading http://www.php.net/get/php-5.6.31.tar.bz2/from/this/mirror via php stream
===> Extracting /opt/buildhome/.phpbrew/distfiles/php-5.6.31.tar.bz2 to /opt/buildhome/.phpbrew/build/tmp.1509030898/php-5.6.31
===> Moving /opt/buildhome/.phpbrew/build/tmp.1509030898/php-5.6.31 to /opt/buildhome/.phpbrew/build/php-5.6.31
===> Checking patches...
Checking patch for replace apache php module name with custom version name
===> Configuring 5.6.31...


Use tail command to see what's going on:
   $ tail -F /opt/buildhome/.phpbrew/build/php-5.6.31/build.log


Error: Configure failed:
The last 5 lines in the log file:
checking whether to enable ctype functions... yes

checking for cURL support... yes

checking for cURL in default path... not found

configure: error: Please reinstall the libcurl distribution -

    easy.h should be in <curl-dir>/include/curl/

Please checkout the build log file for more details:
         tail /opt/buildhome/.phpbrew/build/php-5.6.31/build.log
The command '/bin/sh -c /bin/bash -c 'phpbrew init && source ~/.phpbrew/bashrc && phpbrew install 5.6 +default &&     phpbrew app get composer'' returned a non-zero code: 255

Also note that following #92, I've also commented the whole libvips part of the Dockerfile. It shouldn't have any impact on this issue, but I'm mentioning it for the sake of completeness.

I can not execute builds on the netlify platform, I receive the following error:

4:14:15 PM: Build command unable to start
4:14:15 PM: Error running command: fork/exec /usr/local/bin/build: no such file or directory

I ran a debug build based on this guide which worked without any issues.

I intended to use Node.js version 8.0.0 so added .nvmrc with 8.0.0 but build was failed.

capture:

It look like to me that $(cat .nvmrc) has broken a line.

After that I had set NODE_VERSION via Web GUI then build is successfully.
So this is just report. Thanks awesome service😹

...to support this pattern as used in bugsnag's webpack plugin:

https://github.com/jmshal/webpack-bugsnag-plugin/blob/245f0380b7c98275072a93a3a5a9b6e06e5fdbb5/src/helpers/git.js#L25

cc @biilmann and @rybit

Matt signed us up for this: https://app.intercom.io/a/apps/q245f50x/inbox/unassigned/conversations/6268411915

I have a gatsby app - it lives in the site/ directory of my repo. When I attempt to build the app with netlify I get:

/opt/build/build.sh: line 427: gatsby: command not found

I believe this is because npm install is never successfully run - when it runs it finds no package.json in the repo root.

Is it currently possible to build a subdirectory of the repo?

prisma-archive/prep#9

Builds that need to compile node-gyp will all struggle with this bug, no matter what version of yarn they use:

lovell/sharp#614

The "fix" is updating pkg-config, which I understand may be problematic. But this package is not particularly rare and everyone trying to use it will suffer. In particular a very dear well-paying customer is impacted (see https://app.intercom.io/a/apps/q245f50x/inbox/inbox/1479299/conversations/14675458658).

The error that is produced is this:

5:40:15 PM: readelf: Error: '/usr/local/lib  /libvips-cpp.so': No such file

and that extra space is due to a bug in pkg-config as shipped with ubuntu 14.04, but it beaks node-gyp's (admittedly byzantine) build sequence.

Is there a reason for Netlify to have a limited amount of Node versions available?

It would be great if it would be in sync with all releases, just like NVM
https://nodejs.org/en/download/releases/

Thanks!

This will allow projects using the recent github-pages gem to build without adjustment.

#110

This doesn't happen at present (as it does for Ruby, Node, and soon to be clojure)

Issue

PWD command not found on Ubuntu 16.04

Where

This line

Workaround

I changed this to be lowercase pwd

Not sure what the implications are for other OS's, but I can raise a PR if required

Similar to existing $YARN_VERSION functionality.

Trying to cut down on support requests "why doesn't proxying work?" "because your account level doesn't support it".

I get that this is hairy to implement, but it would be a help.

Low priority, though, compared to saving the world.

I've got a project that depends on exiftool, would love to know if it's possible to install. It's available via apt as libimage-exiftool-perl

Thanks!

The website interface supports the command grunt fetchJson && grunt compile but this local Docker instance doesn't. The run-build.sh echoes the command, here's a screen shot of me using the build function, and the echo is missing the command following &&:

Am I doing something wrong? The code in the shell script looks fine, but I'm not very savvy in there.

Hello,

According to the Docker file, PHP 5.6 is installed through phpbrew:
https://github.com/netlify/build-image/blob/master/Dockerfile#L227

But when I run php -v it return PHP 5.5.9-1ubuntu4.20 (cli): what is the right way to use PHP 5.6?

Thanks a lot!