Giter VIP home page Giter VIP logo

napalm's Introduction

Napalm

This project is looking for a new maintainer, see

When faced with a JavaScript codebase, napalm is just what you need.

-- anonymous

Table of contents

Building npm packages in Nix with Napalm

Basic Napalm usage

Use the buildPackage function provided in the default.nix for building npm packages (replace <napalm> with the path to napalm; with niv: niv add nmattia/napalm):

let
    napalm = pkgs.callPackage <napalm> {};
in napalm.buildPackage ./. {}

All executables provided by the npm package will be available in the derivation's bin directory.

NOTE: napalm uses the package's package-lock.json (or npm-shrinkwrap.json) for building a package database. Make sure there is either a package-lock.json or npm-shrinkwrap.json in the source. Alternatively provide the path to the package-lock file:

let
    napalm = pkgs.callPackage <napalm> {};
in napalm.buildPackage ./. { packageLock = <path/to/package-lock>; }

Napalm with Nix flakes

If you want to use Napalm in your flake project, you can do that by adding it to your inputs and either passing napalm.overlays.default to your Nixpkgs instance, or by using the napalm.legacyPackages buildPackage output. To configure the latter's environment, be sure to look at the complicated scenarios and potentially set the nixpkgs input of napalm with follows.

Example flake.nix

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
  inputs.napalm.url = "github:nix-community/napalm";

  # NOTE: This is optional, but is how to configure napalm's env
  inputs.napalm.inputs.nixpkgs.follows = "nixpkgs";

  outputs = { self, nixpkgs, napalm }: 
  let
    system = "x86_64-linux";
    pkgs = nixpkgs.legacyPackages."${system}";
  in {
    # Assuming the flake is in the same directory as package-lock.json
    packages."${system}".package-name = napalm.legacyPackages."${system}".buildPackage ./. { };

    devShells."${system}".shell-name = pkgs.mkShell {
      nativeBuildInputs = with pkgs; [ nodejs ];
    };
  };
}

Flake Template

There is also a template that can help you use napalm in your project. You can use it in a new, empty directory by running:

nix flake init -t "github:nix-community/napalm"

Handling complicated scenarios with Napalm

Examples below assume that you have imported napalm in some way.

Custom node.js version

Napalm makes it quite simple to use custom node.js (with npm) version. This is controlled via nodejs argument.

Example 1

Changing node.js version to the one that is supplied in nixpkgs:

{ napalm, nodejs-16_x, ... }:
napalm.buildPackage ./. {
	nodejs = nodejs-16_x;
}

Example 2

Changing node.js version to some custom version (just an idea):

{ napalm, nodejs-12_x, ... }:
let
	nodejs = nodejs-12_x.overrideAttrs (old: rec {
		pname = "nodejs";
		version = "12.19.0";
		sha256 = "1qainpkakkl3xip9xz2wbs74g95gvc6125cc05z6vyckqi2iqrrv";
		name = "${pname}-${version}";

		src = builtins.fetchurl {
			url =
			"https://nodejs.org/dist/v${version}/node-v${version}.tar.xz";
			inherit sha256;
		};
	});
in
napalm.buildPackage ./. {
	inherit nodejs;
}

Pre/Post Npm hooks

Napalm allows to specify commands that are run before and after every npm call. These hooks work also for nested npm calls thanks to npm override mechanism.

Example

Patching some folder with executable scripts containing shebangs (that may be generated by npm script):

{ napalm, ... }:
napalm.buildPackage ./. {
	postNpmHook = ''
	patchShebangs tools
	'';
}

Multiple package locks

Napalms allows to specify multiple package locks. This may be useful for some project which consist of some smaller projects.

Example

{ napalm, ... }:
napalm.buildPackage ./. {
	# package-lock.json that is in the root of the project
	# is not required to be specified in `additionalpackagelocks`
	# If you want to specify it, you can use `packageLock` argument.
	additionalPackageLocks = [
	./frontend/package-lock.json
	./tests/package-lock.json
	];
}

Patching npm packages (before fetching them with npm)

This is very useful for errors like: Invalid interpreter

Napalm has an ability to patch fetched npm packages before serving them to the npm. By default patching fixes shebangs and binaries that are localized and the tarballs. Napalm also updates package-lock.json with new integrity hashes.

Example

To enable patching, just use:

{ napalm, ... }:
napalm.buildPackage ./. {
	patchPackages = true;
}

This will force repacking of all dependencies, though, so you might want to patch only specific dependencies by passing an empty attribute set to the next method.

Customizing patching mechanism of npm packages

Sometimes it is required to manually patch some package. Napalm allows that via customPatchPackages attribute. This attribute is a set of that overrides for packages that will be patched.

Example

{ napalm, ... }:
napalm.buildPackage ./. {
	# Arguments that are passed to the overrider:
	# `pkgs` - Nixpkgs used by Napalm
	# `prev` - Current set that will be passed to mkDerivation
	customPatchPackages = {
		"react-native" = {
			"0.65.0" = pkgs: prev: {
				EXAMPLE_ENV_VAR = "XYZ";
				dontBuild = false;
				buildPhase = ''
				# You can copy some stuff here or run some custom stuff
				'';
			};
		};

		# Version is not required. When it is not specified it
		# applies override to all packages with that name.
		"node-gyp-builder" = pkgs: prev: { };
	};
}

How does Napalm work ?

These are general steps that Napalm makes when building packages (if you want to learn more, see source code of default.nix):

  1. Napalm loads all package-lock.json files and parses them. Then it fetches all specified packages into the Nix Store.
  2. (optional) Napalm patches npm packages and stores their output in new location. Then uses this location as default package location in Nix Store.
  3. Napalm creates snapshot that consists of packages names, version and paths to locations in Nix Store that contain them.
  4. (optional) Napalm patches package-lock.json integrity if the packages were patched, so that they will work with npm install.
  5. Napalm sets up napalm-registry which as a main argument accepts snapshot of npm packages and them serves them as if it was npm registry server.
  6. Napalm sets up npm so that it thinks napalm-registry server is default npm registry server.
  7. Napalm overrides npm which allows using custom npm hooks (every time it is called) as well as some other default patching activities.
  8. Napalm calls all the npm commands.
  9. Napalm installs everything automatically or based on what was specified in installPhase.

Napalm - a lightweight npm registry

Under the hood napalm uses its own package registry. The registry is available in default.nix as napalm-registry.

Usage: napalm-registry [-v|--verbose] [--endpoint ARG] [--port ARG] --snapshot ARG

Available options:
  -v,--verbose             Print information about requests
  --endpoint ARG           The endpoint of this server, used in the Tarball URL
  --port ARG               The to serve on, also used in the Tarball URL
  --snapshot ARG           Path to the snapshot file. The snapshot is a JSON
                           file. The top-level keys are the package names. The
                           top-level values are objects mapping from version to
                           the path of the package tarball. Example: { "lodash":
                           { "1.0.0": "/path/to/lodash-1.0.0.tgz" } }
  -h,--help                Show this help text

Similar projects

napalm's People

Contributors

adisbladis avatar aszlig avatar basvandijk avatar cyntheticfox avatar ggreif avatar gobengo avatar gytis-ivaskevicius avatar jtojnar avatar mazurel avatar mdarocha avatar mic92 avatar nmattia avatar paulyoung avatar pikajude avatar scintill avatar sternenseemann avatar willibutz avatar zimbatm avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

napalm's Issues

Example usage inside flake.nix

I'm trying to use napalm inside my flake.nix. However I cannot understand how overlays are supposed to work this way. I kind of expected a lib function. The readme only talks about using niv, but my project already uses flakes as-is. Could you give an example in the readme?

How to pass custom pinned Node.js

Hi,

Sorry for this basic question - I'm still learning Nix/NixOS and I've not used niv at all. I want to pass my custom and pinned nodejs and build a Node.js package based on that.

Let me start with signature reading.

The main napalm signature is:

 { pkgs ? import ./nix {} }:

I'm not able to easily deduce what are the real inputs for napalm from that and how to pass nodejs. In ./nix/default.nix I find this expression:

let
  sources = import ./sources.nix;
  pkgs = import sources.nixpkgs {};
in
  pkgs // { inherit sources; nodejs = pkgs.nodejs-10_x; }

Which again doesn't say me more what are the inputs (it even includes the whole custom nixpkgs) but provides a hint about nodejs.

  1. Can I assume that sources together with sources.nixpkgs are the internal napalm dependencies and that I should not really care about them?

Going back to the question. I was able to pass nodejs to napalm by using this code:

let
  napalmSrc = pkgs.fetchFromGitHub {
    rev = "...";
    owner = "nmattia";
    repo = "napalm";
    sha256 = "...";
  };
  napalmNix = import "${naplmSrc}/nix" {};
  napalm = import napalmSrc { pkgs = napalmNix // { nodejs = myNodejs; }; };
in
  napalm.buildPackage ./. {}
  1. Is the above the correct way to pass my custom nodejs version? Is it be possible to sipmlify it somehow?

Thanks again for building this nice tool!

Support scoped packages

npm supports so-called scoped packages, whose name has the following form: @somescope/somepackagename. They are used all over the modern NPM ecosystem, most notably by the Babel transpiler. I tried implementing it but unfortunately, it does not seem to be expressible in Servant’s types at the moment.

There is an issue about adding more powerful types to Servant: haskell-servant/servant#962 but I doubt it will be resolved soon.

Would you accept a pull request replacing servant with some less type safe but more flexible library (e.g. scotty)?

Missing LICENSE file

The repo as is doesn't look to have a LICENSE with it to declare the terms-of-use

Derivation names not sanitized

Trying to package https://github.com/ActivityWatch/aw-webui, which has the following

    "infer-owner": {
      "version": "1.0.4",
      "resolved": "https://registry.npm.taobao.org/infer-owner/download/infer-owner-1.0.4.tgz?cache=0&other_urls=https%3A%2F%2Fregistry.npm.taobao.org%2Finfer-owner%2Fdownload%2Finfer-owner-1.0.4.tgz",
      "integrity": "sha1-xM78qo5RBRwqQLos6KPScpWvlGc=",
      "dev": true
    },

in dependencies of package-lock.json results in the following Nix error:

store path 'ffyfcwnv2p2xxwsy1f5d9ib4yvq1dfpr-infer-owner-1.0.4.tgz?cache=0&other_urls=https%3A%2F%2Fregistry.npm.taobao.org%2Finfer-owner%2Fdownload%2Finfer-owner-1.0.4.tgz' contains illegal character '&'

In some envrionemtns fails with getaddrinfo ENOTFOUND localhost

Sorry for the weird issue, but I don't really know where to start debugging. My build fails on GitLab (no sandboxing) with a really strange error. Builds on my laptop NixOS (with or without sandboxing) are fine. Building on my laptop using the local GitLab runner does fail.

npm WARN [email protected] No repository field.
npm WARN [email protected] No license field.
npm verb type system
npm verb stack FetchError: request to http://localhost:46645/abab failed, reason: getaddrinfo ENOTFOUND localhost
npm verb stack     at ClientRequest.<anonymous> (/nix/store/nm15xz8bigq33d3x8r8lfrdlnvspmjrm-nodejs-14.15.4/lib/node_modules/npm/node_modules/node-fetch-npm/src/index.js:68:14)
npm verb stack     at ClientRequest.emit (events.js:315:20)
npm verb stack     at Socket.socketErrorListener (_http_client.js:469:9)
npm verb stack     at Socket.emit (events.js:315:20)
npm verb stack     at emitErrorNT (internal/streams/destroy.js:106:8)
npm verb stack     at emitErrorCloseNT (internal/streams/destroy.js:74:3)
npm verb stack     at processTicksAndRejections (internal/process/task_queues.js:80:21)
npm verb cwd /tmp/nix-build-kevincox-web-compiler-0.0.0.drv-0/kevincox-web-compiler
npm verb Linux 4.19.78-coreos
npm verb argv "/nix/store/nm15xz8bigq33d3x8r8lfrdlnvspmjrm-nodejs-14.15.4/bin/node" "/nix/store/nm15xz8bigq33d3x8r8lfrdlnvspmjrm-nodejs-14.15.4/bin/npm" "install" "--loglevel" "verbose"
npm verb node v14.15.4
npm verb npm  v6.14.10
npm ERR! code ENOTFOUND
npm ERR! errno ENOTFOUND
npm ERR! network request to http://localhost:46645/abab failed, reason: getaddrinfo ENOTFOUND localhost
npm ERR! network This is a problem related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network 
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly.  See: 'npm help config'
npm verb exit [ 1, true ]https://gitlab.com/kevincox/nix-ci/-/blob/v1/default.nix
npm timing npm Completed in 5303ms

There is a localhost entry in the hosts file for both IPv4 and IPv6.

$ cat /etc/hosts
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
127.0.0.1	0hshit.hopto.org
127.0.0.1	daymndaymn.myftp.org
127.0.0.1	loba.webhop.me
172.17.0.3	runner-fa6cab46-project-24138563-concurrent-0

Links:

This seems to be stemming from the custom image that the build runs in but I haven't been able to identify the cause yet. https://gitlab.com/kevincox/nix-ci/-/blob/v1/default.nix

I'm opening this bug to help anyone else seeing a similar problem and ideally collect debugging tips and information. Feel free to close if you consider it off-topic.

One other question: Do you think it would be possible to use a loopback IP directly instead of localhost? It would remove a dependency in the stack even if I should get a proper environment set up.

Building -bin packages

This is more of a reminder for myself.

Dependencies like node-sass currently don't work out of the box. They need python2 and the nodejs headers to be symlinked in the right place so that they can build the extension from source.

Some other packages like gifcycle try to download pre-built binaries from the Internet (which fails because/thanks to the sandbox). It would be nice if the package could advertise those URLs in their metadata somehow.

Some other packages (like ???) probably ship with the pre-built binary embedded into the package. In that case we might want to run autopatchElf before running the npm postinstall hook.

Fails on lockfileVersion: 3

When running there are many errors of the following form as well as a lot of gobbledygook as it appears that multiple messages are being character-by-character concurrently.

No such tarball
CallStack (from HasCallStack):
  error, called at Main.hs:212:8 in main:Main
No such tarball

To reproduce:

  1. Download https://gitlab.com/kevincox/filepush/-/commit/08f97910ef4d8e99fa1878938ecd2c2edea1398d as a publicly-available example. (Exact commit is not relevant, just pinning a reference)
  2. Run nix-build and observe that it succeeds.
  3. Run npm install--lockfile-version 3 --package-lock-only.
  4. Run nix-build and observe that it fails.

jitsi-meet build fails, ".staging" directories ENOENT

I have tried to build jitsi-meet from source with napalm. I am only interested in the web parts but the repository also contains "react-native" code for iOS and Android.

[...]
jitsi-meet> Runnig npm command: npm install
jitsi-meet> npm WARN tar ENOENT: no such file or directory, open '/build/source/node_modules/.staging/dropbox-7d59b16d/dist/DropboxTeam-sdk.min.d.ts'
jitsi-meet> npm WARN tar ENOENT: no such file or directory, open '/build/source/node_modules/.staging/dropbox-7d59b16d/dist/DropboxTeam-sdk.min.js'
jitsi-meet> npm WARN tar ENOENT: no such file or directory, open '/build/source/node_modules/.staging/@react-native-community/cli-platform-android-df898059/build/commands/runAndroid/g
[... many more ...]
jitsi-meet> npm WARN tar ENOENT: no such file or directory, open '/build/source/node_modules/.staging/bc-css-flags-73d17359/lib/region-flags/svg/US-IA.svg'
jitsi-meet> npm ERR! code ENOENT5mtar ENOENT: no such file or directory, opeope
jitsi-meet> npm ERR! syscall spawn git
jitsi-meet> npm ERR! path git
jitsi-meet> npm ERR! errno ENOENT
jitsi-meet> npm ERR! enoent Error while executing:
jitsi-meet> npm ERR! enoent undefined ls-remote -h -t ssh://[email protected]/jitsi/rnnoise-wasm.git
jitsi-meet> npm ERR! enoent
jitsi-meet> npm ERR! enoent
jitsi-meet> npm ERR! enoent spawn git ENOENT
jitsi-meet> npm ERR! enoent This is related to npm not being able to find a file.
jitsi-meet> npm ERR! enoent
jitsi-meet> 
jitsi-meet> npm ERR! A complete log of this run can be found in:
jitsi-meet> npm ERR!     /build/tmp.fIXNtDnhqr/.npm/_logs/2020-04-28T09_12_28_142Z-debug.log
jitsi-meet> npm install: failure, aborting

Note that if I run npm install (0.12) outside of nix, there are no .staging directories. But apparently, it is normal that npm first downloads new modules there.

To reproduce: either check out napalm-jitsi-meet or follow these steps:

niv init
niv add nmattia/napalm
niv add jitsi/jitsi-meet

default.nix:

{ nixpkgs ? <nixpkgs>
, pkgs ? import nixpkgs {}
, sources ? import ./nix/sources.nix
, napalmSrc ? sources.napalm
, napalm ? pkgs.callPackage napalmSrc {}
, jitsiMeetSrc ? sources.jitsi-meet
}:

napalm.buildPackage jitsiMeetSrc {}

Execute nix build -L

node_modules get rebuilt with the main derivation

Whenever the source changes, the node_modules get rebuilt. For development this can add quite a lot to the iteration time.

What would you think of building the node_modules in a separate derivation and then symlinking it into the main derivation during build? We are doing something similar in yarn2nix.

Support for `yarn.lock`

I have worked on some projects where yarn.lock is used and teams seem to be quite attached to yarn so it seems unlikely that they would switch away from it completely. Does it make sense to support yarn.lock as well for source of dependencies?

napalm-registry maybe should listen on ::1 as well as on 127.0.0.1?

I just ran into an interesting issue (and arguably, an impurity in flakes on darwin) while trying to build https://github.com/ejgallego/coq-lsp?submodules=1#packages.aarch64_darwin.vscode-extension.

The symptom of the problem is that nix build -L hangs with npm error messages like:

...
coq-lsp> npm verb audit error FetchError: request to http://localhost:60584/-/npm/v1/security/audits/quick failed, reason: connect ECONNREFUSED ::1:60584

and then npm and napalm need to be force-killed in order to get the nix builder to exit (even after hitting Ctrl-C to interrupt the build).

Digging into the logs further, I noticed that although napalm is nominally binding to "localhost" while npm is connecting to "::1", which ought to work, it turns out that napalm is actually only listening on 127.0.0.1:

$ sudo lsof -i -n -P | grep LISTEN | grep napalm
napalm-re 44648       _nixbld1    3u  IPv4 0x2da8d7fea15bbb1f      0t0    TCP 127.0.0.1:60584 (LISTEN)

Anyway, after noodling on this for a bit, I took a look at /etc/hosts:

$ cat /etc/hosts
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1       localhost
255.255.255.255 broadcasthost
::1             localhost

and, sure enough, commenting out the ::1 entry enables the build to succeed.

Nix flake check is failing

This repo provides a basic flake.nix file, but it looks like it is not working correctly when it comes to nix flake check. Running it like nix flake check "github:nix-community/napalm" --no-write-lock-file results in error:

error: a 'aarch64-linux' with features {} is required to build '/nix/store/qmkfj51wpz34yrwl4zhavl7q02gvcmql-cabal2nix-napalm-registry.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}

From my tests, it looks like this error does not occur only when supported systems is only x86_64-linux. This issue can be easily fixed by supporting only this system, although I do not know how the maintainers would see this solution. I would also suggest using pinned nixpkgs as flake inputs so that the flake would be "safer" aka using flake.lock.

In case the solution that I suggested is acceptable, I can make PR with proper changes.

Installing `sqlite3` ir `better-sqlite3` fails

Installing this dependency fails with something along these lines. I have tried multiple workarounds with customPatchPackages and patchPackages with no luck. I am also having no luck with alternatives like npmlock2nix or nix-npm-buildpackage. Anyone has success with these?

npm info run [email protected] install node_modules/better-sqlite3 prebuild-install || node-gyp rebuild --release
npm info run [email protected] install { code: 1, signal: null }nst
npm timing reify:rollback:createSparse Completed in [email protected] inst
npm timing reify:rollback:retireShallow Completed in 0ms
npm timing command:install Completed in 1104ms
npm verb stack Error: command failed
npm verb stack     at ChildProcess.<anonymous> (/nix/store/6g5y6hq3npg9gy04q4ya1xzz9nj0bh59-nodejs-16.15.0/lib/node_modules/npm/node_modules/@npmcli/promise-spawn/index.js:64:27)
npm verb stack     at ChildProcess.emit (node:events:527:28)
npm verb stack     at maybeClose (node:internal/child_process:1092:16)
npm verb stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)
npm verb pkgid [email protected]
npm verb cwd /build/server
npm verb Linux 6.1.34
npm verb argv "/nix/store/6g5y6hq3npg9gy04q4ya1xzz9nj0bh59-nodejs-16.15.0/bin/node" "/nix/store/6g5y6hq3npg9gy04q4ya1xzz9nj0bh59-nodejs-16.15.0/bin/npm" "install" "--loglevel" "verbose" "--nodedir=/nix/store/6g5y6hq3npg9gy04q4ya1xzz9nj0bh59-nodejs-16.15.0/include/node"
npm verb node v16.15.0
npm verb npm  v8.5.5
npm ERR! code 1
npm ERR! path /build/server/node_modules/better-sqlite3
npm ERR! command failed
npm ERR! command sh -c prebuild-install || node-gyp rebuild --release
npm ERR! sh: /build/server/node_modules/.bin/prebuild-install: /usr/bin/env: bad interpreter: No such file or directory
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp verb cli [
npm ERR! gyp verb cli   '/nix/store/6g5y6hq3npg9gy04q4ya1xzz9nj0bh59-nodejs-16.15.0/bin/node',
npm ERR! gyp verb cli   '/nix/store/6g5y6hq3npg9gy04q4ya1xzz9nj0bh59-nodejs-16.15.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js',
npm ERR! gyp verb cli   'rebuild',
npm ERR! gyp verb cli   '--release'
npm ERR! gyp verb cli ]
npm ERR! gyp info using [email protected]
npm ERR! gyp info using [email protected] | linux | x64
npm ERR! gyp verb command rebuild []
npm ERR! gyp verb command clean []
npm ERR! gyp verb clean removing "build" directory
npm ERR! gyp verb command configure []
npm ERR! gyp verb find Python Python is not set from command line or npm configuration
npm ERR! gyp verb find Python Python is not set from environment variable PYTHON
npm ERR! gyp verb find Python checking if "python3" can be used
npm ERR! gyp verb find Python - executing "python3" to get executable path
npm ERR! gyp verb find Python - "python3" is not in PATH or produced an error
npm ERR! gyp verb find Python checking if "python" can be used
npm ERR! gyp verb find Python - executing "python" to get executable path
npm ERR! gyp verb find Python - "python" is not in PATH or produced an error
npm ERR! gyp ERR! find Python 
npm ERR! gyp ERR! find Python Python is not set from command line or npm configuration
npm ERR! gyp ERR! find Python Python is not set from environment variable PYTHON
npm ERR! gyp ERR! find Python checking if "python3" can be used
npm ERR! gyp ERR! find Python - "python3" is not in PATH or produced an error
npm ERR! gyp ERR! find Python checking if "python" can be used
npm ERR! gyp ERR! find Python - "python" is not in PATH or produced an error
npm ERR! gyp ERR! find Python 
npm ERR! gyp ERR! find Python **********************************************************
npm ERR! gyp ERR! find Python You need to install the latest version of Python.
npm ERR! gyp ERR! find Python Node-gyp should be able to find and use Python. If not,
npm ERR! gyp ERR! find Python you can try one of the following options:
npm ERR! gyp ERR! find Python - Use the switch --python="/path/to/pythonexecutable"
npm ERR! gyp ERR! find Python   (accepted by both node-gyp and npm)
npm ERR! gyp ERR! find Python - Set the environment variable PYTHON
npm ERR! gyp ERR! find Python - Set the npm configuration variable python:
npm ERR! gyp ERR! find Python   npm config set python "/path/to/pythonexecutable"
npm ERR! gyp ERR! find Python For more information consult the documentation at:
npm ERR! gyp ERR! find Python https://github.com/nodejs/node-gyp#installation
npm ERR! gyp ERR! find Python **********************************************************
npm ERR! gyp ERR! find Python 
npm ERR! gyp ERR! configure error 
npm ERR! gyp ERR! stack Error: Could not find any Python installation to use
npm ERR! gyp ERR! stack     at PythonFinder.fail (/nix/store/6g5y6hq3npg9gy04q4ya1xzz9nj0bh59-nodejs-16.15.0/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:330:47)
npm ERR! gyp ERR! stack     at PythonFinder.runChecks (/nix/store/6g5y6hq3npg9gy04q4ya1xzz9nj0bh59-nodejs-16.15.0/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:159:21)
npm ERR! gyp ERR! stack     at PythonFinder.<anonymous> (/nix/store/6g5y6hq3npg9gy04q4ya1xzz9nj0bh59-nodejs-16.15.0/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:202:16)
npm ERR! gyp ERR! stack     at PythonFinder.execFileCallback (/nix/store/6g5y6hq3npg9gy04q4ya1xzz9nj0bh59-nodejs-16.15.0/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:294:16)
npm ERR! gyp ERR! stack     at exithandler (node:child_process:406:5)
npm ERR! gyp ERR! stack     at ChildProcess.errorhandler (node:child_process:418:5)
npm ERR! gyp ERR! stack     at ChildProcess.emit (node:events:527:28)
npm ERR! gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:289:12)
npm ERR! gyp ERR! stack     at onErrorNT (node:internal/child_process:478:16)
npm ERR! gyp ERR! stack     at processTicksAndRejections (node:internal/process/task_queues:83:21)
npm ERR! gyp ERR! System Linux 6.1.34
npm ERR! gyp ERR! command "/nix/store/6g5y6hq3npg9gy04q4ya1xzz9nj0bh59-nodejs-16.15.0/bin/node" "/nix/store/6g5y6hq3npg9gy04q4ya1xzz9nj0bh59-nodejs-16.15.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild" "--release"
npm ERR! gyp ERR! cwd /build/server/node_modules/better-sqlite3
npm ERR! gyp ERR! node -v v16.15.0
npm ERR! gyp ERR! node-gyp -v v9.0.0
npm ERR! gyp ERR! not ok
npm verb exit 1
npm timing npm Completed in 1223ms
npm verb unfinished npm timer reify 1702399506617
npm verb unfinished npm timer reify:build 1702399507602
npm verb unfinished npm timer build 1702399507603
npm verb unfinished npm timer build:deps 1702399507603
npm verb unfinished npm timer build:run:install 1702399507606
npm verb unfinished npm timer build:run:install:node_modules/better-sqlite3 1702399507606
npm verb code 1

npm ERR! A complete log of this run can be found in:
npm ERR!     /build/tmp.nRE4fM60rN/.npm/_logs/2023-12-12T16_45_06_500Z-debug-0.log

error: builder for '/nix/store/wphiq151fwzf09b2n75nh2rrbcn01dxb-server-0.0.0.drv' failed with exit code 1;
       last 10 log lines:
       > npm verb unfinished npm timer reify:build 1702399507602
       > npm verb unfinished npm timer build 1702399507603
       > npm verb unfinished npm timer build:deps 1702399507603
       > npm verb unfinished npm timer build:run:install 1702399507606
       > npm verb unfinished npm timer build:run:install:node_modules/better-sqlite3 1702399507606
       > npm verb code 1
       > 
       > npm ERR! A complete log of this run can be found in:
       > npm ERR!     /build/tmp.nRE4fM60rN/.npm/_logs/2023-12-12T16_45_06_500Z-debug-0.log
       > 
       For full logs, run 'nix log /nix/store/wphiq151fwzf09b2n75nh2rrbcn01dxb-server-0.0.0.drv'.
error: build of '/nix/store/gjiifk5hh2q9xg7gjl0nrfw7vl2qvzvr-client-0.1.0.drv', '/nix/store/wphiq151fwzf09b2n75nh2rrbcn01dxb-server-0.0.0.drv' failed

Hydra jobs continue "building" if npm errors out via napalm

I've experienced this with both npm install and npm run test (by providing npmCommands to buildPackage)

Hydra logs end with the npm ERR! ... error output that typically appears before npm exits with a non-zero exit code, e.g. when a dependency can't be found or a test fails.

Insert `npmCommands` directly into the build shell

Thanks a lot for building this tool!

I've encountered some problems during my project packaging - simple patching command like sed failed when it was included in the npmCommand.
It seems that the problem is related to the fact that currently commands are passed and used as shell variables by napalm. The simplest example of a similar failing flow could be:

#!/usr/bin/env bash

COMMAND='sed -i "s/x/y/" "file"'
$COMMAND

The above fails with sed: -e expression #1, char 1: unknown command: `"' even though sed command is correct by itself (and can be run direcc I'm not sure but probably arguments are passed altogether with quotes into the sed program when we execute it by variable expansion.

This simple change is enough to fix the problem by avoiding variable usage during npmCommand execution.

Do you think that my approach is correct and I can provide a PR with this change?

Non npm deps

Hi. I am running into an issue with dependecies that I would like to install via npm but do not come from the npm package regersity.

For example xlsx is installed with the command: 'npm install https://cdn.sheetjs.com/xlsx-0.20.1/xlsx-0.20.1.tgz'.

This results in a package-lock.json entry of:

"node_modules/xlsx": {
  "version": "0.20.1",
  "resolved": "https://cdn.sheetjs.com/xlsx-0.20.1/xlsx-0.20.1.tgz",
  "integrity": "sha512-hA7SYmn/H3cJ1VGi7kmSKxbEXCEn5UVJlwYFSFXhLe54wPUvW+80TmR5l+TZh4nJCw8G/e0SuZUfvqlTIh40hw==",
  "license": "Apache-2.0",
  "bin": {
    "xlsx": "bin/xlsx.njs"
  },
  "engines": {
    "node": ">=0.8"
  }
}

When building with napalm I get the following output when napalm is installing deps:

npm ERR! request to https://cdn.sheetjs.com/xlsx-0.20.1/xlsx-0.20.1.tgz failed, reason: getaddrinfo EAI_AGAIN cdn.sheetjs.com

I wrote a minimal test example here (dominicegginton@ef77510) that follow the tests examples in this repositoty, to demintrate the issue im having (nix build github:dominicegginton/napalm#hello-world-non-npm-deps).

Any help would be much appricated.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.