swarajban / npm-cache Goto Github PK

View Code? Open in Web Editor NEW

168.0 168.0 57.0 305 KB

Cache (npm | bower | composer | jspm) installs

License: MIT License

JavaScript 100.00%

npm-cache's People

Contributors

$fracmak avatar$

Stargazers

Watchers

npm-cache's Issues

Got "EXDEV, cross-device link not permitted" after upgrade to 0.6.1

It seems related to #58 .

Doesn't actually extract the modules

I don't get any extracted modules. Everything looks fine in the command prompt.
It clears the old folder but never creates a new one.

C:\TeamCity\buildAgent\work\6e8ae23de4bcc9d9\***SE_src\***.SKT\***.SKT.Api\***.SKT.Api\Application>npm-cache install npm
[npm-cache] [INFO] using C:\Users\****\.package_cache as cache directory
[npm-cache] [INFO] [npm] config file exists
[npm-cache] [INFO] [npm] cli exists
[npm-cache] [INFO] [npm] hash of C:\TeamCity\buildAgent\work\6e8ae23de4bcc9d9\***SE_src\***.SKT\***.SKT.Api\***.SKT.Api\Application\package.json: 6cd7ceb02dc80477b9c0084e50507a72
[npm-cache] [INFO] [npm] cache exists
[npm-cache] [INFO] [npm] clearing installed dependencies at C:\TeamCity\buildAgent\work\6e8ae23de4bcc9d9\***SE_src\***.SKT\***.SKT.Api\***.SKT.Api\Application\node_modules
[npm-cache] [INFO] [npm] ...cleared
[npm-cache] [INFO] [npm] extracting dependencies from C:\Users\****\.package_cache\npm\3.6.0\6cd7ceb02dc80477b9c0084e50507a72.tar.gz
[npm-cache] [INFO] [npm] done extracting
[npm-cache] [INFO] successfully installed all dependencies

I'm using Windows Server. Any ideas?

Edit: I can add that bower_components works fine.
I might also add I'm using Node 5.7.0 NPM 3.6
Added an issue in node-tar: isaacs/node-tar#67

Infinite run when package has file dependencies

Example modules structure:

/vagrant/a/package.json:

{
  "dependencies": {
    "b": "file:../b"
  }
}

/vagrant/b/package.json (no file dependencies)

Trying to run npm-cache install in /vagrant/a leads to infinite repeat of npm install:

[npm-cache] [INFO] using /home/vagrant/.package_cache as cache directory                                                          
[npm-cache] [INFO] [composer] Dependency config file /vagrant/a/composer.json does not exist. Skipping install
[npm-cache] [INFO] [npm] config file exists                                                                                       
[npm-cache] [INFO] [npm] cli exists                                                                                               
[npm-cache] [INFO] [npm] hash of /vagrant/a/package.json: df06dcab837ecf93c78fdb2b768c83dc                    
[npm-cache] [INFO] [npm] running [npm install]...                                                                                 

> [email protected] install /vagrant/a                                                                
> npm-cache install npm                                                                                                           

[npm-cache] [INFO] using /home/vagrant/.package_cache as cache directory
...

Ubuntu 14.04
Node: v5.9.0
npm: 3.7.3

I'd be happy to provide a pull request if anybody could give me a hint about what can cause this. I couldn't find such behavior in sources.

npm-cache shuold respect NODE_ENV or --production

its pretty common to have dependencies and devDependencies in package.json and use NODE_ENV=production or npm install --production to omit devDependencies. Would be great if npm-cache would consider NODE_ENV and/or have a --production flag and work the same way.

Support for "npm install --production"

On my build server I run "npm install --production". I tried "npm-cache install --production" and I get a usage error.

All symbol links in node_modules/.bin are broken after npm-cache install extract the archive

I checked the archive and it's ok, and v0.4.10 do not have this bug. So the problem is due to using Decompress package. I suggest to switch back to targz or just call tar cli.

Doesn't work on Windows because tar is not available

Windows doesn't have a tar exe so an error is thrown. This module is very useful and it would be nice to see it work in Windows.

0.5.2 new tar compressor breaks on existing cache

I just tried to rebase on the PR you just merged and in my test it fails when trying to extract an existing cache file. Maybe it should invalidate the cache or add something to the cache hash filename.

[npm-cache] [INFO] [jspm] extracting dependencies from /Users/avweiss/.package_cache/jspm/0.16.27/0733d590c33222a8f95fe2090c28ccbd.tar.gz
[npm-cache] [ERROR] [jspm] Error extracting /Users/avweiss/.package_cache/jspm/0.16.27/0733d590c33222a8f95fe2090c28ccbd.tar.gz: Error: invalid tar file
[npm-cache] [ERROR] [jspm] Error extracting /Users/avweiss/.package_cache/jspm/0.16.27/0733d590c33222a8f95fe2090c28ccbd.tar.gz: Error: invalid tar file
[npm-cache] [ERROR] [jspm] Error extracting /Users/avweiss/.package_cache/jspm/0.16.27/0733d590c33222a8f95fe2090c28ccbd.tar.gz: Error: invalid tar file
.......

Support for multiple verions of node

Some node modules have a binary compilation piece to the (phantomjs for example) which results in an "npm install" that is only compatible with the verion of node that compiled it. So if I try to build the same project with two different versions of node, the second one gets an error because it pulls in the node_modules of the first one.

I could manually script different --cacheDirectory locations for each node version, but it would be nice if the tool did that. Maybe under the --cacheDirectory should be a directory for the node version used to create it, then the md5sum. For example:

~/.package_cache/0.10.32/8a944a089acdeeafbf831f58d2813c85.tar.gz
~/.package_cache/0.12.4/8a944a089acdeeafbf831f58d2813c85.tar.gz

Add ability to store cached packages at server

It would be really great if npm-cache could cache packages not only locally but at some server. Then we could use it as corporate package proxy server.

give option to use symlinks

First of all - thank you. npm-cache install saves my nerves.

Since i work in a lot of small projects i realized how crazy it gets with modules. Yesterday in order to backup my full drive and update Ubuntu i deleted 10 GB of node modules. NPM is probably killing every developers SSD drive. Most of the projects use very similar modules, mostly even in the same version.

What i'd love would be a global npm storage that makes sure every module is only present once per version. In the projects node_modules folder there should be a symlink to the versions folder.

Why?

I never change node modules (if u do, fork them!)
npm-cache stores the modules already in a proper way (just remove archives)
i don't want to waste time and disk space anymore

npm run <script> does not work as intended

npm run <script> where any <script> that refers to the node_modules/.bin generally has problems. e.g. mocha and grunt which maybe installed in the the relative node_modules folder

npm-cache --cacheDirectory exits doing nothing

$ cat package.json
{
  "name": "machtfit",
  "version": "1.0.0",
  "devDependencies": {
    "gulp": "^3.8.9",
    "gulp-coffee": "^2.2.0",
    "gulp-coffeelint": "^0.4.0",
    "gulp-plumber": "^0.6.6",
    "gulp-sourcemaps": "^1.2.4",
    "gulp-util": "^3.0.1"
  },
  "private": true,
  "repository": {
    "type": "git",
    "url": "https://github.com/machtfit/machtfit-oscar.git"
  }
}
$ node --version
v0.12.2
$ npm --version
2.7.4
$ npm-cache --version
0.3.0
$ ls node_modules
ls: cannot access node_modules: No such file or directory
$ time npm-cache --cacheDirectory /home/django/.package_cache-0.12 install npm

real    0m0.118s
user    0m0.104s
sys     0m0.016s
$ ls node_modules
ls: cannot access node_modules: No such file or directory
$ time npm-cache -c /home/django/.package_cache-0.12 install npm

real    0m0.111s
user    0m0.100s
sys     0m0.016s
$ ls node_modules
ls: cannot access node_modules: No such file or directory

npm install works fine:

$ time npm install --quiet
[email protected] node_modules/gulp-plumber
└── [email protected] ([email protected], [email protected])

[email protected] node_modules/gulp-sourcemaps
├── [email protected]
├── [email protected]
├── [email protected] ([email protected], [email protected])
├── [email protected] ([email protected], [email protected])
└── [email protected] ([email protected], [email protected])

[email protected] node_modules/gulp-coffee
├── [email protected]
├── [email protected] ([email protected], [email protected])
├── [email protected]
└── [email protected] ([email protected])

[email protected] node_modules/gulp-util
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected] ([email protected], [email protected])
├── [email protected] ([email protected], [email protected], [email protected], [email protected], [email protected])
├── [email protected] ([email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected])
├── [email protected] ([email protected], [email protected])
├── [email protected] ([email protected], [email protected])
└── [email protected] ([email protected])

[email protected] node_modules/gulp-coffeelint
├── [email protected] ([email protected], [email protected])
├── [email protected] ([email protected], [email protected])
├── [email protected]
└── [email protected] ([email protected], [email protected], [email protected], [email protected], [email protected], [email protected])

[email protected] node_modules/gulp
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected] ([email protected])
├── [email protected] ([email protected])
├── [email protected]
├── [email protected] ([email protected], [email protected], [email protected], [email protected], [email protected])
├── [email protected]
├── [email protected] ([email protected], [email protected], [email protected])
├── [email protected] ([email protected], [email protected], [email protected], [email protected])
└── [email protected] ([email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected])

real    0m4.761s
user    0m4.140s
sys     0m0.528s
$ ls node_modules
gulp  gulp-coffee  gulp-coffeelint  gulp-plumber  gulp-sourcemaps  gulp-util

npm-cache checksum downloaded dependencies

we're seeing this error every few days in our ci

[npm-cache] [INFO] [npm] extracting dependencies from /path/to/caches/npm/2.3.0/fe56bf256353b99775417bb4904da265.tar.gz 

gzip: stdin: decompression OK, trailing garbage ignored 
tar: Child returned status 2 
tar: Error is not recoverable: exiting now 
[npm-cache] [ERROR] [npm] error untar-ing /path/to/caches/npm/2.3.0/fe56bf256353b99775417bb4904da265.tar.gz 
[npm-cache] [ERROR] error installing dependencies 

Fatal error: local() encountered an error (return code 1) while executing 'npm-cache install --cacheDirectory /path/to/caches/'

it seems like file is corrupted during download?
npm-cache should checksum downloaded dependencies?

Running two instances of npm-cache simultaneously may create an invalid archive

We use this package on our build server which runs many simultaneous builds at once. Today, we upgraded npm (which makes npm-cache use a new cache directory), and experienced that the new archive created by npm-cache was invalid. This caused subsequent runs of npm-cache install to fail, since it tried to use an invalid archive.

This error seems to be caused by npm-cache archiving directly to the cache directory, which is the same for all instances of npm-cache with the same package.json file. I think the error occurred because two instances of npm-cache tried to write to the same file simultaneously, causing it to become invalid.

To fix this, npm-cache should create the archive in some unique location first, and then move it to the cache directory after it has completed creating the archive.

Why does it clear installed dependencies if nothing changed?

This seemed like a great idea. It uses cache instead of a million http requests. Thats great. But why does it clear all my existing node_modules and bower_components every time I run "npm-cache install"?

I thought the point is to reduce the amount of time npm install takes. Now it just deletes the dependencies and reinstalls the exact same ones.

Can you make it check if installed dependencies matches the hashtag for cached dependencies, and if so, stop execution? That would make it really speedy.

Hide progress bar on npm install

Word on the street is that the NPM progress bar is slow. As far as I can tell it's not explicitly disabled by this library, and seeing as the progress is hidden perhaps the --progress=false flag should be passed by default?

Wrong path for bower_components after upgrade to v0.4.10

In v0.3.9 it works well, but now the bower_components are in the root of my project and not in the location define by my bowerrc...

Any idea ?

is bower-installer supported? postinstall will not run when dependencies extracted from cache

Hi,

I have a package.json with a postinstall script to run bower-installer.
When I use npm-cache the postinstall script is not run and bower dependencies are missing.
I need bower-installer to move/rename some files from bower packages. So using bower is not an option.

package.json

{
  "name": "my-package",
  "private": true,
  "version": "0.1.0",
  "repository": {
    "type": "git",
    "url": "http://github.com/some/repo.git"
  },
  "dependencies": {
    "bower-installer": "git+ssh://[email protected]:rebuy-de/bower-installer.git#master"
  },
  "scripts": {
    "postinstall": "node_modules/bower-installer/bower-installer.js"
  }
}

The project is in two different folders due to some concurrency issues with our automatic and manual deployment.
So when npm-cache is run from folder 1 everything is fine. When run from folder 2 it uses the cached files, but there the bower dependencies from bower-installer are missing.

Is there a possibility to force run postinstall scripts from the package.json?
Am I doing it wrong?

Please add 'typings' package manager support

Not detecting changes to devDependencies if npm-shrinkwrap.json is present.

Hi! we're using npm-cache to improve the time our CI takes to lint/test our project. We have a npm-shrinkwrap.json used to freeze dependencies versions.

What I found is that if I update only a devDependency, the hash generated is the same (because it uses npm-shrinkwrap.json to generate the hash and it didn't change).

I know that I can use --dev to add devDependencies to npm-shrinkwrap, but I like how it works by default.

What I think will be a good solution is to add package.json's devDependencies here (https://github.com/swarajban/npm-cache/blob/master/cacheDependencyManagers/npmConfig.js#L28) if npm-shrinkwrap.json didn't have them.

Do you think this is a good idea? If so, I can try to do a PR adding this feature.

tar-fs vs node-tar

I read another issue about performance. Ran across this test:
https://gist.github.com/mafintosh/8102201

Is it possible to use tar-fs instead of node-tar if this test is really correct, and I'm not misunderstanding anything.

Cannot find module 'fstream' in cacheDependencyManager

After updating from version 0.5.1 to 0.5.3 today, npm-cache immediately throws the following error:

Error: Cannot find module 'fstream' at Function.Module._resolveFilename (module.js:336:15) at Function.Module._load (module.js:278:25) at Module.require (module.js:365:17) at require (module.js:384:17) at Object.<anonymous> (/Users/patrickdavis/cvent.git/developer-portal/developer-portal-ui/node_modules/npm-cache/cacheDependencyManagers/cacheDependencyManager.js:10:15) at Module._compile (module.js:460:26) at Object.Module._extensions..js (module.js:478:10) at Module.load (module.js:355:32) at Function.Module._load (module.js:310:12) at Module.require (module.js:365:17)

Not sure how fstream is supposed to be installed into npm-cache, but it's completely missing. Installing fstream and adding it to npm-cache's node_modules folder resolves this issue.

Error while running on windows

I'm trying to run npm-cache install npm from windows shell and get the following:
[npm-cache] [INFO] [npm] installed npm dependencies, now archiving
[npm-cache] [INFO] [npm] archiving dependencies from node_modules
tar (child): Cannot execute remote shell: No such file or directory
tar (child): C:\Users\username.package_cache\18c7544ea9d5afdef076583351ae2
7a2.tar.gz: Cannot open: I/O error
tar (child): Error is not recoverable: exiting now
stdout: Invalid argument

gzip: [npm-cache] [ERROR] [npm] error tar-ing node_modules
rm: no such file or directory: C:\Users\username.package_cache\18c7544ea9d5afde
f076583351ae27a2.tar.gz

[npm-cache] [ERROR] error installing dependencies

The same works fine on linux.

npm-cache -v 0.3.0

Cache Invalidation

Hi,

Thanks for your work on a great tool, it's taken part of our build process down from 75 seconds to 3, which is obviously wonderful.

The only downside that we have with it right now is the lack of cache invalidation, currently cache will grow until you manually clear it. It would be great if there could be an optional, built-in invalidation. The most sensible for me would be to keep the X most recently used, where X could be passed in. I think you would need to touch each zip before you use it, to ensure it remains freshest and doesn't get deleted each time. What do you think?

I'm sorry I'm not nodeJS dev, otherwise I would have a go and make a PR!

Thanks!

npm-cache fails to install: exit code 255

When I run npm install -g npm-cache I get the following error message:

npm ERR! not a package C:\Users\Me\appdata\roaming\npm\npm-cache
npm ERR! addLocal Could not install C:\Users\Me\appdata\roaming\npm\npm-cache
npm ERR! Windows_NT 6.1.7601
npm ERR! argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\Me\\AppData\\Roaming\\npm\\node_modules\\npm\\bin\\npm-cli.js" "install" "-g" "npm-cache"
npm ERR! node v5.10.1
npm ERR! npm  v3.10.2
npm ERR! path C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9\package.json
npm ERR! code ENOENT
npm ERR! errno -4058
npm ERR! syscall open

npm ERR! enoent ENOENT: no such file or directory, open 'C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9\package.json'
npm ERR! enoent ENOENT: no such file or directory, open 'C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9\package.json'
npm ERR! enoent This is most likely not a problem with npm itself
npm ERR! enoent and is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! Please include the following file with any support request:
npm ERR!     C:\Users\buildmeister\appdata\roaming\npm\npm-debug.log

"Me" is a replacement for a longer username. Otherwise the log file contents are exactly as stated.

Here is the full log file:

0 info it worked if it ends with ok
1 verbose cli [ 'C:\\Program Files\\nodejs\\node.exe',
1 verbose cli   'C:\\Users\\Me\\AppData\\Roaming\\npm\\node_modules\\npm\\bin\\npm-cli.js',
1 verbose cli   'install',
1 verbose cli   '-g',
1 verbose cli   'npm-cache' ]
2 info using [email protected]
3 info using [email protected]
4 silly loadCurrentTree Starting
5 silly install loadCurrentTree
6 silly install readGlobalPackageData
7 silly fetchPackageMetaData npm-cache
8 silly fetchOtherPackageData npm-cache
9 silly cache add args [ 'npm-cache', null ]
10 verbose cache add spec npm-cache
11 silly cache add parsed spec Result {
11 silly cache add   raw: 'npm-cache',
11 silly cache add   scope: null,
11 silly cache add   escapedName: 'npm-cache',
11 silly cache add   name: null,
11 silly cache add   rawSpec: 'npm-cache',
11 silly cache add   spec: 'C:\\Users\\Me\\appdata\\roaming\\npm\\npm-cache',
11 silly cache add   type: 'local' }
12 silly addLocalTarball shasum (computed) f1f9665493e7089d78850103dd0e4d283416bd05
13 verbose addTmpTarball C:\Users\Me\appdata\roaming\npm\npm-cache not in flight; adding
14 verbose correctMkdir C:\Users\Me\AppData\Roaming\npm-cache correctMkdir not in flight; initializing
15 verbose addTmpTarball validating metadata from C:\Users\Me\appdata\roaming\npm\npm-cache
16 verbose tar unpack C:\Users\Me\appdata\roaming\npm\npm-cache
17 verbose tar unpacking to C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9
18 silly gentlyRm C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9 is being purged
19 verbose gentlyRm don't care about contents; nuking C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9
20 silly gunzTarPerm modes [ '777', '666' ]
21 error not a package C:\Users\Me\appdata\roaming\npm\npm-cache
22 error addLocal Could not install C:\Users\Me\appdata\roaming\npm\npm-cache
23 silly fetchPackageMetaData Error: ENOENT: no such file or directory, open 'C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9\package.json'
23 silly fetchPackageMetaData     at Error (native)
23 silly fetchPackageMetaData  error for npm-cache { [Error: ENOENT: no such file or directory, open 'C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9\package.json']
23 silly fetchPackageMetaData   errno: -4058,
23 silly fetchPackageMetaData   code: 'ENOENT',
23 silly fetchPackageMetaData   syscall: 'open',
23 silly fetchPackageMetaData   path: 'C:\\Users\\Me~1\\AppData\\Local\\Temp\\3\\npm-15596-c052a037\\unpack-6d94b7f9\\package.json' }
24 silly rollbackFailedOptional Starting
25 silly rollbackFailedOptional Finishing
26 silly runTopLevelLifecycles Starting
27 silly runTopLevelLifecycles Finishing
28 silly install printInstalled
29 verbose stack Error: ENOENT: no such file or directory, open 'C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9\package.json'
29 verbose stack     at Error (native)
30 verbose cwd C:\Users\Me\appdata\roaming\npm
31 error Windows_NT 6.1.7601
32 error argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\Me\\AppData\\Roaming\\npm\\node_modules\\npm\\bin\\npm-cli.js" "install" "-g" "npm-cache"
33 error node v5.10.1
34 error npm  v3.10.2
35 error path C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9\package.json
36 error code ENOENT
37 error errno -4058
38 error syscall open
39 error enoent ENOENT: no such file or directory, open 'C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9\package.json'
40 error enoent ENOENT: no such file or directory, open 'C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9\package.json'
40 error enoent This is most likely not a problem with npm itself
40 error enoent and is related to npm not being able to find a file.
41 verbose exit [ -4058, true ]

Any help troubleshooting this problem would be appreciated.

Jspm support

jspm is yet another package manager coming to frontend repositories. It would be great to also have support for caching those files.

The folder to be cached can be extrapolated from require('package.json').jspm.directories.packages with a fallback to require('package.json').jspm.directories.baseURL with a fallback to ./jspm_packages

install npm --production gives same result as without --production

When running
npm-cache install npm --production
if you later run without --production on same package.json-file you get the same result

Ex
have a package.json file with devDependencies
run: npm-cache install npm --production
delete node_modules folder
run: npm-cache install npm
then node_modules doesn't container devDependencies

Add option to store dependencies compressed

Hi,

I stumbled upon npm-cache yesterday and I totally loved the idea. Then I made a few tests with showed quite promising results (npm i took 5,7 mins npm-cache install npm 3,43). After further investigation I found out that the slowest part of npm-cache install npm is extracting the archive (obviously) and I wondered if we could add a switch to store the dependencies without compressing them. Because in our case the compressed tar-file has 219MB and the uncompressed node_modules directory has 246MB (and normally disk space is not the problem in a CI-environment) and just resolving the hash and copying a directory should be blazingly fast.

We could probably do something like npm-cache install --deflate in order to just copy the node_modules to the cacheDirectory without compressing them. And for resolving the dependencies we could scan the cacheDirectory for either #{HASH}.tar.gz files or directories with the specified hash.

What do you think about that idea? If you like it as well I'd be happy to contribute a PR.

'npm link'd repository handling

If I link a local directory as a dependency to my project (i.e. npm link my-local-package), that dependency is then represented as a symlink in the node_modules directory. If I now run npm-cache install, and I had previously used a different version of that local package, it will unpack the cached version into the symlinked directory, overwriting whatever is currently there.

To avoid this, I would suggest that npm-cache install removes any linked packages/symlinked directories before unpacking the cache.

Error tar-ing path containing spaces

Hi there,
If one has a path containing a space, e.g. /Users/John/Projects/My project, npm-cache cannot tar packages, like the exemple below.

[npm-cache] [INFO] [bower] archiving dependencies from /Users/John/Projects/My project
tar: could not chdir to '/Users/John/Projects/My'
[npm-cache] [ERROR] [bower] error tar-ing /Users/John/Projects/My project
[npm-cache] [ERROR] error installing dependencies

When calling npm-cache from a Node.js exec, if NPM_CACHE_DIR isn't set, an exception is thrown.

Hi!

I'm using npm-cache as part of my deployment system, and found that if NPM_CACHE_DIR isn't set on the environment when called from child_process.exec, it throws the following exception:

path.js:8
    throw new TypeError('Path must be a string. Received ' +
    ^

TypeError: Path must be a string. Received undefined
    at assertPath (path.js:8:11)
    at Object.posix.resolve (path.js:426:5)
    at main (/usr/lib/node_modules/npm-cache/index.js:36:48)
    at Object.<anonymous> (/usr/lib/node_modules/npm-cache/index.js:159:1)
    at Module._compile (module.js:410:26)
    at Object.Module._extensions..js (module.js:417:10)
    at Module.load (module.js:344:32)
    at Function.Module._load (module.js:301:12)
    at Function.Module.runMain (module.js:442:10)
    at startup (node.js:136:18)
    at node.js:966:3

I did the following to prevent this error: export NPM_CACHE_DIR=/path/to/cache && npm-cache install

Note: this happened even when specifying the cache directory through the cacheDirectory or c parameters.

0.4.x issue

Hey,
I've just upgraded our CI runner to use 0.4.10 of npm-cache, but I'm running into an issue. The first fresh npm-cache install runs fine, but subsequent calls that use the cache cause ember-cli to fail with:

ember -v

module.js:340
    throw err;
          ^
Error: Cannot find module 'rsvp'
    at Function.Module._resolveFilename (module.js:338:15)
    at Function.Module._load (module.js:280:25)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object. (/Users/jkay/node_modules/ember-cli/lib/ext/promise.js:3:15)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)

0.3.9 appears to work fine. Here is a list of the node_modules folder after a full install and after a cached npm-cache install. A tonne of packages are missing when installed from the cache using 0.4.x, but when using 0.3.x the installation works fine and all modules are un-archived correctly.

Error thrown in 0.3.5 if composer.json is not present

With the 0.3.5 release last night my npm-cache installs are all failing (exit code 1) with this error:

[npm-cache] [INFO] [bower] Dependency config file {my code directory}/bower.json does not exist. Skipping install
module.js:338
    throw err;
          ^
Error: Cannot find module '{my code directory}/composer.json'
    at Function.Module._resolveFilename (module.js:336:15)
    at Function.Module._load (module.js:278:25)
    at Module.require (module.js:365:17)
    at require (module.js:384:17)
    at Object.<anonymous> ({my code directory}/node_modules/npm-cache/cacheDependencyManagers/composerConfig.js:27:16)
    at Module._compile (module.js:460:26)
    at Object.Module._extensions..js (module.js:478:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Module.require (module.js:365:17)

0.3.4 does not have this issue, I do not need composer.json and bower.json to be present to run npm-cache install.

EOF error when extracting npm cache

laptop:myproject adambaker$ npm-cache install npm
[npm-cache] [INFO] using /Users/adambaker/.package_cache as cache directory
[npm-cache] [INFO] [npm] config file exists
[npm-cache] [INFO] [npm] cli exists
[npm-cache] [INFO] [npm] hash of /Users/adambaker/Development/...projectnamehere.../ember/package.json:88508382ccc5f6dba26f358779b3f3e2
[npm-cache] [INFO] [npm] cache exists
[npm-cache] [INFO] [npm] clearing installed dependencies at /Users/adambaker/Development/...projectnamehere.../ember/node_modules
[npm-cache] [INFO] [npm] ...cleared
[npm-cache] [INFO] [npm] extracting dependencies from /Users/adambaker/.package_cache/npm/2.12.1/88508382ccc5f6dba26f358779b3f3e2.tar.gz
[npm-cache] [ERROR] [npm] Error extracting /Users/adambaker/.package_cache/npm/2.12.1/88508382ccc5f6dba26f358779b3f3e2.tar.gz: Error: unexpected eof
[npm-cache] [ERROR] error installing dependencies

I have played around with this for a while. I think this might be failing when the cache is just too large?

add an option to install single package, e.g. npm-cache install <package_name>

I'm wondering is it possible to install single package? If not, will you plan add such feature? It would be very helpful.

Command to check if there is a cache-hit (feature request)

Sometimes it is helpful to know if the install will occur from cache or not, for conditional processing.

Something like: npm-cache iscached npm that returns an error status code upon cache-miss, or the hash string if the cache exists.

npm-shrinkwrap

How about support for npm-shrinkwrap.json as the checksum source? You could have different npm-shrinkwrap.json from the same package.json, but npm-cache would not pick that up.

npm ERR! Error: EPERM: operation not permitted, rmdir

Frequently, when trying to execute the npm-cache install npm script, I get this failed error message. If I run it again it usually succeeds. What is causing rmdir to fail? It only happens when node_modules is not empty.

npm ERR! Windows_NT 10.0.10586
npm ERR! argv "C:\\dev\\nodejs\\node.exe" "C:\\dev\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "cache" "clean"
npm ERR! node v4.4.4
npm ERR! npm  v2.15.1
npm ERR! path C:\Users\nparson\AppData\Roaming\npm-cache
npm ERR! code EPERM
npm ERR! errno -4048
npm ERR! syscall rmdir

npm ERR! Error: EPERM: operation not permitted, rmdir 'C:\Users\nparson\AppData\Roaming\npm-cache'
npm ERR!     at Error (native)
npm ERR!  { [Error: EPERM: operation not permitted, rmdir 'C:\Users\nparson\AppData\Roaming\npm-cache']
npm ERR!   errno: -4048,
npm ERR!   code: 'EPERM',
npm ERR!   syscall: 'rmdir',
npm ERR!   path: 'C:\\Users\\nparson\\AppData\\Roaming\\npm-cache' }
npm ERR! 
npm ERR! Please try running this command again as root/Administrator.

npm ERR! Please include the following file with any support request:
npm ERR!     C:\dev\#Projects\ASP.Net.Core1.Angular2.Webpack.Starter\src\starterproject\npm-debug.log

npm ERR! Windows_NT 10.0.10586
npm ERR! argv "C:\\dev\\nodejs\\node.exe" "C:\\dev\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "run" "clean"
npm ERR! node v4.4.4
npm ERR! npm  v2.15.1
npm ERR! code ELIFECYCLE
npm ERR! [email protected] preclean: `npm run installglobals && npm cache clean`
npm ERR! Exit status 4294963248

Not working when called as a root for installing bower dependencies

[npm-cache] [ERROR] [bower] error running bower install
[npm-cache] [ERROR] error installing dependencies

Question: Unpack cache not in cwd

Hey, the way my project is setup i have node_modules at cwd, and jspm_packages in my static folder which isn't. Kind of like a maven setup.

Does this lib have any options for that type of setup?

Thank you

npm-cache install for npm packages only

Is there a way to do this? I'm getting bower errors, and its pretty time consuming to set them up for my cluster slave nodes. Just need to install npm stuff, so is there a way to skip bower?

Re-Archive feature request

Thanks for sharing your work. It has proven to be very useful for speeding up my CI pipeline.

Some background - My development CI process currently works like this:

1. npm-cache install npm                  # Install (hopefully from cache)
2. npm update                             # Update any new dependencies from last cache
3. npm-cache install --forceRefresh npm   # Re-archive the dependencies

In my package.json some of my internal project dependency versions are set to "latest" (it's continuous delivery-related decision...). Since the cache is keyed to a hash of the package.json file itself and not the latest versions of each dependencies, I need to run an update after the initial install to pick up any changes from the first cache.

It would be great if I could do this all in one shot. Something like an: npm-cache update npm, where upon a cache-hit, it extracts, runs update, then re-archives (otherwise on cache-miss, just install as normal).

Or instead/additionally, a new command like npm-cache rearchive npm would be helpful in speeding up my step 3 (I don't need to actually run an install again, only need to cache it).

I might be missing an undocumented feature, but wanted to ask about this regardless. Thanks again!

Setting cacheDirectory to node_modules writes to the disk infinitely

It is ideal for my use case to use a directory in node_modules for the cache. I can understand why that doesn't work. I tried it anyway, and the process seemed to hang after installing, and it turns out that it was writing to node_modules directory during that time. It filled up 10GB. I believe it would have gone on to fill up the entire disk if not stopped. I propose one of the following:

If cacheDirectory is or is nested in node_modules, throw an error that cacheDirectory cannot be under node_modules
Allow the cacheDirectory to function under node_modules by ignoring it when interacting with node_modules.

I would like to submit a PR for this at some point. I think the first thing that needs to happen is for there to be some tests so that things can be changed and refactored safely. For now, I just wanted this issue to be known in case it helps someone avoid filling up their disk!

Error: Cannot find module 'lodash'

packages.json

{
     // ...
    "devDependencies": {
        "beepbeep": "^1.2.0",
        "browser-sync": "^2.10.0",
        "gulp-autoprefixer": "^3.1.0",
        "gulp-concat": "^2.6.0",
        "gulp-html-minifier": "^0.1.6",
        "gulp-imagemin": "^2.4.0",
        "gulp-less": "^3.0.5",
        "gulp-rename": "^1.2.2",
        "gulp-sourcemaps": "^1.6.0",
        "gulp-tslint": "^4.1.0",
        "gulp-typescript": "^2.9.2",
        "gulp-uglify": "^1.5.1",
        "gulp-uncss": "^1.0.4",
        "gulp-util": "^3.0.7",
        "imagemin-pngquant": "^4.2.0",
        "less-plugin-clean-css": "^1.5.1",
        "merge2": "^0.3.6",
        "tslint": "^3.1.1"
    }
}

gulp is globally installed

requiring gulp-less ends in the following error. default npm install works fine

module.js:339
    throw err;
    ^

Error: Cannot find module 'lodash'
    at Function.Module._resolveFilename (module.js:337:15)
    at Function.Module._load (module.js:287:25)
    at Module.require (module.js:366:17)
    at require (module.js:385:17)
    at Object.<anonymous> (/var/www/xxx/builds/835110e/src/node_modules/gulp-less/node_modules/accord/lib/index.js:11:7)
    at Object.<anonymous> (/var/www/xxx/builds/835110e/src/node_modules/gulp-less/node_modules/accord/lib/index.js:100:4)
    at Module._compile (module.js:435:26)
    at Object.Module._extensions..js (module.js:442:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:311:12)

Use composer.lock instead of composer.json

Hi, I just noticed this package is using the md5 of composer.json to see if there's any changes. This seems really wrong to me. Every project should commit his composer.lock and that file should be used, not the json.

Is it possible to use npm-cache with other registries?

npm has the --registry flag to allow it to use a different registry, such as a company-internal registry. Does npm-cache allow some sort of passthrough of config options to the eventual package manager?

How to install specific NPM packages with global flag?

Hello. I want to execute this
npm install -g typings webpack-dev-server rimraf webpack

What is the proper syntax to tell npm-cache to execute npm install on specific pacakges only, and globally?

I don't see the option to specify which npm package I want npm-cache to install, nor the option to pass global. Does it support this?

Pull Requests against `master` or `develop`?

Which branch should I create pull requests against? It looks like develop is work in progress (portable tar etc.) - which one should I use for a PR fixing some inconsistencies?

Failing to clear installed dependencies

Trying to run the npm-cache command back to back results in an error coming up from RimRaf saying that a directory is not empty.

>npm-cache install --cacheDirectory "C:\Users\Andrew Stoker\AppData\Roaming\npm-cache" npm
[npm-cache] [INFO] using C:\Users\Andrew Stoker\AppData\Roaming\npm-cache as cache directory
[npm-cache] [INFO] [npm] config file exists
[npm-cache] [INFO] [npm] cli exists
[npm-cache] [INFO] [npm] hash of C:\Workflow\Branches\CentralPrototype\Projects\Source\WebApplication\package.json: bf9c78e40ae0c10c7636ed4d576043fa
[npm-cache] [INFO] [npm] cache exists
[npm-cache] [INFO] [npm] clearing installed dependencies at C:\Workflow\Branches\CentralPrototype\Projects\Source\WebApplication\node_modules
C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:304
        throw er
        ^

Error: ENOTEMPTY: directory not empty, rmdir 'C:\Workflow\Branches\CentralPrototype\Projects\Source\WebApplication\node_modules\bower-installer\node_modules\glob'
    at Error (native)
    at Object.fs.rmdirSync (fs.js:806:18)
    at rmkidsSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:334:11)
    at rmdirSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:324:7)
    at rimrafSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:295:9)
    at C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:332:5
    at Array.forEach (native)
    at rmkidsSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:331:26)
    at rmdirSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:324:7)
    at rimrafSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:295:9)

Third attempt:

>npm-cache install --cacheDirectory "C:\Users\Andrew Stoker\AppData\Roaming\npm-cache" npm
[npm-cache] [INFO] using C:\Users\Andrew Stoker\AppData\Roaming\npm-cache as cache directory
[npm-cache] [INFO] [npm] config file exists
[npm-cache] [INFO] [npm] cli exists
[npm-cache] [INFO] [npm] hash of C:\Workflow\Branches\CentralPrototype\Projects\Source\WebApplication\package.json: bf9c78e40ae0c10c7636ed4d576043fa
[npm-cache] [INFO] [npm] cache exists
[npm-cache] [INFO] [npm] clearing installed dependencies at C:\Workflow\Branches\CentralPrototype\Projects\Source\WebApplication\node_modules
C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:304
        throw er
        ^

Error: ENOTEMPTY: directory not empty, rmdir 'C:\Workflow\Branches\CentralPrototype\Projects\Source\WebApplication\node_modules\es6-module-loader\test'
    at Error (native)
    at Object.fs.rmdirSync (fs.js:806:18)
    at rmkidsSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:334:11)
    at rmdirSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:324:7)
    at rimrafSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:295:9)
    at C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:332:5
    at Array.forEach (native)
    at rmkidsSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:331:26)
    at rmdirSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:324:7)
    at rimrafSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:295:9)

The directory it's failing on is not consistent, which leads me to believe that it's doing a little bit more deleting every time.

Running v. 0.5.3 of npm-cache.

swarajban / npm-cache Goto Github PK

npm-cache's People

Contributors

Stargazers

Watchers

Forkers

npm-cache's Issues

packages.json

Recommend Projects

Recommend Topics

Recommend Org