duniul / clean-modules Goto Github PK

View Code? Open in Web Editor NEW

102.0 1.0 4.0 3.77 MB

🧹 Clean up/prune unnecessary files and reduce the size of your node_modules directory.

License: ISC License

JavaScript 0.23% TypeScript 99.77%

node node-modules npm yarn cli-tool ci

clean-modules's People

Contributors

Stargazers

Watchers

Forkers

imedadel miikis smorimoto awsp

clean-modules's Issues

--json option throws a TypeError

The --json option disables stdout logging in the cleanCommad() function.

The problem is that the noop Console object created in makeLogger() is empty but the cleanCommand() function attempts to use the log methods still.

The bug is here.

It should reduce the console object's keys — not the console.log function's keys.

Aggressive Results Using Many `--include` Flags

In preparing a fill-in for #3, I've converted our original .yarnclean into something that picomatch can use. I'm parsing that config, removing comments, and building a list of -i flags which represent globs from our config file. It appears that clean-modules sees the flags correctly.

When I don't use these flags, clean-modules correctly trims about 80mb from node_modules. When I use these flags, everything in node_modules is removed. I've run the list of globs through picomatch directly and it correctly identified the files it should based on those globs. picomatch did not match the same number of files that clean-modules does. Something is amiss.

Here's the analysis output:

Analyzed results:
{
  globs: {
    defaultDirs: '**/@(__tests__|test|tests|powered-test|docs|doc|.idea|.vscode|example|examples|coverage|.nyc_output|.nycrc|.circleci|.github|.git|website)',
    defaultFiles: '**/@(*.test.*|*.map|*.@(md|mkd|markdown|mdown)|*.mustache|*.asciidoc|*.DOCS|!(*.d).ts|*.coffee|*.zip|*.7z|*.rar|*.tar|*.tgz|*.@(jpg|jpeg)|*.png|*.gif|*.h|*.c|*.hpp|*.cpp|*.o|*.mk|*.log|*.tlog|*.patch|*.sln|*.pdb|*.jst|*.swp|*.lock|*.vcxproj*|makefile*|gemfile*|gulpfile*|gruntfile*|rakefile*|Jenkinsfile|.travis.yml|.gitlab-ci.yml|.appveyor.yml|circle.yml|.DS_Store|.eslintrc*|.stylelintrc*|stylelint.config.js|.htmllintrc*|htmllint.js|.jshintrc*|.lint|tslint.json|.prettierrc*|prettier.config.js|test.js|jest.config.js|karma.conf.js|wallaby.js|wallaby.conf.js|.coveralls.yml|readme*|changelog*|changes*|authors*|contributors*|contributing*|notice*|license|licence|tsconfig.json|.tsbuildinfo|.npmrc|.*ignore|yarn.lock|.yarnclean|.yarn-metadata.json|.gitmodules|.gitattributes|.babelrc|.editorconfig|.tern-project|.flowconfig|.documentup.json|.yo-rc.json|_config.yml|bower.json|.vimrc*|binding.gyp|component.json|composer.json|.jamignore|.jscsrc|*.todo|*.md|*.markdown|contributors|*.orig|*.rej|.zuul.yml|.editorconfig|.npmrc|.jshintignore|.eslintignore|.lint|.lintignore|cakefile|.istanbul.yml)',
    includeArgs: [
      '**/__tests__',
      '**/tests',
      '**/powered-test',
      '**/*.test.js',
      '**/*.tap.js',
      '**/*.test.ts',
      '**/*tests.html',
      '**/docs',
      '**/doc',
      '**/website',
      '**/images',
      '**/assets',
      '!**/@aws-cdk/**/assets',
      '!**/@aws-cdk/**/images',
      '**/example',
      '**/examples',
      '**/coverage',
      '**/.nyc_output',
      '**/Makefile',
      '**/Gulpfile.js',
      '**/Gruntfile.js',
      '**/.github/**/*.yml',
      '**/.*ignore',
      '**/.babelrc',
      '**/.browserslistrc',
      '**/.dist.babelrc',
      '**/.dist.eslintrc',
      '**/.documentup.json',
      '**/.editorconfig',
      '**/.eslintrc',
      '**/.eslintrc.js',
      '**/.eslintrc.json',
      '**/.eslintrc.yml',
      '**/.flowconfig',
      '**/.gitattributes',
      '**/.idea',
      '**/.jshintrc',
      '**/.lib.babelrc',
      '**/.lib.eslintrc',
      '**/.tern-project',
      '**/.travis.yml',
      '**/.yarn-metadata.json',
      '**/.zuul.yml',
      '**/appveyor.yml',
      '**/bower.json',
      '**/circle.yml',
      '**/codeship-services.yml',
      '**/codeship-steps.yml',
      '**/jsdoc_conf.json',
      '**/karma-ci.conf.js',
      '**/rollup.config.js',
      '**/tsconfig.json',
      '**/webpack.config.js',
      '**/wercker.yml',
      '**/*.html',
      '**/*.gif',
      '**/*.jpg',
      '**/*.lock',
      '**/*.log',
      '**/*.markdown',
      '**/*.map',
      '**/*.md',
      '**/*.png',
      '**/*.sh',
      '**/*.sln',
      '**/*.txt',
      '**/*.un~',
      '**/AUTHORS',
      '**/LICENSE',
      '**/MIT-LICENSE'
    ],
    excludeArgs: �undefined
  },

It's worth nothing that if I build the --include flag mirroring the defaultFiles property, I get the correct number of excluded files:

Analyzed results:
{
  globs: {
    defaultDirs: '**/@(__tests__|test|tests|powered-test|docs|doc|.idea|.vscode|example|examples|coverage|.nyc_output|.nycrc|.circleci|.github|.git|website)',
    defaultFiles: '**/@(*.test.*|*.map|*.@(md|mkd|markdown|mdown)|*.mustache|*.asciidoc|*.DOCS|!(*.d).ts|*.coffee|*.zip|*.7z|*.rar|*.tar|*.tgz|*.@(jpg|jpeg)|*.png|*.gif|*.h|*.c|*.hpp|*.cpp|*.o|*.mk|*.log|*.tlog|*.patch|*.sln|*.pdb|*.jst|*.swp|*.lock|*.vcxproj*|makefile*|gemfile*|gulpfile*|gruntfile*|rakefile*|Jenkinsfile|.travis.yml|.gitlab-ci.yml|.appveyor.yml|circle.yml|.DS_Store|.eslintrc*|.stylelintrc*|stylelint.config.js|.htmllintrc*|htmllint.js|.jshintrc*|.lint|tslint.json|.prettierrc*|prettier.config.js|test.js|jest.config.js|karma.conf.js|wallaby.js|wallaby.conf.js|.coveralls.yml|readme*|changelog*|changes*|authors*|contributors*|contributing*|notice*|license|licence|tsconfig.json|.tsbuildinfo|.npmrc|.*ignore|yarn.lock|.yarnclean|.yarn-metadata.json|.gitmodules|.gitattributes|.babelrc|.editorconfig|.tern-project|.flowconfig|.documentup.json|.yo-rc.json|_config.yml|bower.json|.vimrc*|binding.gyp|component.json|composer.json|.jamignore|.jscsrc|*.todo|*.md|*.markdown|contributors|*.orig|*.rej|.zuul.yml|.editorconfig|.npmrc|.jshintignore|.eslintignore|.lint|.lintignore|cakefile|.istanbul.yml)',
    includeArgs: [
      '**/@(,__tests__|tests|powered-test|*.test.js|*.tap.js|*.test.ts|*tests.html|docs|doc|website|images|assets|!**/@aws-cdk/**/assets|!**/@aws-cdk/**/images|example|examples|coverage|.nyc_output|Makefile|Gulpfile.js|Gruntfile.js|.github/**/*.yml|.*ignore|.babelrc|.browserslistrc|.dist.babelrc|.dist.eslintrc|.documentup.json|.editorconfig|.eslintrc|.eslintrc.js|.eslintrc.json|.eslintrc.yml|.flowconfig|.gitattributes|.idea|.jshintrc|.lib.babelrc|.lib.eslintrc|.tern-project|.travis.yml|.yarn-metadata.json|.zuul.yml|appveyor.yml|bower.json|circle.yml|codeship-services.yml|codeship-steps.yml|jsdoc_conf.json|karma-ci.conf.js|rollup.config.js|tsconfig.json|webpack.config.js|wercker.yml|*.html|*.gif|*.jpg|*.lock|*.log|*.markdown|*.map|*.md|*.png|*.sh|*.sln|*.txt|*.un~|AUTHORS|LICENSE|MIT-LICENSE,)'
    ],
    excludeArgs: �undefined
  },

ESLint new configuration file paths

https://eslint.org/docs/latest/use/configure/configuration-files-new#configuration-file

eslint.config.js
eslint.config.mjs
eslint.config.cjs

clean-modules/.cleanmodules-default

Line 116 in efcc25c

.eslintrc*(.*)

Multiple exclude patterns ignored except one.

If you exclude more that one pattern with the ! (in the glob file) or --exclude flag then only one is effective. For example:

Install mongoose and serverless

npm install mongoose serverless

List files to be excluded

ls node_modules/@serverless/utils/test

  account.js
  cloudformation-schema.js
  config.js
  download.js
  fixture.zip
  get-notifications-mode.js
  inquirer
  is-in-china.js
  log.js
  process-backend-notification-request.js
  telemetry.js

ls node_modules/mongoose/lib/cursor

  AggregationCursor.js
  ChangeStream.js
  QueryCursor.js

Clean modules

clean-modules --yes --exclude "/@serverless/utils/test/" --exclude "/mongoose/lib/cursor/"

Files excluded in mongoose package 😀

ls node_modules/mongoose/lib/cursor
AggregationCursor.js
ChangeStream.js
QueryCursor.js

Files included in serverless package 😲

ls node_modules/@serverless/utils/test
ls: node_modules/@serverless/utils/test: No such file or directory

I'd be happy to hop in and contribute to this but I'm not a Typescripter :(

Running on node project with npm install --production returns always 0 results

Hello,

I want to shrink node_modules installed for nodejs procject with production flag. But none of the files are removed:

clean-modules

Are you sure you want to clean D:\Projects\hsk-webkd\backend\node_modules? Files will be permanently removed. (Y/N)
Y

Cleaning up node_modules...
Done in 153ms!

Cleaning up empty dirs...
Done in 0ms!

Results:
- size reduced: 0 B
- files removed: 0
- empty dirs removed: 0

What is the issue here?

Some false positives: mongoose ChangeStream, lab .eslintrc

Hi there, thanks for your work on this project! Here is some ~~money~~ user demand for your efforts.

"filePath": "/home/ddv/projects/ahg/ahpi/node_modules/mongoose/lib/cursor/ChangeStream.js",
"includedByDefault": true,
"includedByGlobs": [
  {
    "original": "changes*",
    "derived": ".../node_modules/**/changes*"
  }

Also node_modules/lab/lib/linter/.eslintrc is required.

I'm doing fine with this command, but you might want to do something about those.

clean-modules node_modules -y -e "**/lab/lib/linter/.eslintrc*" -e "**/ChangeStream.js"

yargs reserved word warning

Started noticing this recently, not exactly sure when it started.

(node:33) Warning: "version" is a reserved word.
Please do one of the following:
- Disable version with `yargs.version(false)` if using "version" as an option
- Use the built-in `yargs.version` method instead (if applicable)
- Use a different option key
https://yargs.js.org/docs/#api-reference-version

It seems to be related to the code here:

clean-modules/src/cli/cli.ts

Lines 32 to 37 in 8e77f83

 .option('version', { 

 alias: 'v', 

 description: 'Show script version', 

 type: 'boolean', 

 global: true, 

 })

Definitely not a critical issue, but would be nice to trim that warning from our deployment log output. Our deployment logs have alerts and filters setup around them, and the bells are going off for each deployment with the warning in there. It's an inconvenience, and not a blocker, but maybe there's something that can be done to eliminate that.

pnpm support?

Thanks for your project! Will it work with pnpm node_modules structure?

Add adoc to AsciiDoc glob

https://docs.asciidoctor.org/asciidoc/latest/directives/include/#include-nonasciidoc

If the file is recognized as an AsciiDoc file (i.e., it has one of the following extensions: .asciidoc, .adoc, .ad, .asc, or .txt)

https://docs.gitlab.com/ee/user/asciidoc.html

AsciiDoc documents (.adoc or .asciidoc) inside repositories

clean-modules/.cleanmodules-default

Line 53 in efcc25c

*.asciidoc

Globbing by file paths not working

I use this command: npx clean-modules analyze --exclude "**/googleapis/**/docs" >> clean-modules-result.json
But all files in node_modules/googleapis/build/src/apis/docs folder are still being removed.
And got this result:

...
{
    "filePath": "/home/ubuntu/www/my-projects/node_modules/googleapis/build/src/apis/docs/index.js",
    "includedByDefault": true,
    "includedByGlobs": [
      {
        "original": "docs/",
        "derived": "/home/ubuntu/www/my-projects/node_modules/**/docs/**"
      }
    ]
  },
...

intermittent ENOENT errors seen during execution

Hi,

When running clean-modules as part of a container build from a node:11.1.0-alpine base image we've periodically run into errors like the following:

{ [Error: ENOENT: no such file or directory, scandir '/usr/src/api/app/node_modules/resolve/test']
  errno: -2,
  code: 'ENOENT',
  syscall: 'scandir',
  path: '/usr/src/api/app/node_modules/resolve/test' }
Removing intermediate container 04a4039cca36
The command '/bin/sh -c clean-modules --yes ./app/node_modules       --exclude "**/googleapis/build/src/apis/docs/**"       --exclude "**/mongoose/**"       --exclude "**/grpc/**"' returned a non-zero code: 1

The actual file not being found varies from one run to the next, and this error doesn't always occur but it appears there may be a timing issue running this? We saw a few failures in files under the grpc package so that's why we added that exclusion. Has anyone else encountered this?

Autoreport detected issues to offending repositories

First of all, this is an amazing tool! It broke my Sveltekit application due to some commands for the dev server being dependent on *.md files, but it did what it's supposed to (removed 100+ MB) and worked without any issues.

I generated a clean-modules-result.json and would probably start reporting issues to GitHub repos, but then had the idea that there must be some way to do this automatically. At least for findings with 100% certainty that they shouldn't be in node_modules e.g. coverage or test files.

Did you consider this yet?

.yarnclean compatibility

Hey there 👋 Great project, I'm glad that someone else has done this legwork. I would like to know if you'd be willing to add the ability to read globs from a resource/config file, just like .yarnclean. Our ecosystem was setup to use yarn, but we're slowly migrating to pnpm, and our shared .yarnclean file is significant in size - it wouldn't be easy to add each line as a command line flag. What are your thoughts about supporting globs from a file like that?

OSSMETADATA default exclusion

https://shields.io/badges/oss-lifecycle

You might also want to add a OSSMETADATA file and badge to your repo as well.

Not finding anything to remove

> npx clean-modules -d -D node_modules --include "**/*.js"

clean-modules (dry run)

Cleaning up node_modules...
Done in 98ms!

Cleaning up empty dirs...
Skipped on dry runs!

Results:
- size reduced: 0 B
- files removed: 0
- empty dirs removed: 0

There are definitely *.js files in that folder that should've been found in this run?

System Info

 System:
    OS: Windows 10 10.0.19044
    CPU: (24) x64 AMD Ryzen 9 5900X 12-Core Processor
    Memory: 90.57 GB / 127.90 GB
  Binaries:
    Node: 16.14.2
    npm: 8.5.5

is there import.meta' in cjs?

/Users/bing/Projects/xxxx/node_modules/clean-modules/dist/chunk-475NGVCH.cjs:113
fileDir(import.meta),
^^^^

SyntaxError: Cannot use 'import.meta' outside a module
at Object.compileFunction (node:vm:352:18)
at wrapSafe (node:internal/modules/cjs/loader:1033:15)

vite can not get worked

	.option('version', {
	alias: 'v',
	description: 'Show script version',
	type: 'boolean',
	global: true,
	})