pastelsky / package-build-stats Goto Github PK

This is the cloud function that powers the core of building, minifying and gzipping of packages in bundlephobia

License: MIT License

JavaScript 74.67% TypeScript 25.26% Shell 0.07%

package-build-stats's Introduction

This is the function that powers the core of building, minifying and gzipping of packages in bundlephobia.

Usage

const { getPackageStats } = require('package-build-stats')

Building packages from npm

Building the latest stable version

const results = await getPackageStats('moment')

Building a specific version / tag

const results = await getPackageStats('[email protected]')

Building local packages (beta)

const results = await getPackageStats('~/dev/my-npm-package') // must have a package.json

Passing options to the build

const results = await getBuiltPackageStats('moment', options)

Options

Option	Values	Default	Description
client	`npm` or `yarn`	`npm`	Which client to use to install package for building
limitConcurrency	`true` or `false`	`false`	When using `yarn` as the client, use the network mutex to limit concurrency
networkConcurrency	`number`	`false`	When using `yarn` as client, limit simultaneous installs to this number.
customImports	`Array<string>`	`null`	By default, the default export is used for calculating sizes. Setting this option allows calculation of package stats based on more granular top-level exports.
minifier	`terser` or `esbuild`	`terser`	ESbuild is faster, albeit with marginally larger file sizes
installTimeout	number (ms)	30000	Timeout for package install

Listening to events

package-build-stats emits various lifecycle events when building a package. You can listen to these events by subscribing to the event emitter (based on mitt).

import { eventQueue } from 'package-build-stats'

// Listen to all events
eventQueue.on('*', callback)

// Listen to specific events
eventQueue.on('TASK_PACKAGE_BUILD', callback)

For a list of all events, see this.

Contributing

See contributing guide.

package-build-stats's People

Contributors

Stargazers

Watchers

package-build-stats's Issues

getPackageExportSizes fails on packages with conditional export maps

While investigating why Exports Analysis fails for fullcalendar on bundlephobia, I wrote this script:

import { getPackageExportSizes } from 'package-build-stats'

const sizes = await getPackageExportSizes('@fullcalendar/[email protected]')
console.log(sizes)

I get the following error:

Error: Package path . is not exported from package /tmp/tmp-build/packages/build-@fullcalendarcore-fIi/node_modules/@fullcalendar/core (see exports field in /tmp/tmp-build/packages/build-@fullcalendarcore-fIi/node_modules/@fullcalendar/core/package.json)
    at /Users/adam/Scratch/bundlephobia-test/node_modules/enhanced-resolve/lib/ExportsFieldPlugin.js:104:7
    at Hook.eval [as callAsync] (eval at create (/Users/adam/Scratch/bundlephobia-test/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:22:1)
    at Hook.CALL_ASYNC_DELEGATE [as _callAsync] (/Users/adam/Scratch/bundlephobia-test/node_modules/tapable/lib/Hook.js:18:14)
    at Resolver.doResolve (/Users/adam/Scratch/bundlephobia-test/node_modules/enhanced-resolve/lib/Resolver.js:432:16)
    at /Users/adam/Scratch/bundlephobia-test/node_modules/enhanced-resolve/lib/DescriptionFilePlugin.js:74:17
    at /Users/adam/Scratch/bundlephobia-test/node_modules/enhanced-resolve/lib/DescriptionFileUtils.js:118:13
    at /Users/adam/Scratch/bundlephobia-test/node_modules/enhanced-resolve/lib/forEachBail.js:16:12
    at onJson (/Users/adam/Scratch/bundlephobia-test/node_modules/enhanced-resolve/lib/DescriptionFileUtils.js:108:6)
    at Array.<anonymous> (/Users/adam/Scratch/bundlephobia-test/node_modules/enhanced-resolve/lib/DescriptionFileUtils.js:68:7)
    at runCallbacks (/Users/adam/Scratch/bundlephobia-test/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:27:15)

It seems like a conditional exports map inpackage.json like this is not supported. Here's what fullcalendar has:

  "exports": {
    "./package.json": "./package.json",
    "./index.cjs": "./index.cjs",
    "./index.js": "./index.js",
    ".": {
      "types": "./index.d.ts",
      "require": "./index.cjs",
      "import": "./index.js"
    },

`getPackageStats` options object required?

It seems the typing now requires an options object be passed to getPackageStats. The code seems to run fine without it (and the docs indicate it's optional), so this might just be a typing error.

src/bundlephobiaStats.ts:50:32 - error TS2554: Expected 2 arguments, but got 1.

50     const packageStats = await getPackageStats(cwd);
                                  ~~~~~~~~~~~~~~~~~~~~

  node_modules/package-build-stats/build/getPackageStats.d.ts:8:3
    8   optionsRaw: GetPackageStatsOptions
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    An argument for 'optionsRaw' was not provided.

Upgrade to Webpack 5

Webpack 5 supports current language features like optional chaining and nullish coalescing. Many of the build error issues on the Bundlephobia seem to be related to language support.

Custom webpack config

I'd like to run this with my own webpack config. Any thoughts on adding that feature?

write-file-webpack-plugin dependency

Seems like write-file-webpack-plugin is a dependency and not a dev dependency. Installing package-build-stats through npm and then requiring the package throws the following error: Error: Cannot find module 'write-file-webpack-plugin'

Support for WebAssembly (wasm) modules

I'm working on a package that uses WebAssembly, here: https://www.npmjs.com/package/wasm-marker-clusterer

Currently, .wasm files need to be loaded asynchronously, making it difficult to package them in the standard way that webpack would.

This is the build error message that comes up when I try to analyze my package:

/tmp/tmp-build/packages/[email protected]/node_modules/wasm-marker-clusterer/pkg/webassembly_marker_clusterer_bg.wasm
WebAssembly module is included in initial chunk.
This is not allowed, because WebAssembly download and compilation must happen asynchronous.
Add an async splitpoint (i. e. import()) somewhere between your entrypoint and the WebAssembly module:
* /tmp/tmp-build/packages/[email protected]/index.js 
  --> /tmp/tmp-build/packages/[email protected]/node_modules/wasm-marker-clusterer/dist/index.js 
  --> /tmp/tmp-build/packages/[email protected]/node_modules/wasm-marker-clusterer/pkg/webassembly_marker_clusterer.js 
  --> /tmp/tmp-build/packages/[email protected]/node_modules/wasm-marker-clusterer/pkg/webassembly_marker_clusterer_bg.wasm
* ... --> /tmp/tmp-build/packages/[email protected]/node_modules/wasm-marker-clusterer/pkg/webassembly_marker_clusterer.js 
  --> /tmp/tmp-build/packages/[email protected]/node_modules/wasm-marker-clusterer/pkg/webassembly_marker_clusterer_bg.wasm 
  --> /tmp/tmp-build/packages/[email protected]/node_modules/wasm-marker-clusterer/pkg/webassembly_marker_clusterer.js 
  --> /tmp/tmp-build/packages/[email protected]/node_modules/wasm-marker-clusterer/pkg/webassembly_marker_clusterer_bg.wasm

I think Wasm modules on NPM will be more and more popular as the technology spreads, and it would be awesome to be able to analyze them with this tool.

If you think this is worthwhile, I'd be interested in putting a PR together.

Support for Brotli Compression

I wanted to make a size comparison between actual size, gzip, and brotli. Would you be open to adding the brotli compression size?

I would imagine it being some like the following change within build.utils.ts:

import { gzipSync, brotliCompressSync } from 'zlib'

const brotli = brotliCompressSync(bundleContents, {}).length

and then returning that brotli size.

    return {
        ...,
        type: extension,
        size: asset.size,
        gzip,
        brotli,
        parse: parseTimes,
    }

`package-lock.json` warning appears when using Yarn

warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.

We should settle on using either npm or Yarn to manage packages, not both 🙂

I'd vote for Yarn personally as the main Bundlephobia repository also uses Yarn.

Support for custom npm registry

All our private packages are in a private registry. Any interest in adding support to install from there?

Cannot find module 'core-js/modules/es.string.replace.js'

When using package-build-stats v7.2.2 locally I get this error: Cannot find module 'core-js/modules/es.string.replace.js'

This is caused be a dependency error in csso-webpack-plugin (core-js is only listed as a devDependency). See: zoobestik/csso-webpack-plugin#28

Until csso-webpack-plugin publishes a fix, maybe core-js could be added as dependency to package-build-stats?

Full error (from rafgraph/rollpkg#6)

internal/modules/cjs/loader.js:1088
  throw err;
  ^

Error: Cannot find module 'core-js/modules/es.string.replace.js'
Require stack:
- /home/pierre/s/use-nft/node_modules/csso-webpack-plugin/lib/index.js
- /home/pierre/s/use-nft/node_modules/package-build-stats/build/config/makeWebpackConfig.js
- /home/pierre/s/use-nft/node_modules/package-build-stats/build/utils/build.utils.js
- /home/pierre/s/use-nft/node_modules/package-build-stats/build/getPackageStats.js
- /home/pierre/s/use-nft/node_modules/package-build-stats/build/index.js
- /home/pierre/s/use-nft/node_modules/rollpkg/dist/bundlephobiaStats.js
- /home/pierre/s/use-nft/node_modules/rollpkg/dist/cli.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:1085:15)
    at Function.Module._load (internal/modules/cjs/loader.js:928:27)
    at Module.require (internal/modules/cjs/loader.js:1145:19)
    at require (internal/modules/cjs/helpers.js:75:18)
    at Object.<anonymous> (/home/pierre/s/use-nft/node_modules/csso-webpack-plugin/lib/index.js:6:1)
    at Module._compile (internal/modules/cjs/loader.js:1256:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1277:10)
    at Module.load (internal/modules/cjs/loader.js:1105:32)
    at Function.Module._load (internal/modules/cjs/loader.js:967:14)
    at Module.require (internal/modules/cjs/loader.js:1145:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/home/pierre/s/use-nft/node_modules/csso-webpack-plugin/lib/index.js',
    '/home/pierre/s/use-nft/node_modules/package-build-stats/build/config/makeWebpackConfig.js',
    '/home/pierre/s/use-nft/node_modules/package-build-stats/build/utils/build.utils.js',
    '/home/pierre/s/use-nft/node_modules/package-build-stats/build/getPackageStats.js',
    '/home/pierre/s/use-nft/node_modules/package-build-stats/build/index.js',
    '/home/pierre/s/use-nft/node_modules/rollpkg/dist/bundlephobiaStats.js',
    '/home/pierre/s/use-nft/node_modules/rollpkg/dist/cli.js'
  ]
}

"WebpackOptionsValidationError: Invalid configuration object." when building a local package

WebpackOptionsValidationError: Invalid configuration object. Webpack has been initialised using a configuration object that does not match the API schema.
  Error: WebpackOptionsValidationError: Invalid configuration object. Webpack has been initialised using a configuration object that does not match the API schema.
   - configuration.module.rules[5].loader should be one of these:
     non-empty string | non-empty string | function | object { ident?, loader?, options?, query? } | function | [non-empty string | function | object { ident?, loader?, options?, query? }]

Full output is available here: https://github.com/typeofweb/schema/runs/1710304249?check_suite_focus=true

Did I do anything wrong? This is how I use the package:

https://github.com/typeofweb/typeofweb-bundlephobia-pr-stats-action/blob/a04a9c61d4d3f8a528dddbba1e54afa3de091834/src/build.ts#L31-L41

And this is the package I'm trying to measure:
https://github.com/typeofweb/schema

Wrong package size of `"type": "module"` package

https://bundlephobia.com/package/[email protected] says [email protected] is 921 B minified and 437 B minified+gzipped, but that's not correct. It is 724 B and 331 B, respectively.

I think the issue is that somehow this package is manually minifying src/index.js in the package rather than respecting the path set in package.json via exports (as well as module), but not main (the package does not support CJS). The path pointed to is dist/index.js.

See https://github.com/TomerAberbach/svkc

ERROR in main.bundle.js from Terser "Export" statement may only appear at the top level

This error occurs when the browser/module/main file of a package contains es6 style import/export statements.

Example where the error occurs: https://bundlephobia.com/[email protected]
Related issues: pastelsky/bundlephobia#164, pastelsky/bundlephobia#383

Modern libraries usually provide bundles with all es5 code except for import/export statements. This allows for optimal tooling support (treeshaking, etc.). Bundlephobia should support these kinds of libraries.

The reason for this error is that files that match this regex /\.min\.js$/ are not processed by webpack and thus the import/export statements are not replaced by webpack imports/exports.

The fix would be to remove the noParse rule from the webpack config. See https://github.com/pastelsky/package-build-stats/blame/master/src/webpack.config.js#L92

If you agree, I can prepare a pull request.

Error on using getPackageStats() on 'chalk' module

On typing the following code,

const results = await getPackageStats("chalk");
console.log(results);

the function returns an error as shown below:

(node:12162) UnhandledPromiseRejectionWarning: EntryPointError: EntryPointError
    at Object.buildPackage (/home/lenovo/Desktop/My Drive/Package Cleaner/node_modules/package-build-stats/build/utils/build.utils.js:204:17)
    at process._tickCallback (internal/process/next_tick.js:68:7)

It doesn't seem to be working for chalk module, but it works for others.

Please let me know if I am using this module incorrectly.

If a package is already built, should the stats just be measured from those pre-built assets?

for example, the packages mentioned here: pastelsky/bundlephobia#591
are all pre-built, and need specific build config, and I don't understand how a generic webpack config could possibly guess how to build them correctly 🤷
but I'm new to the bundlephobia details, so... I could obvs be missing some information about how this all works 🙃

move node-sass -> sass

I often install node-sass 4.14.1 failed when run postinstall script, please move node-sass to sass. node-sass had been deprecated.

Why don't we mangle?

https://github.com/pastelsky/package-build-stats/blob/v6.2.2/src/getDependencySizeTree.js#L139-L140

We use Terser.minify with mangle: false, which is different than almost all production builds. Effectively, we are penalizing packages who have readable variable names in node_modules :)

(npm already shows Unpacked Size, which is useful if you care about how much node_modules bloats, rather than your actual build.)

I know adding mangling would be a big change to historical data. Would you consider it?

Incorrect size for packages with the same name as Node core modules

Eg: https://bundlephobia.com/[email protected]

The generated entry point does require('util') which I guess turns into an external require() call because of this?

package-build-stats/src/getPackageStats.js

Lines 111 to 114 in 9aeb7a6

 const builtInNode = {} 

 builtinModules.forEach(mod => { 

 builtInNode[mod] = 'empty' 

 })

One kind of hacky way to work around this could be to append a / to the generated require() call—that'll force the resolution algorithm to use the dependency from node_modules rather than the builtin.

Nodejs 17: error:0308010C:digital envelope routines::unsupported

On a dev server with node.js 17, when you make a request:

curl 'http://localhost:3000/size?p=moment'

You get an error:

Error: error:0308010C:digital envelope routines::unsupported
    at new Hash (node:internal/crypto/hash:67:19)
    at Object.createHash (node:crypto:130:10)
    at module.exports (/Users/otp/Projects/package-build-stats/node_modules/webpack/lib/util/createHash.js:135:53)
    at NormalModule._initBuildHash (/Users/otp/Projects/package-build-stats/node_modules/webpack/lib/NormalModule.js:417:16)
    at handleParseError (/Users/otp/Projects/package-build-stats/node_modules/webpack/lib/NormalModule.js:471:10)
    at /Users/otp/Projects/package-build-stats/node_modules/webpack/lib/NormalModule.js:503:5
    at /Users/otp/Projects/package-build-stats/node_modules/webpack/lib/NormalModule.js:358:12
    at /Users/otp/Projects/package-build-stats/node_modules/loader-runner/lib/LoaderRunner.js:373:3
    at iterateNormalLoaders (/Users/otp/Projects/package-build-stats/node_modules/loader-runner/lib/LoaderRunner.js:214:10)
    at Array.<anonymous> (/Users/otp/Projects/package-build-stats/node_modules/loader-runner/lib/LoaderRunner.js:205:4)
    at Storage.finished (/Users/otp/Projects/package-build-stats/node_modules/webpack/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:55:16)
    at /Users/otp/Projects/package-build-stats/node_modules/webpack/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:91:9
    at /Users/otp/Projects/package-build-stats/node_modules/graceful-fs/graceful-fs.js:123:16
    at FSReqCallback.readFileAfterClose [as oncomplete] (node:internal/fs/read_file_context:68:3)

Bug the same with webpack/webpack#14532

Workaround:

NODE_OPTIONS=--openssl-legacy-provider npm run dev

To escape this issue in future, maybe need to lock node.js version in package.json or .nvmrc.

Support for react native packages

Currently, react-native packages are not supported as they are written in non-standard javascript.
This is an umbrella issue to track work required for adding basic support for react-native packages —

Upgrade to webpack 5
Enable babel transpilation of react-native packages
Deploy support for react native

tail: invalid option -1

I'm facing this issue which is apparently triggered by this line of package-build-stats:

package-build-stats/src/utils/installation.utils.ts

Line 18 in 3545073

`$(npm pack --ignore-scripts ${packagePath} | tail -1)`

The error message ("tail: invalid option -- 1") is not very helpful.
I have tried to create a minimal reproduction case but failed so far, also because I don't fully understand the purpose of that line of code (that is the wrapCommand function).
What are you using tail for?
And would you consider adding an error handler there and printing the packagePath if an error occurs?
That would have helped me greatly in creating a minimal reproduction case.

System info:
Windows 10
node v16.13.1 (LTS)
npm v8.3.0

Support for import.meta

Hi, I hope this is the right place to report, my apologies if not. I have a package that bundlephobia is not able to process, and it looks like the reason is that I am using import.meta.url. It's likely just a matter of updating the parser. Here's where you can see the error produced: https://bundlephobia.com/[email protected]

when i want to find out size of package 'zx', it throw a build error: 'Cannot use keyword 'await' outside an async function'

it seems, zx used new feature of node. Is package-build-stats only support to build web package?

How can i get size of node package which used node feature without webpack loader?

	const builtInNode = {}
	builtinModules.forEach(mod => {
	builtInNode[mod] = 'empty'
	})