Giter VIP home page Giter VIP logo

ncc's Introduction

ncc

CI Status

Simple CLI for compiling a Node.js module into a single file, together with all its dependencies, gcc-style.

Motivation

  • Publish minimal packages to npm
  • Only ship relevant app code to serverless environments
  • Don't waste time configuring bundlers
  • Generally faster bootup time and less I/O overhead
  • Compiled language-like experience (e.g.: go)

Design goals

  • Zero configuration
  • TypeScript built-in
  • Only supports Node.js programs as input / output
  • Support all Node.js patterns and npm modules

Usage

Installation

npm i -g @vercel/ncc

Usage

$ ncc <cmd> <opts>

Eg:

$ ncc build input.js -o dist

If building an .mjs or .js module inside a "type": "module" package boundary, an ES module output will be created automatically.

Outputs the Node.js compact build of input.js into dist/index.js.

Note: If the input file is using a .cjs extension, then so will the corresponding output file. This is useful for packages that want to use .js files as modules in native Node.js using a "type": "module" in the package.json file.

Commands:

  build <input-file> [opts]
  run <input-file> [opts]
  cache clean|dir|size
  help
  version

Options:

  -o, --out [dir]          Output directory for build (defaults to dist)
  -m, --minify             Minify output
  -C, --no-cache           Skip build cache population
  -s, --source-map         Generate source map
  -a, --asset-builds       Build nested JS assets recursively, useful for
                           when code is loaded as an asset eg for workers.
  --no-source-map-register Skip source-map-register source map support
  -e, --external [mod]     Skip bundling 'mod'. Can be used many times
  -q, --quiet              Disable build summaries / non-error outputs
  -w, --watch              Start a watched build
  -t, --transpile-only     Use transpileOnly option with the ts-loader
  --v8-cache               Emit a build using the v8 compile cache
  --license [file]         Adds a file containing licensing information to the output
  --stats-out [file]       Emit webpack stats as json to the specified output file
  --target [es]            ECMAScript target to use for output (default: es2015)
                           Learn more: https://webpack.js.org/configuration/target
  -d, --debug              Show debug logs

Execution Testing

For testing and debugging, a file can be built into a temporary directory and executed with full source maps support with the command:

$ ncc run input.js

With TypeScript

The only requirement is to point ncc to .ts or .tsx files. A tsconfig.json file is necessary. Most likely you want to indicate es2015 support:

{
  "compilerOptions": {
    "target": "es2015",
    "moduleResolution": "node"
  }
}

If typescript is found in devDependencies, that version will be used.

Package Support

Some packages may need some extra options for ncc support in order to better work with the static analysis.

See package-support.md for some common packages and their usage with ncc.

Programmatically From Node.js

require('@vercel/ncc')('/path/to/input', {
  // provide a custom cache path or disable caching
  cache: "./custom/cache/path" | false,
  // externals to leave as requires of the build
  externals: ["externalpackage"],
  // directory outside of which never to emit assets
  filterAssetBase: process.cwd(), // default
  minify: false, // default
  sourceMap: false, // default
  assetBuilds: false, // default
  sourceMapBasePrefix: '../', // default treats sources as output-relative
  // when outputting a sourcemap, automatically include
  // source-map-support in the output file (increases output by 32kB).
  sourceMapRegister: true, // default
  watch: false, // default
  license: '', // default does not generate a license file
  target: 'es2015', // default
  v8cache: false, // default
  quiet: false, // default
  debugLog: false // default
}).then(({ code, map, assets }) => {
  console.log(code);
  // Assets is an object of asset file names to { source, permissions, symlinks }
  // expected relative to the output code (if any)
})

When watch: true is set, the build object is not a promise, but has the following signature:

{
  // handler re-run on each build completion
  // watch errors are reported on "err"
  handler (({ err, code, map, assets }) => { ... })
  // handler re-run on each rebuild start
  rebuild (() => {})
  // close the watcher
  void close ();
}

Caveats

  • Files / assets are relocated based on a static evaluator. Dynamic non-statically analyzable asset loads may not work out correctly

ncc's People

Contributors

0xflotus avatar arunoda avatar dependabot[bot] avatar fenix20113 avatar fkhadra avatar guybedford avatar heypiotr avatar huozhi avatar igorklopov avatar ijjk avatar legendecas avatar maxbeatty avatar mreis1 avatar novascreen avatar odensc avatar pi0 avatar rafaelkr avatar rauchg avatar smorimoto avatar sokra avatar songkeys avatar stigkj avatar stscoundrel avatar styfle avatar thboop avatar timer avatar timneutkens avatar tootallnate avatar xom9ikk avatar ziishaned avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ncc's Issues

Fixing circular case

Here is a code example of what is breaking with cycles:

a.js

exports.run = function () {
  console.log('parent');
}
require('./b.js');

b.js

require('./a.js').run();

The issue is that this gets converted into an ES module tree looking like:

a.js

import './b.js';
import 'commonjs-proxy:./b.js';

var fn = function () {
  console.log('parent');
}

var a = {
        fn: fn
};

export default a;
export { a as __moduleExports };
export { fn };

b.js

import './a.js';
import require$$0 from 'commonjs-proxy:./a.js';

require$$0.fn();

var b = {

};

export default b;
export { b as __moduleExports };

And so "fn" isn't defined as b.js executes before a.js entirely, no exports are permitted before the require statement.

Now one naive fix for this particular case would be to use a function binding in the output, as these hoist through circular references in ES modules. That would only fix exported functions though, so executed binding values would still throw. This may possibly be adequate though as most test cases are functions as classes, as it seems from most of the bug reports. So it could be an interesting first step to try, although would be invalidated by a single exports.x = 'evaluated value'; require('./circular') bug.

Bundle Node builtins

Currently, it seems it does not bundle them, because I can find

/* 0 */
/***/ (function(module, exports) {

module.exports = require("process");

/***/ }),

In the bundle. And worse, it does not use the webpack require but the normal one and I can bet it won't work in the browser.

Even without this problem, it still won't work in the browser because it module.exports = webpack stuff, so I can suggest a new flag to bundle for the browser.

V8 Compile Cache

The idea is that we can emit not just .js files, but also the v8 code cache blobs.
And unlike v8-compile-cache, we could benefit from making all bootups fast, including the first one. Also unlike it, we can use the new createCachedData APIs which have some benefits.

  • Make sure we disable the optimization gracefully when invoking in old Node versions
  • Make sure to add a flag to disable it (--no-v8-cache)

More:

grpc bug

The issue with the package.json error there is the following line in node_modules/grpc/src/grpc_extension.js:

var binding_path =
    binary.find(path.resolve(path.join(__dirname, '../package.json')));

where __dirname then has a different meaning in the built file than from the original.

the above binding_path is then passed directly to require. Ideally binding_path should be statically computed in its entitreity.

Few questions around context

Love this space, especially with the upcoming / ongoing shift to ES module support.

Would love to hear your guy's thoughts on how this project relates to something like microbundle.

Would it be accurate to say ncc : modules :: pkg : executables?

Thanks! 😄

An amd test proposal

Let's add this snippet as a test for f1eec46

console.log(
  ('function' === typeof define) && (define.amd != null)
);

Compiled javascript must return false. Without the commit, compiled snippet crashes.

Self-hosting fails

When I try to ncc ncc:

▲  ncc/ (master) ncc index.js
(node:22762) UnhandledPromiseRejectionWarning: Error: 'sync' is not exported by node_modules/resolve/index.js
    at error (/usr/local/lib/node_modules/@zeit/ncc/node_modules/rollup/dist/rollup.js:3460:30)
    at Module.error (/usr/local/lib/node_modules/@zeit/ncc/node_modules/rollup/dist/rollup.js:13349:9)
    at handleMissingExport (/usr/local/lib/node_modules/@zeit/ncc/node_modules/rollup/dist/rollup.js:13039:21)
    at Module.traceVariable (/usr/local/lib/node_modules/@zeit/ncc/node_modules/rollup/dist/rollup.js:13453:17)
    at ModuleScope.findVariable (/usr/local/lib/node_modules/@zeit/ncc/node_modules/rollup/dist/rollup.js:12726:29)
    at FunctionScope.Scope.findVariable (/usr/local/lib/node_modules/@zeit/ncc/node_modules/rollup/dist/rollup.js:4022:68)
    at Scope.findVariable (/usr/local/lib/node_modules/@zeit/ncc/node_modules/rollup/dist/rollup.js:4022:68)
    at BlockScope.Scope.findVariable (/usr/local/lib/node_modules/@zeit/ncc/node_modules/rollup/dist/rollup.js:4022:68)
    at FunctionScope.Scope.findVariable (/usr/local/lib/node_modules/@zeit/ncc/node_modules/rollup/dist/rollup.js:4022:68)
    at Scope.findVariable (/usr/local/lib/node_modules/@zeit/ncc/node_modules/rollup/dist/rollup.js:4022:68)
(node:22762) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:22762) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

`request` errors

ncc version: 0.0.4

Source: (index.js)

const request = require("request");
const url = "https://dog.ceo/api/breeds/image/random";

module.exports = (req, res) => {
  request.get(url, { json: true }, (err, resp, body) => {
    if (err || body.status != "success")
      return res.end("https://dog.ceo API error");
    res.end(`<!doctype html><img src="${body.message}"/>`);
  });
};

Works:

npx micro index.js

image

Doesn't work:

▲  request/ npx micro req.js
npx: installed 20 in 2.62s
tough-cookie: can't load punycode; won't use punycode for domain normalization
micro: Error when importing /Users/rauchg/Projects/now-demos/request/req.js: TypeError [ERR_INVALID_ARG_TYPE]: The "superCtor" argument must be of type Function. Received type undefined
    at Object.inherits (util.js:291:11)
    at Object.<anonymous> (/Users/rauchg/Projects/now-demos/request/req.js:1:294079)
    at Module._compile (internal/modules/cjs/loader.js:707:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:718:10)
    at Module.load (internal/modules/cjs/loader.js:605:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:544:12)
    at Function.Module._load (internal/modules/cjs/loader.js:536:3)
    at Module.require (internal/modules/cjs/loader.js:643:17)
    at require (internal/modules/cjs/helpers.js:22:18)
    at module.exports (/Users/rauchg/.npm/_npx/28627/lib/node_modules/micro/lib/handler.js:8:15)
micro: https://err.sh/micro/invalid-entry

Out flag option does not work correctly

If you want the output to be printed to another file, different than dist/index.js you should use the --out flag, which describes that you should provide "out file".

But when you do

ncc build src/api.js --out dist/api.js

Then it creates a dist/api.js/index.js and not dist/api.js what we need.

`twilio` errors

ncc version: 0.0.4

Source:

const twilio = require('twilio')
const client = new twilio('a', 'b');

Crashes with:

▲  twilio/ node out.js
tough-cookie: can't load punycode; won't use punycode for domain normalization
util.js:291
    throw new ERR_INVALID_ARG_TYPE('superCtor', 'Function', superCtor);
    ^

TypeError [ERR_INVALID_ARG_TYPE]: The "superCtor" argument must be of type Function. Received type undefined
    at Object.inherits (util.js:291:11)
    at Object.<anonymous> (/Users/rauchg/Projects/now-demos/twilio/out.js:1:1505162)
    at Module._compile (internal/modules/cjs/loader.js:707:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:718:10)
    at Module.load (internal/modules/cjs/loader.js:605:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:544:12)
    at Function.Module._load (internal/modules/cjs/loader.js:536:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:760:12)
    at startup (internal/bootstrap/node.js:308:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:878:3)

`pdfkit` errors

ncc version: 0.22

Source:

const fs = require("fs");
const PDFDocument = require("pdfkit");
const doc = new PDFDocument();
doc.fontSize(15).text("Hi there", 50, 50);
doc.pipe(fs.createWriteStream("out.pdf"));
doc.end();

Expected:

No output, working pdf:

▲  play-ncc/ node index.js
▲  play-ncc/

image

Received:

▲  play-ncc/ node out.js
internal/modules/cjs/loader.js:589
    throw err;
    ^

Error: Cannot find module 'iconv-lite'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:587:15)
    at Function.Module._load (internal/modules/cjs/loader.js:513:25)
    at Module.require (internal/modules/cjs/loader.js:643:17)
    at require (internal/modules/cjs/helpers.js:22:18)
    at Object.<anonymous> (/Users/rauchg/Projects/play-ncc/out.js:1:270)
    at Module._compile (internal/modules/cjs/loader.js:707:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:718:10)
    at Module.load (internal/modules/cjs/loader.js:605:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:544:12)
    at Function.Module._load (internal/modules/cjs/loader.js:536:3)

Compilation output:

▲  play-ncc/ ncc index.js -o out.js
'iconv-lite' is imported by node_modules/fontkit/index.js, but could not be resolved – treating it as an external dependency
'iconv-lite' is imported by node_modules/restructure/src/DecodeStream.js, but could not be resolved – treating it as an external dependency
'iconv-lite' is imported by node_modules/restructure/src/EncodeStream.js, but could not be resolved – treating it as an external dependency
'iconv-lite' is imported by commonjs-external:iconv-lite, but could not be resolved – treating it as an external dependency
Circular dependency: node_modules/pdfkit/js/object.js -> node_modules/pdfkit/js/reference.js -> node_modules/pdfkit/js/object.js
Circular dependency: node_modules/pdfkit/js/object.js -> node_modules/pdfkit/js/reference.js -> commonjs-proxy:/Users/rauchg/Projects/play-ncc/node_modules/pdfkit/js/object.js -> node_modules/pdfkit/js/object.js
Circular dependency: node_modules/pdfkit/js/font.js -> node_modules/pdfkit/js/font/standard.js -> node_modules/pdfkit/js/font.js
Circular dependency: node_modules/pdfkit/js/font.js -> node_modules/pdfkit/js/font/standard.js -> commonjs-proxy:/Users/rauchg/Projects/play-ncc/node_modules/pdfkit/js/font.js -> node_modules/pdfkit/js/font.js
Circular dependency: node_modules/pdfkit/js/font.js -> node_modules/pdfkit/js/font/embedded.js -> node_modules/pdfkit/js/font.js

Failing to build "strong-globalize".

Start

package.json

{
  "dependencies": {
    "strong-globalize": "4.1.2"
  }
}

index.js

const fail = require('strong-globalize')
module.exports = fail

Reproducing

$ npm i
$ ncc run index.js

Output

Error: Cannot find module 'cldr'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:580:15)
    at Function.Module._load (internal/modules/cjs/loader.js:506:25)
    at Module.require (internal/modules/cjs/loader.js:636:17)
    at require (internal/modules/cjs/helpers.js:20:18)
    at Object.<anonymous> (/tmp/12be0a0a6e869/index.js:538:18)
    at __webpack_require__ (/tmp/12be0a0a6e869/index.js:21:30)
    at /tmp/12be0a0a6e869/index.js:126:4
    at Object.<anonymous> (/tmp/12be0a0a6e869/index.js:133:2)
    at __webpack_require__ (/tmp/12be0a0a6e869/index.js:21:30)
    at Object.<anonymous> (/tmp/12be0a0a6e869/index.js:7594:18)

Original topic: https://spectrum.chat/?t=98452abc-a7ce-4a11-8529-9fb6db1cf690

Native module support

$ cat index.js
const time = require('time');

const d = new time.Date();
d.setTimezone('US/Hawaii');
console.log(d.toString());

$ ncc index.js > out.js
$ node out.js
/Users/nrajlich/t/out.js:1703
  throw err
  ^

Error: Could not locate the bindings file. Tried:
 → /Users/nrajlich/t/build/time.node
 → /Users/nrajlich/t/build/Debug/time.node
 → /Users/nrajlich/t/build/Release/time.node
 → /Users/nrajlich/t/out/Debug/time.node
 → /Users/nrajlich/t/Debug/time.node
 → /Users/nrajlich/t/out/Release/time.node
 → /Users/nrajlich/t/Release/time.node
 → /Users/nrajlich/t/build/default/time.node
 → /Users/nrajlich/t/compiled/10.13.0/darwin/x64/time.node
    at bindings (/Users/nrajlich/t/out.js:1700:9)
    at Object.<anonymous> (/Users/nrajlich/t/out.js:344:39)
    at __webpack_require__ (/Users/nrajlich/t/out.js:21:30)
    at Object.<anonymous> (/Users/nrajlich/t/out.js:326:14)
    at __webpack_require__ (/Users/nrajlich/t/out.js:21:30)
    at /Users/nrajlich/t/out.js:85:18

However, even if I manually copy over the .node file into ./build/time.node, I get the same error. I think due to some webpack magic related to dynamic require calls. Webpack turns this:

      b = opts.path ? require.resolve(n) : require(n)

Into this:

      b = opts.path ? /*require.resolve*/(__webpack_require__(3).resolve(n)) : __webpack_require__(3)(n)

CI

Currently there is no CI setup.

Will there be one?

Got is failing to compile

Original thread: https://spectrum.chat/zeit/now/now-is-failing-because-of-missing-npm-dependencies~d3155f18-59a9-48b0-900a-69038d337e08

Referenced GH issue: sindresorhus/got#345 (comment)

More information:

ERROR in /tmp/3694173/user/node_modules/path-platform/path.js 32:2
Module parse failed: 'return' outside of function (32:2)
You may need an appropriate loader to handle this file type.
| if (_path.posix) {
|   module.exports = _path;
>   return;
| }
|
 @ /tmp/3694173/user/node_modules/parents/index.js 1:19-43
 @ /tmp/3694173/user/node_modules/module-deps/index.js
 @ /tmp/3694173/user/node_modules/browserify/index.js
 @ /tmp/3694173/user/node_modules/lyo/lib/task.js
 @ /tmp/3694173/user/node_modules/lyo/index.js
 @ /tmp/3694173/user/lib/compile.js
 @ /tmp/3694173/user/handlers/compile.js
    at compiler.run (/tmp/3694173/ncc/node_modules/@zeit/ncc/index.js:65:23)
    at finalCallback (/tmp/3694173/ncc/node_modules/@zeit/ncc/webpack/lib/Compiler.js:210:39)
    at hooks.done.callAsync.err (/tmp/3694173/ncc/node_modules/@zeit/ncc/webpack/lib/Compiler.js:226:13)
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/tmp/3694173/ncc/node_modules/tapable/lib/HookCodeFactory.js:32:10), <anonymous>:6:1)
    at AsyncSeriesHook.lazyCompileHook (/tmp/3694173/ncc/node_modules/tapable/lib/Hook.js:154:20)
    at onCompiled (/tmp/3694173/ncc/node_modules/@zeit/ncc/webpack/lib/Compiler.js:224:21)
    at hooks.afterCompile.callAsync.err (/tmp/3694173/ncc/node_modules/@zeit/ncc/webpack/lib/Compiler.js:552:14)
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/tmp/3694173/ncc/node_modules/tapable/lib/HookCodeFactory.js:32:10), <anonymous>:6:1)
    at AsyncSeriesHook.lazyCompileHook (/tmp/3694173/ncc/node_modules/tapable/lib/Hook.js:154:20)
    at compilation.seal.err (/tmp/3694173/ncc/node_modules/@zeit/ncc/webpack/lib/Compiler.js:549:30) reported: true }
2018-11-23T07:08:25.646Z worker exited with code 0 and signal null

Webpack options like allow top-level require can help with this issue.

CC @igorklopov

Require is unable to resolve JSON files without explicit ".json" extension

For example, I'm trying to build an app with FIrebase Admin SDK, and it fails.

npx ncc index.js -o bundle.js

Module directory "/node_modules/@google-cloud/firestore/build/src/v1beta1" attempted to require "./firestore_client_config" but could not be resolved, assuming external.

/node_modules/@google-cloud/firestore/build/src/v1beta1 folder looks like this:
screenshot 2018-11-19 at 11 44 59

And the require() looks like this const gapicConfig = require('./firestore_client_config');

Cache

We need a simple yet effective strategy to make several invocations of ncc build or ncc run in a row re-use as much work as possible.

  • FS cache for the CLI
    • Uses the most appropriate location for each OS (like yarn)
    • Flag to ignore it for both run and build (--no-cache)
    • Subcommands to find it and clean it (ncc cache dir and ncc cache clean)
  • In-memory interface for the programmatic API (ncc())

Filenames of resources must be unique to avoid a crash

Considering the examples for NCC with Apollo and an abstract structure with schemas as resources and sharing the same filenames. This will cause NCC to compile the first/last file only, and throwing an error.

schemas\
      blogs\
           index.js
           schema.gql
      posts\
           index.js
           schema.gql

Output

   3kB  /schema.gql
6127kB  /index.js

At the moment this can only be resolved by renaming one of the schemas and make sure the file names are unique:

const typeDefs = String(fs.readFileSync(__dirname + "/schema2.gql"));

Output

   1kB  /schema2.gql
   3kB  /schema.gql
6127kB  /index.js

Add examples folder

  • Hello world
  • Multiple bundles (e.g.: cli.js -> dist/cli/ and main.js -> dist/main)

Supporting require.resolve dynamic passing

Within the use cases around custom loaders (think Babel plugins, webpack loaders), there are a number of edge cases of dynamic require that come up. While Webpack can get quite far in computing dynamic requires like require(require.resolve('./asdf.js')), there is a tricky case where there is a separation between the resolution and the require:

// example use case something like "plugin: require.resolve('./asdf')" being passed as an argument
const req = require.resolve('./asdf.js');
require(eval('"' + req + '"')); // or any other untracable logic before passing to require

In this case, what Webpack does is replace the require.resolve part with the ID of the module in the webpack bundle, so we get something like:

const req = 234;
__webpack_require__(123)(eval('"' + req + '"'))()

Where 123 is effectively an inlined "throwing" module which will give the error "Module 234 not found".

The above then fails as a module not found, all of the time.

The naive fix I was thinking to implement would to instrument the "throwing module" (123 in the example) to first check the webpack_require cache for the numeric ID.

FS inlining support

Lack of fs inlining breaks modules like:

  • saslprep
  • pdfjs (fontkit dep)

It yields a warning in mongoose, but that's due to find-package-json, which I think might not be statically analyzable[1]

[1] famous last words

Consider inlining files that are read via the `fs` API

If I manually "fix" #2 by adding iconv-lite, I get:

▲  play-ncc/ node out.js
fs.js:122
    throw err;
    ^

Error: ENOENT: no such file or directory, open '/Users/rauchg/Projects/play-ncc/data.trie'
    at Object.openSync (fs.js:448:3)
    at Object.readFileSync (fs.js:348:35)
    at /Users/rauchg/Projects/play-ncc/out.js:1:85193
    at createCommonjsModule (/Users/rauchg/Projects/play-ncc/out.js:1:677)
    at Object.<anonymous> (/Users/rauchg/Projects/play-ncc/out.js:1:85090)
    at Module._compile (internal/modules/cjs/loader.js:707:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:718:10)
    at Module.load (internal/modules/cjs/loader.js:605:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:544:12)
    at Function.Module._load (internal/modules/cjs/loader.js:536:3)

This thread is for us to discuss the static analysis possibilities around detecting filesystem reads and embedding them in the resulting artifact.

Issue with dynamically required files

I'm using a custom Hapi made server and I use this kind of require a lot (for loading plugins, methods and route :
await server.register(require(filepath))

I use this within loops that read folders and files to load everything dynamically.
It's not working (using run or build, either way), the required object/library is not available.

Ignore option

Sometimes an output bundle is designed to be a dependency, not final destination, and that would be useful to keep some its deps not packed, to share them with adjacent modules. Or the case of optional dependencies, included only if they are required by the adjacent modules.

With rollup-plugin-commonjs there is ‘ignore’ option for that.

Is there any plan to include that for ncc?

`aws-sdk` errors

Code: (sdk.js)

const AWS = require("aws-sdk");
AWS.config.apiVersions = {
  s3: "2006-03-01"
};

module.exports = async (req, res) => {
  var s3 = new AWS.S3();
};

Output:

▲  aws-sdk/ node sdk.js
/Users/rauchg/Projects/now-demos/aws-sdk/sdk.js:1
(function (exports, require, module, __filename, __dirname) { "use strict";function _interopDefault(e){return e&&"object"==typeof e&&"default"in e?e.default:e}var util=_interopDefault(require("util")),fs=_interopDefault(require("fs")),crypto=_interopDefault(require("crypto")),stream=_interopDefault(require("stream")),dgram=_interopDefault(require("dgram")),os=_interopDefault(require("os")),path=_interopDefault(require("path")),string_decoder=_interopDefault(require("string_decoder")),events=_interopDefault(require("events")),timers=_interopDefault(require("timers")),https=_interopDefault(require("https")),http=_interopDefault(require("http")),buffer=_interopDefault(require("buffer")),domain=_interopDefault(require("domain")),url=_interopDefault(require("url")),querystring=_interopDefault(require("querystring"));function JsonBuilder(){}function translate(e,t){if(t&&null!=e)switch(t.type){case"structure":return translateStructure(e,t);case"map":return translat

TypeError: Cannot read property 'memoizedProperty' of undefined
    at Object.<anonymous> (/Users/rauchg/Projects/now-demos/aws-sdk/sdk.js:1:5034)
    at Module._compile (internal/modules/cjs/loader.js:707:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:718:10)
    at Module.load (internal/modules/cjs/loader.js:605:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:544:12)
    at Function.Module._load (internal/modules/cjs/loader.js:536:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:760:12)
    at startup (internal/bootstrap/node.js:308:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:878:3)

Ditch stdout and make `-o` be a directory

If we want to support #28 and potentially non-inlined FS, we can't make the signature have the assumption that you target a specific file.

Instead, we need an output directory, in which for the vast majority of programs there will be only one file.

Render outputs

▲ ncc run
101kb  dist/index.js   [3ms]
Listening on localhost:3000                         # program output
▲ ncc build
 10kb  dist/a.mem     
101kb  dist/index.js   [3ms]
  • Sort by size
  • Include time elapsed on last entry
  • Use bytes and ms for human-friendly sizes and times
  • Add a --quiet option to suppress ncc output

Add more integration tests

  • memcached
  • stripe
  • mailgun
  • segment
  • bugsnag
  • mariadb
  • hot-shots
  • google cloud sdks
    • storage
    • bigquery
  • azure sdks
    • storage
    • cosmosdb new
    • cosmosdb old

File with shebang doesn't work

The first script I tried ncc on failed like this:

(node:71332) UnhandledPromiseRejectionWarning: Error: Hash: c93a3aedc91c7aab0770
Version: webpack 4.26.0
Time: 127ms
Built at: 2018-11-27 16:04:22
 Asset     Size  Chunks  Chunk Names
out.js  4.7 KiB       0  main
Entrypoint main = out.js
[0] ./2-stats2boundaries.js 1.11 KiB {0} [built] [failed] [1 error]

ERROR in ./2-stats2boundaries.js
Module build failed (from (webpack)/ncc/asset-relocator.js):
SyntaxError: Unexpected character '#' (1:0)
    at Parser.module.exports.pp$4.raise (/Users/stevebennett/npm/lib/node_modules/@zeit/ncc/dist/ncc/asset-relocator.js:16125:13)
...

The first line of that script is:

#!/usr/bin/env node --max-old-space-size=4096

`auth0` errors

ncc version: 0.0.4

Source:

var AuthenticationClient = require("auth0").AuthenticationClient;

var auth0 = new AuthenticationClient({
  domain: "{YOUR_ACCOUNT}.auth0.com",
  clientId: "{OPTIONAL_CLIENT_ID}"
});

crashes with:

Use of eval is strongly discouraged, as it poses security risks and may cause issues with minification
(node:30620) UnhandledPromiseRejectionWarning: Error: Illegal reassignment to import 'commonjsHelpers'
    at error (/usr/local/lib/node_modules/@zeit/ncc/node_modules/rollup/dist/rollup.js:3460:30)
    at Module.error (/usr/local/lib/node_modules/@zeit/ncc/node_modules/rollup/dist/rollup.js:13349:9)
    at MemberExpression$$1.disallowNamespaceReassignment (/usr/local/lib/node_modules/@zeit/ncc/node_modules/rollup/dist/rollup.js:11372:26)
    at MemberExpression$$1.deoptimizePath (/usr/local/lib/node_modules/@zeit/ncc/node_modules/rollup/dist/rollup.js:11340:18)
    at AssignmentExpression.bind (/usr/local/lib/node_modules/@zeit/ncc/node_modules/rollup/dist/rollup.js:9920:19)
    at ExpressionStatement$$1.NodeBase.bind (/usr/local/lib/node_modules/@zeit/ncc/node_modules/rollup/dist/rollup.js:9183:23)
    at IfStatement.NodeBase.bind (/usr/local/lib/node_modules/@zeit/ncc/node_modules/rollup/dist/rollup.js:9183:23)
    at IfStatement.bind (/usr/local/lib/node_modules/@zeit/ncc/node_modules/rollup/dist/rollup.js:10807:31)
    at Program.NodeBase.bind (/usr/local/lib/node_modules/@zeit/ncc/node_modules/rollup/dist/rollup.js:9179:31)
    at Module.bindReferences (/usr/local/lib/node_modules/@zeit/ncc/node_modules/rollup/dist/rollup.js:13303:18)
(node:30620) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:30620) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Accessing `module.id` breaks

Log from the UI https://zeit.co/olstenlarck/tunnckocore-release-now-v2/agkrauawz

Error while initializing entrypoint: TypeError: u.startsWith is not a function
    at Object.<anonymous> (/var/task/user/index.js:152:281)
    at Object.<anonymous> (/var/task/user/index.js:153:30)
    at __webpack_require__ (/var/task/user/index.js:21:30)
    at Object.<anonymous> (/var/task/user/index.js:130:19)
    at Object.<anonymous> (/var/task/user/index.js:146:30)
    at __webpack_require__ (/var/task/user/index.js:21:30)
    at module.exports.module.exports.module.deprecate (/var/task/user/index.js:85:18)
    at Object.<anonymous> (/var/task/user/index.js:88:10)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)

Minification

Currently, minimize: true being turned on is failing for an unknown reason with all integration tests.

We should be mindful here to only enable optimizations that are deemed safe for most Node.js programs and npm modules (including those that perform runtime introspection or reflection).

The pros / cons of this have to be carefully evaluated in contrast with what #51 provides.

"Cannot read property `filter` of null" error related to asset relocation

Simple repro fails to compile on [email protected]:

console.log(function (a, b) {
  return b.filter(function (c) {
    return __dirname.length;
  });
});
$ npx @zeit/ncc build t.js
npx: installed 1 in 2.631s
(node:3287) UnhandledPromiseRejectionWarning: Error: Hash: 79a94431771e08d2182b
Version: webpack 4.26.0
Time: 100ms
Built at: 11/26/2018 7:00:52 PM
 Asset      Size  Chunks  Chunk Names
out.js  4.86 KiB       0  main
Entrypoint main = out.js
[0] ./t.js 1.27 KiB {0} [built] [failed] [1 error]

ERROR in ./t.js
Module build failed (from ../.npm/_npx/3287/lib/node_modules/@zeit/ncc/dist/ncc/asset-relocator.js):
TypeError: Cannot read property 'filter' of null
    at walk (/Users/nrajlich/.npm/_npx/3287/lib/node_modules/@zeit/ncc/dist/ncc/asset-relocator.js:7465:27)
    at walk (/Users/nrajlich/.npm/_npx/3287/lib/node_modules/@zeit/ncc/dist/ncc/asset-relocator.js:7443:26)
    at walk (/Users/nrajlich/.npm/_npx/3287/lib/node_modules/@zeit/ncc/dist/ncc/asset-relocator.js:7482:20)
    at walk (/Users/nrajlich/.npm/_npx/3287/lib/node_modules/@zeit/ncc/dist/ncc/asset-relocator.js:7500:20)
    at module.exports.module.exports (/Users/nrajlich/.npm/_npx/3287/lib/node_modules/@zeit/ncc/dist/ncc/asset-relocator.js:7533:7)
    at computeStaticValue (/Users/nrajlich/.npm/_npx/3287/lib/node_modules/@zeit/ncc/dist/ncc/asset-relocator.js:1962:12)
    at leave (/Users/nrajlich/.npm/_npx/3287/lib/node_modules/@zeit/ncc/dist/ncc/asset-relocator.js:2061:32)
    at visit (/Users/nrajlich/.npm/_npx/3287/lib/node_modules/@zeit/ncc/dist/ncc/asset-relocator.js:840:3)
    at visit (/Users/nrajlich/.npm/_npx/3287/lib/node_modules/@zeit/ncc/dist/ncc/asset-relocator.js:830:5)
    at visit (/Users/nrajlich/.npm/_npx/3287/lib/node_modules/@zeit/ncc/dist/ncc/asset-relocator.js:835:4)
    at compiler.run (/Users/nrajlich/.npm/_npx/3287/lib/node_modules/@zeit/ncc/dist/ncc/index.js:63613:23)
    at finalCallback (/Users/nrajlich/.npm/_npx/3287/lib/node_modules/@zeit/ncc/dist/ncc/index.js:64538:39)
    at hooks.done.callAsync.err (/Users/nrajlich/.npm/_npx/3287/lib/node_modules/@zeit/ncc/dist/ncc/index.js:64554:13)
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/Users/nrajlich/.npm/_npx/3287/lib/node_modules/@zeit/ncc/dist/ncc/index.js:14672:10), <anonymous>:6:1)
    at AsyncSeriesHook.lazyCompileHook (/Users/nrajlich/.npm/_npx/3287/lib/node_modules/@zeit/ncc/dist/ncc/index.js:14274:20)
    at onCompiled (/Users/nrajlich/.npm/_npx/3287/lib/node_modules/@zeit/ncc/dist/ncc/index.js:64552:21)
    at hooks.afterCompile.callAsync.err (/Users/nrajlich/.npm/_npx/3287/lib/node_modules/@zeit/ncc/dist/ncc/index.js:64880:14)
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/Users/nrajlich/.npm/_npx/3287/lib/node_modules/@zeit/ncc/dist/ncc/index.js:14672:10), <anonymous>:6:1)
    at AsyncSeriesHook.lazyCompileHook (/Users/nrajlich/.npm/_npx/3287/lib/node_modules/@zeit/ncc/dist/ncc/index.js:14274:20)
    at compilation.seal.err (/Users/nrajlich/.npm/_npx/3287/lib/node_modules/@zeit/ncc/dist/ncc/index.js:64877:30)
(node:3287) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:3287) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Build failures result in exit code 0

$ cat t.js
{

$ npx @zeit/ncc build t.js
npx: installed 1 in 1.515s
(node:3552) UnhandledPromiseRejectionWarning: Error: Hash: 4d2c982ea3709348e69b
Version: webpack 4.26.0
Time: 87ms
Built at: 11/26/2018 7:03:27 PM
 Asset      Size  Chunks  Chunk Names
out.js  3.72 KiB       0  main
Entrypoint main = out.js
[0] ./t.js 134 bytes {0} [built] [failed] [1 error]

ERROR in ./t.js 2:0
Module parse failed: Unexpected token (2:0)
You may need an appropriate loader to handle this file type.
| {
>
    at compiler.run (/Users/nrajlich/.npm/_npx/3552/lib/node_modules/@zeit/ncc/dist/ncc/index.js:63613:23)
    at finalCallback (/Users/nrajlich/.npm/_npx/3552/lib/node_modules/@zeit/ncc/dist/ncc/index.js:64538:39)
    at hooks.done.callAsync.err (/Users/nrajlich/.npm/_npx/3552/lib/node_modules/@zeit/ncc/dist/ncc/index.js:64554:13)
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/Users/nrajlich/.npm/_npx/3552/lib/node_modules/@zeit/ncc/dist/ncc/index.js:14672:10), <anonymous>:6:1)
    at AsyncSeriesHook.lazyCompileHook (/Users/nrajlich/.npm/_npx/3552/lib/node_modules/@zeit/ncc/dist/ncc/index.js:14274:20)
    at onCompiled (/Users/nrajlich/.npm/_npx/3552/lib/node_modules/@zeit/ncc/dist/ncc/index.js:64552:21)
    at hooks.afterCompile.callAsync.err (/Users/nrajlich/.npm/_npx/3552/lib/node_modules/@zeit/ncc/dist/ncc/index.js:64880:14)
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/Users/nrajlich/.npm/_npx/3552/lib/node_modules/@zeit/ncc/dist/ncc/index.js:14672:10), <anonymous>:6:1)
    at AsyncSeriesHook.lazyCompileHook (/Users/nrajlich/.npm/_npx/3552/lib/node_modules/@zeit/ncc/dist/ncc/index.js:14274:20)
    at compilation.seal.err (/Users/nrajlich/.npm/_npx/3552/lib/node_modules/@zeit/ncc/dist/ncc/index.js:64877:30)
(node:3552) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:3552) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

$ echo $?
0

Dependency installation

When I use go and go.mod, installing deps is a side-effect of execution or building.

▲  golang-example/ (master) cd hello/
▲  hello/ (master) ls
.               ..              go.mod          hello.go
▲  hello/ (master) go run hello.go
go: finding rsc.io/quote v1.5.2
go: finding rsc.io/sampler v1.3.0
go: finding golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c
go: downloading rsc.io/quote v1.5.2
go: downloading rsc.io/sampler v1.3.0
go: downloading golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c
Hello, world.

I think ncc should do the same, with the ability to opt-out.

Asset wildcards and directory emission

Currently the static analysis of assets only supports emitting "files". Whenever there is a static expression that exactly corresponds to a file, that file is emitted.

Many dynamic loading cases use dynamic directory emission though:

exports.getThing = function (name) {
  return fs.readFileSync(__dirname + '/things/' + name);
}

Now one thing we could do here is adjust the asset emission to support directory expressions as well.

This requires supporting wildcards in static analysis - when name is encountered, basically treating it as a wildcard string path and not stopping but continuing the static analysis. So in the above analysis when /path/to/things/* is formed as an expression, we see that it matches real files and then emit the folder with numeric deduping, along with all files matching the wildcard such that any loads within that folder will work out.

It gets a bit tricky, but it is very much possible to do sensibly.

This would fix super important real-world use cases like cowsay.

Source-maps

  • On by default for ncc run
  • Off by default for ncc build (IMO)

Issues with `strong-globalize`

I'm using Loopback to build my project. Loopback/Stroploop internally uses Globalize and cldr for globalization in strong-globalize. Looks like they use webpack while building.

And I'm getting the following error:

16753kB  /index.js                 [12444ms]
internal/modules/cjs/loader.js:589
    throw err;
    ^

Error: Cannot find module 'cldr'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:587:15)
    at Function.Module._load (internal/modules/cjs/loader.js:513:25)
    at Module.require (internal/modules/cjs/loader.js:643:17)
    at require (internal/modules/cjs/helpers.js:22:18)
    at Object.<anonymous> (/tmp/86f4f7db49348/index.js:77424:18)
    at __webpack_require__ (/tmp/86f4f7db49348/index.js:21:30)
    at /tmp/86f4f7db49348/index.js:90591:4
    at Object.<anonymous> (/tmp/86f4f7db49348/index.js:90598:2)
    at __webpack_require__ (/tmp/86f4f7db49348/index.js:21:30)
    at Object.<anonymous> (/tmp/86f4f7db49348/index.js:306120:18)

Native addons support

Assuming the target machine is the same architecture as the current machine (for this reason the suggestion would be to implement this behind a --binaries flag or similar), here's a suggested approach for remapping and emitting binaries.

The assumption is that the "patterns" for binary emission are generally only of a few types, and that we can build static analysis to detect those patterns and provide an adequate substitution for them through (1) detecting where the binary is based on this static analysis and (2) rewriting the locator code as we do for other fs relocations to point to this new binary location.

Detection Cases (not comprehensive, just what I can find on a quick review):

  1. Statically detect any calls to node-pre-gyp's find method. (literally detecting require('node-pre-gyp').find and all its natural simple variations. Eg like in https://unpkg.com/[email protected]/src/grpc_extension.js.
  2. Statically detect any calls to require('bindings')('binding.node') and its associated lookup algorithm.
  3. Direct requires to .node binaries being treated as emitted relocated externals

Rewriting

The rewriting just full replaces the statically known path. Eg require('node-pre-gyp').find(staticExpression) -> __dirname + '/emitted.node'. This is exactly like the asset relocator already does, but simply extended to these more complex patterns.

There may still be edge cases that are not analyzable, but this partial executor already takes into account all path functions, __dirname and __filename expressions. We can tailor the analysis to the real-world test cases, and likely get quite far here too.

Go-style signature

The more I think about it, the more I'd like to support:

$ ncc run

and

$ ncc build

We would:

  • Add support for pkg.main / index.js if no argument is supplied
  • run to build to a temporary location and clean it up on a best-effort basis
  • build to output to a default dist directory if -o is not supplied (in tandem with #30)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.