Deliverables

• should support the common 80% of use cases
• no config necessary
• config inferred from package.json
• should only need to provide input module path and output bundle path

Available options

vantage.js

Pros:

• debug compilation problems interactively, as they come up
• enter configuration as its needed, and then output a config file at the end to reproduce the compilation
• integrate with the dev server
• integrate with other plugins

Cons:

• more complex than other available options

This should be separate from the built-in Interlock.compile method. Not clear yet what this will look like exactly.

Non-JS requires should result in either 1) transformation into JavaScript and treated like any module, or 2) output of that module with eventual bundles and alternate transformation pipeline.

• publish packages
• use new APIs to attach source-document information to node loc properties and sourcemaps
• close PR: babel/babel#3323

Can determine at run-time what HTTP version is supported by the browser. Will effect loading of JS modules based on HTTP/2 feature availability.

May be implemented as extensible runtime module resolution plugin (see #28).

Relates to and is blocked by #31.

Plugins should be able to register "module providers". When a request occurs for an uninstalled module (identified by hash), the default behavior is to check the URLs hash and make a request for that script.

However, if a provider is registered, that provider will be called with two arguments, the moduleHash and the next callback. moduleHash is the hash string of the requested module. The next callback should be invoked by the provider if that provider is unable to fulfill the request. This next callback will invoke the next provider (if there is one), ultimately terminating with the default behavior (or an error, if that fails).

Providers should be invoked in order of their registration.

This should lead the way to caching mechanisms (like localStorage and IndexedDb) and a patch mechanism (to be implemented in interlock-patch).

• Create plugin to generate a manifest for source builds.
  • Map modules to bundle filenames (identical to URLs hash included in bundles).
  • Map module uri (ns:nsPath) to module hashes.
• Create plugin to ingest and apply manifest.
  • Resolve any files referred to relative to the manifest.
  • Optionally take modulesPath property for ingest-module resolution.
  • Override resolve behavior such that require strings containing a matching ns/nsPath will be replaced with mapped hashes.
  • Any require strings mapped to manifest modules should not be recursively inspected.
  • Override getUrls, removing any entries where ingest module hashes are the keys, and replacing them with the same key, but the name of the corresponding ingest bundle.
  • Remove modules with ingest module hashes from bundles being generated.
  • If bundle has had all modules removed (because of matching module hashes in the injest), remove it from the bundle stream.
  • Manifest bundles should be included in output bundle stream via emitRawBundles.

Create plugin in interlock-share repo.

Blocked by #18.

• no config necessary
• config inferred from package.json
• should only need to provide input module path and output bundle path

Add feature similar to require.ensure, but call it require.async.

• These function calls should be detected in much the same way as require(...) calls.
• Modules should still be recursively discovered and included in build.
• Build-time behavior should be identical - require.async has no effect on splitting. That is still managed by the build config itself.
• Run-time behavior is different - module is not included in dependencies hash, and is not guaranteed to have been invoked prior to containing module.
• When require.async (or compiled equivalent) is invoked at run-time, 1) modules are resolved and invoked, 2) callback is invoked with params matching the required modules.

Example input

const libA = require("./lib-a");
require.async(["./lib-b", "./lib-c"], function (libB, libC) {
  // ...
});

Example output

{
  'aaaaaa': {
    deps: [],
    fn: function (require, module, exports) {
      var libA = require("bbbbbb")
      require.async("cccccc"/*["dddddd", "eeeeee"]*/, function (libB, libC) {
        // ...
      });
    }
  },
  'bbbbbb': {
    deps: [],
    fn: function (require, module, exports) {
      module.exports = "libA";
    }
  },
  'cccccc': {
    deps: ["dddddd", "eeeeee"],
    fn: function (require, module, exports) {
      module.exports = function (cb) {
        cb(require("dddddd"), require("eeeeee"));
      };
    }
  },
  'dddddd': {
    deps: [],
    fn: function (require, module, exports) {
      module.exports = "libB";
    }
  },
  'eeeeee': {
    deps: [],
    fn: function (require, module, exports) {
      module.exports = "libC";
    }
  }
}

When the require.async is invoked within the example module, the Interlock will call the cccccc module as if it were an entry point - first ensuring that all dependencies are installed and invoked, resolving/installing/invoking any missing dependencies, and finally calling the cccccc module itself. This module's exports will be a function that invokes a callback, requiring and passing constituent dependencies as parameters.

The require.async implementation will have recorded the callback passed to it, and pass that to cccccc's returned function when available.

• custom error prototype chain
• lots of test cases

Haskell's QuickCheck for JavaScript jsVerify seems like it could be really well used here. In short it is a tool where you define the properties of your output rather than the value itself. Doing so allows it to test random inputs and verify their outputs which can help identify edge cases.

This tool/technique lends itself to pure functions and comes from the functional world so it may be well suited to some of the areas of this project.

Blocked by #42.

This was errantly removed, although all the underlying functionality is still there.

repo

Blocked by #28.

repo

This is not dependent on #20. Should provide functionality comparable to:

• stylus-loader - transform Stylus files into raw CSS
• autoprefixr - transform raw CSS into vendor-prefixed CSS
• style-loader - load CSS onto page when required via new <style> tag
• all of the above should be toggleable.

repo

May utilize joi.js or validate.js. Or no third-party dependency.

Interlock Design Objectives

This issue is meant to track high-level objectives for Interlock v1.0. It is subject to revision as the work progresses.

First-class support for large teams

• each module has a unique, canonical ID that is independent of build time or machine, and dependent on module path, namespace, contents (version), and dependencies
• modules that have already been downloaded once should never be downloaded again, unless bundled with a module that has not already been downloaded
• module-to-URL resolution is easily overridable (should be able to modify the module-request mechanism altogether). (#28)
• simple hooks are provided for CDN integration at build-time
• simple mechanism is provided to import and consume pre-compiled JavaScript from application code (#29)

Run-time performance

• code splitting
• asynchronous loading of split files (#30)
• JS applications that have been compiled separately can both execute in the same browser environment without special configuration
• applications that depend on shared resources should require minimal configuration for those dependencies, and zero knowledge of each other
• under HTTP/1 bundle profile, ALL modules are bundled into the same JS EXCEPT where split points are defined
• under HTTP/2 bundle profile, NO modules are bundled into the same JS EXCEPT where split points are defined (all other files are emitted as individual files) (#31)
• hybrid runtime loader can determine which JS bundles to load based on HTTP/2 feature availability (#32)
• loader can leverage swappable caching mechanisms where browser support is available, including: (#28)
  • Local Storage (#33)
  • IndexedDB (#34)
  • conditional loading from CDN (#35)

Build-time performance

• completely async/non-blocking build process
  • stream- and promise-based implementation
  • remove all synchronous fs calls - async Pluggables only
• use AST as end-to-end intermediate data-structure during build process
  • Note: Webpack already parses JS into AST three times during build, which is a very expensive operation. JS is also parsed into AST by ESLint, Babel, and other tools.

Friendliness to application developers

• built-in, first-class support for ES2015 and JSX
• support for CommonJS, AMD, and ES2015 modules
• first-class support for Babel plugins (#36)
• workflows optimized for three use cases / personas:
  • new/mid-tier dev, a browserify-like experience; one command line, no config necessary, just inputs and outputs (#18, #37)
  • mid-tier/advanced dev, drive compilation through well-documented and familiar config
  • advanced dev, composability: support custom pipelines that are separate from built-in interlock compile method (#18, #38)
• CLI tool with flags supporting the common 80% of use cases (#18)

Powerful extensibility

• plugin system is well documented
  • documentation is automatically generated
  • freshness and correctness of documentation is enforced by CI
• plugin system supports multiple concurrent compilations
• plugin system does not sacrifice either the readiblity or the "pure functional" architecture of the code base
• extension points are discoverable through documentation and through code
• for each extensible piece of code, there are be pre-defined ways that the output can be extended (i.e. different extension points do not have different interfaces)
  • transform-input
  • override
  • transform
• multiple types of asynchronicity are supported (promises, streams, and synchronous code)
• where practical, extension points work with AST all the way through: "Hello Babel-, ESlint-, and Esprima-plugin writers; you are welcome here."
• official plugins
  • customizable sourcemapping URL plugin (#39)
  • handlebars loader plugin
  • json loader plugin

Parity with other tools

• partial/cached compile and watch mechanism
• fast and straightforward dev server (#42)
• hot module reloading mechanism (#43)
• non-JS assets are supported both as JavaScript and as non-JS via alternate pipelines (#44)
• source-maps
  • source-included
  • source-separate
• validation for config (#19)
• helpful error messages (#45)

Tests

• 100% test coverage (#46)
• two or three end-to-end integration tests (#47)
• test to ensure that example app always works (#48)

Documentation

• website (#49)
• getting started tutorial (#50)
• 4-10 "recipes" for common use cases (#51)
• API documentation (#52)
• extension documentation (#53)

General philosophy

• embrace the Zen of Python and in particular:
  • There should be one-- and preferably only one --obvious way to do it.
  • Explicit is better than implicit.
  • Simple is better than complex. Complex is better than complicated.
  • Readability counts.
• only diverge from functional patterns as a special case
• adopt AST as the "right" intermediate data-structure for JavaScript
• one more time, because it is important... "readability counts"

• single .babelrc and .eslintrc
• manage interdependencies with lerna
• manage build/test/watch with JS

As modules are recursively resolved and generated, the plugin should track an esImported value for every module that is found.

The esImported value will be false in either of the following conditions:

• the module was imported using require or AMD syntax
• the module was imported like import foo from "./bar"

However, the esImported value will be a truthy array if the module was only ever imported like import { subFeature } from "./bar" or import { otherFeature } from "./bar". In this case, the esImported value will be an array with string elements "subFeature" and "otherFeature".

After all modules have been resolved and generated, the plugin will take another pass over each module, looking for export nodes at the root level of the AST. For each module with a truthy esImported value, any export nodes that do not match an element in the esImported array will be removed.

Lastly, sub-dependencies that are no longer needed must be detected and removed from all dependency lists. This is because the function that consumed a particular dependency may have been purged. Some investigation may be necessary to ensure that we don't just implement a minifier. If it proves to be too difficult, we should have a two-stage process - one step before a minifier is applied and one step after, at which point any removed requires would have to be detected.

Notes:

• Base features should work without a minification step.
• Can tree shaking be extended to a subset of comparable CommonJS-style imports? One approach can be to remove entries from the export object and let a minifier clean up declared but unused functions/imports. This could be detected using either import { foo } from "./bar" or var foo = require("./bar").foo. This will not work with all packages, but may help with common use-cases (like Lodash).
• How should removal of exports impact the module's hash? Should there be a second hash step for modules that are streamlined? (yes, there should)

Sanity check that changes to build process don't destroy anything when run in supported browsers.

In theory, it should be possible to add full support for webpack loaders in Interlock (see Writing a Loader).

https://github.com/estools/esmangle

• watch for source changes and recompile
• serve static assets
• hot reloading
• consider integration with a separate tool for this functionality (e.g. amok [1] [2])

If babelConfig value is not supplied via config file or command-line flag, the babel config should be loaded on a per-file basis. This means that, whenever a file is about to be transformed (and possibly before parsing), we should:

• check for a .babelrc file in the file's ancestor path,
• fallback to package.json babel config, and finally
• assume no babel transformation otherwise.

Plugin that will result in UMD bundle output. In addition to supporting AMD, CommonJS, and global variable registration, it should also register itself with an Interlock run-time if present.

As my understanding and experience with Most monadic streams has progressed, I've identified a number of characteristics of those streams that don't align well with the problem space. Unfortunately, they don't seem well suited to Interlock and add subtle complexity.

Goal: Remove most as a dependency, and replace all previous operations with bluebird. Promises will be the foundational primitive of Interlock async behavior.

Blocked by #28.

repo

Blocked by #28.

May involve separate utility to generate JS that will:

• register a module provider
• map module hashes to CDN URLs
• resolve requests for module hash to URL and create script tag

Implement the interlock-signed plugin.

This plugin will ensure that bundles have not been tampered with, and allow for trusted use of third-party hosting for bundles (such as a public CDN). The general implementation will be as follows:

• add an additional provider with the following behavior:
  • on load: check for a global variable containing a map of bundle filenames to their SHA-1 hashes
  • on load: if global variable not found, notify the user with an alert() and set internal state such that any requests go to the next runtime bundle provider
  • on module request: if the module is not contained within a hashed bundle, fall back to the next runtime module provider
  • on module request: if module is contained within a hashed bundle, create an XHR request for the bundle in question, rather than a script tag
  • on request success: when XHR 200 response comes back, generate a SHA-1 hash of the bundle's raw text [1]
  • on request success: if the hash does not match, notify the user with an alert()
  • on request success: if the hash does match, create a new <script> tag with a src value of data:text/javascript,... [2]
• add transform to emitRawBundles step
  • generate SHA-1 hashes of bundle.raw values [1]
  • create map of bundle filenames to SHA-1 hashes
  • append output file to array of bundles, which when loaded sets a global variable that contains the filename:SHA-1 map

Constraints:

• script containing hashes must be loaded as a separate script or inlined into the HTML on initial page load
• server hosting signed bundles must support XHR requests for the loaded site

[1] http://www.movable-type.co.uk/scripts/sha1.html
[2] https://developer.mozilla.org/en-US/Add-ons/Code_snippets/Rosetta, line 45 of code sample

New pattern for defining plugins should be:

return function (override, transform, transformInput) {
  transformInput("readSource", args => {
    // ...
    return args;
  });
};

💃

Relates to #36.

Whereas under the default HTTP/1 bundle profile, all modules are bundles into the same JS except where split points are defined, under the HTTP/2 profile, no modules will be bundled into the same JS except where split points are defined.

• output one JS file per input file
• output manifest that maps inputs to outputs, and includes dependencies
• build HTTP/2 server push demo using dependency manifest, nghttpx, and docker

Blocked by #44.

Build on top of postcss.

https://github.com/substack/node-browserify#bignorefile
https://webpack.github.io/docs/webpack-for-browserify-users.html#ignore

Where the define function is RequireJS, the below code should result in the inner function being invoked with the modules bundled with Interlock.

define(["interlock!underscore:lib/index.js"], function (_) {
  // ...
});

RequireJS config can be configured to alias certain paths with their interlock! equivalent.

This should take the form of a plugin. See the browserify handbook for details on how transforms work.

Currently, plugin function definitions look like:

function (override, transform, control) {
   // maybe do something with control.CONTINUE    
};

Instead, remove control and add pertinent symbols to the related functions. So:

function (override, transform) {
   // maybe do something with override.CONTINUE    
};

Instead of:

{
  entry: {
    "./app.js": "app.bundle.js"
  },
  split: {
    "./lib.js": "lib.bundle.js
  }
}

do instead:

{
  bundles: {
    "./app.js": "app.bundle.js",
    "./lib.js": {  dest: "lib.bundle.js", entry: false }
  }
}