formidablelabs / builder Goto Github PK
View Code? Open in Web Editor NEWAn npm-based task runner
Home Page: https://github.com/FormidableLabs/builder
License: MIT License
An npm-based task runner
Home Page: https://github.com/FormidableLabs/builder
License: MIT License
Related to #16
Currently, we have only one archetype, builder-react-component
that has devDependencies
that both:
We don't necessarily want to force project dev deps unrelated to the project ones, so consider a more nuanced scheme of:
dependencies
be controlled by ARCHETYPE/dev/package.json:dependencies
and not ARCHETYPE/package.json:devDependencies
utils/dev.js
script to first read and stash ARCHETYPE/dev/package.json:dependencies
, then preserve those when mutating over the root package.json.When I run builder run build, the CSS files I'm importing such as
import './MuiLauncher.css';
are not being included with my .js files in the /lib folder. If this is intentional, any ideas of how I can work around it? This is commonly done in our UI library.
Instead of concurrently running lots of npm scripts
tasks, instead split concurrent processes on a list of environmental variables to be parsed and passed in to exec
environment in an OS-agnostic way.
Use case: Selenium tests and ROWDY_SETTINGS
$ builder envs test-func \
ROWDY_SETTINGS="local.phantomjs" \
ROWDY_SETTINGS="local.firefox" \
ROWDY_SETTINGS="sauceLabs.chrome_43_Windows_2012_R2_Desktop" \
ROWDY_SETTINGS="sauceLabs.firefox_38_Windows_2012_R2_Desktop" \
ROWDY_SETTINGS="sauceLabs.IE_9_Windows_2008_Desktop" \
ROWDY_SETTINGS="sauceLabs.IE_10_Windows_2012_Desktop" \
ROWDY_SETTINGS="sauceLabs.IE_11_Windows_2012_R2_Desktop"
OR
$ builder envs test-func \
'[ { "ROWDY_SETTINGS":"sauceLabs.IE_8_Windows_2008_Desktop" }, \
{ "ROWDY_SETTINGS":"sauceLabs.IE_9_Windows_2008_Desktop" }, \
{ "ROWDY_SETTINGS":"sauceLabs.IE_10_Windows_2012_Desktop" }, \
{ "ROWDY_SETTINGS":"sauceLabs.safari_7_OS_X_10_9_Desktop" }, \
{ "ROWDY_SETTINGS":"sauceLabs.safari_8_OS_X_10_10_Desktop" }, \
{ "ROWDY_SETTINGS":"sauceLabs.chrome_43_OS_X_10_10_Desktop" }, \
{ "ROWDY_SETTINGS":"sauceLabs.chrome_43_Windows_2012_R2_Desktop" }, \
{ "ROWDY_SETTINGS":"sauceLabs.firefox_9_Windows_2012_R2_Desktop" }, \
{ "ROWDY_SETTINGS":"local.firefox" } ]'
Probably leaning toward the latter (JSON string) as:
{}
)Current strawman:
$ builder envs \
--tries=3 \
--queue=3 \
--envs-path=FILE_PATH \
TASK \
JSON_STRING
npm
supports defining scripts starting with pre
and post
to be run before and after the named script. E.g. you can define predev
which would be run prior to running dev
. This is a very useful lifecycle hook, and a use case for it is to generate an Ecology docs index.html
from static-index.jsx
prior to starting up builder run docs-dev|docs-hot
(FormidableLabs/victory-component-boilerplate#20).
From the npm
docs:
Additionally, arbitrary scripts can be executed by running [
npm run myscript
]. Pre and post commands with matching names will be run for those as well (e.g. premyscript, myscript, postmyscript).
pre<task>
, post<task>
.-- --FLAG
to main task like npm
"scripts": {
"prepretemp": "echo PREPRE",
"postpretemp": "echo POSTPRE",
"pretemp": "echo PRE",
"temp": "echo TEMP",
"posttemp": "echo POST",
"preposttemp": "echo PREPOST",
"postposttemp": "echo POSTPOST",
}
Note: A --
passed flag only attaches to <task>
, not pre or post for both npm
and yarn
$ npm run temp -- --hi
> [email protected] pretemp /Users/rye/scm/fmd/builder
> echo PRE
PRE
> [email protected] temp /Users/rye/scm/fmd/builder
> echo TEMP "--hi"
TEMP --hi
> [email protected] posttemp /Users/rye/scm/fmd/builder
> echo POST
POST
$ yarn run temp -- --hi
$ echo PRE
PRE
$ echo TEMP --hi
TEMP --hi
$ echo POST
POST
Note - npm: If running a pre<task>
command directly, a prepre<task>
or postpre<task>
will not run.
$ npm run pretemp
> [email protected] pretemp /Users/rye/scm/fmd/builder
> echo PRE
PRE
$ npm run posttemp
> [email protected] posttemp /Users/rye/scm/fmd/builder
> echo POST
POST
Note - yarn: If running a pre<task>
command directly, a prepre<task>
or postpre<task>
will run, unlike npm
$ yarn run pretemp
yarn run v1.3.2
$ echo PREPRE
PREPRE
$ echo PRE
PRE
$ echo POSTPRE
POSTPRE
$ yarn run posttemp
yarn run v1.3.2
$ echo PREPOST
PREPOST
$ echo POST
POST
$ echo POSTPOST
POSTPOST
Courtesy of @chaseadamsio
The idea is that ARCHETYPE/package.json
has some warts from the scripts
not really being for the archetype's dev / release workflow but for builder projects. This would formally split things to have a package.json
that is only for developers of the archetype
Side note: If we're re-org-ing structure, probably worth discussing / consolidating builder init
layout too.
/packages
-- archetype # this gets published as the archetype
---- package.json # and configs and stuff
-- archetype-dev # this gets published as the archetype-dev
---- package.json # by itself
-- package.json # this is where we actually have the archetype lifecycle scripts
If using builder concurrent
to run multiple tests in parallel, or builder envs
to run something like a build matrix, it would be nice to disable the short-circuiting behavior when one task fails.
Especially in the case of the build matrix, you don't want one environment failing to cancel the whole build – you want to know the full set of results (e.g. imagine if Sauce Labs cancelled your jobs for every other browser just because a test failed in one browser).
Probably as a part of builder-support
(?) or builder
if we have to.
builder help ARCHETYPE
from archetype without needing to be installed in a project.README.md
Idea: Have a list of tasks that are:
npm install
in victory-voronoi
fails.
victory-voronoi
depends on victory-animation
, which runs a postinstall script: builder run npm:postinstall
, which fails because rimraf
can't be found.
victory-animation
uses the builder-react-component
archetype, which rimraf
is part of.
rimraf
also happens to be a dependency of victory-voronoi
, but I assume it isn't installed yet at the time victory-animation
runs its' postinstall script.
$ npm -v
2.14.7
$ node -v
v4.2.2
> [email protected] postinstall /Users/per/dev/formidable/victory-voronoi/node_modules/victory-animation
> npm run build-lib
|
> [email protected] build-lib /Users/per/dev/formidable/victory-voronoi/node_modules/victory-animation
> npm run clean-lib && babel --stage 0 src -d lib
/
> [email protected] clean-lib /Users/per/dev/formidable/victory-voronoi/node_modules/victory-animation
> rimraf lib
sh: rimraf: command not found
npm ERR! Darwin 15.0.0
npm ERR! argv "/Users/per/.nvm/versions/v4.2.2/bin/node" "/Users/per/.nvm/versions/v4.2.2/bin/npm" "run" "clean-lib"
npm ERR! node v4.2.2
npm ERR! npm v2.14.7
npm ERR! file sh
npm ERR! code ELIFECYCLE
npm ERR! errno ENOENT
npm ERR! syscall spawn
npm ERR! [email protected] clean-lib: `rimraf lib`
npm ERR! spawn ENOENT
npm ERR!
npm ERR! Failed at the [email protected] clean-lib script 'rimraf lib'.
npm ERR! This is most likely a problem with the victory-animation package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! rimraf lib
/cc @exogen @ryan-roemer
Builder should come with (or should have a supplementary package) the ability to publish builder-<archetype>-dev
packages without the archetype maintainer needing to do anything other than the following:
postpublish
script for the dependency)This package / helper should:
<archetype>
and <archetype>-dev
versions<archetype>-dev
versionCurrently, builder-react-component
does this, but the actual process should be abstracted out into an easy to consume package to remove any duplicated code and provide a less abrasive workflow for publishing.
When a component (like victory-bar
) is:
the npmv3 flattening leaves it unable to find the archetype (which is not in victory-bar/node_modules/builder-react-component
).
Writing tests for the init
task and one of the things I'd like is to silence logs in testing, except for when there's a reason to test the logger, something like this maybe?
var testCheck = function (callback) {
if (process.env.NODE_ENV === "test" && !process.env.TEST_LOGGER) { return; }
callback();
};
....
info: function (type, msg) {
testCheck(function () {
cons.log([chalk.green(wrapType(type)), msg].join(" "));
});
},
warn: function (type, msg) {
testCheck(function () {
cons.warn([chalk.yellow(wrapType(type)), msg].join(" "));
});
},
error: function (type, msg) {
testCheck(function () {
cons.error([chalk.red(wrapType(type)), msg].join(" "));
});
}
Opening another ticket here per @boygirl 's request... See FormidableLabs/victory#97 for more details & examples.
While I was working on Victory, I discovered that builder
scripts which call node binaries like babel
end up using the globally installed versions of those binaries, rather than the local versions in node_modules
. This is contrary to the behavior of regular old npm
scripts which use the local versions. This can cause major problems and build inconsistencies if the user's global module version does not match the local version.
In investigating FormidableLabs/radium#420, I'm beginning to suspect that the users' shell configuration is a contributing factor. For example, you can configure your shell in a strict mode so that if any command fails in a cmd1 || cmd2
chain, the whole command still fails, as if they were joined with &&
. It's possible these people have some global setting that's causing the ||
to fail (yes, even though NPM attempts to run your script with plain old sh
).
Anyway, whether or not that is the issue, I dug into how npm
runs scripts, and found that they use spawn
(they wrap Node's spawn
slightly). In order to not have to parse the command, they run sh -c
which takes the shell command to run as a string.
While I'm fine with exec
, I thought I'd bring this back up because it's a difference in how Builder runs your script vs. NPM – and I anticipate that difference being a source of issues and confusion in a future where Builder is used more widely.
(We can put this off for now, just thought I'd add it as a TODO.)
Preliminary Matter: This will need two PRs:
builder
: The init supportbuilder-react-component
: Has completely untested templates in WIP branch https://github.com/FormidableLabs/builder-react-component/compare/feature-init Please take this branch over.Tasks
builder-init
via CLI inputbuilder-init
without CLI input somehow (needed for travis).builder-react-component
Some helpful guidlines
async.js
control flows for async stuff.require()
's are ok)A user can bootstrap a project off an archetype. Use case:
$ mkdir foo && cd foo
$ git init .
$ npm install -g builder-init
$ builder-init builder-react-component
The first thing builder init ARCHETYPE
does is internally
$ npm install builder-react-component
And from there, the ARCHETYPE's templates, etc. can be read.
After that, the user is asked about ARCHETYPE's context variables needed to
inflate the templates:
During the init, the user should be prompted for the following questions
for builder-react-component
.
OrgName
: (1) change to orgName
, (2) ask ;)projectNpmDescription
: Description for package.json:description
and README.orgGitHubName
projectNpmName
This other information should be automatically inferred:
dateYear
: Current year.componentClassName
: Derived from Pascal-casing projectNpmName
projectComponentPath
: Uses value in projectNpmName
since right now they can't be skewed.Things that aren't handled currently:
Once all of those questions are answered, builder init
then:
package.json
which we'll template in from scratch).npm install
with the newly-templated package.json
builder run check
to verify everything works.Open questions at this point:
builder run build
? This creates dist/
files which shouldn't be in initial git commit probably.builder run check
when builder init
runs"? Again, would like to avoid an ARCHETYPE/init-meta.js
meta config file if possible, but...As part of the testing of the scaffolding, we should test in Travis via something like the following script
commands in the builder-react-component/.travis.yml
(don't use the templated one).
A psuedo code version of the .travis.yml
additions would be:
script:
- npm --version
- npm run builder:check
# Create test directory and change into it.
- mkdir SCAFFOLD_TEST && cd SCAFFOLD_TEST
- npm install builder
# Need support for relative path to archetype probably vs. from `npm`
# Also need "no CLI input" version of `init`
- node_modules/.bin/builder init ../
- node_modules/.bin/builder run check-ci
Create a cross-OS (win, mac, linux) way of executing with environment variables.
And take a file (basically work like builder envs
does).
builder run|concurrent <task(s)> --env='{"NAME":"value"}'
builder run|concurrent <task(s)> --env-file=./JSON_FILE_OF_ENV_VARS.json
builder run|concurrent <task(s)> --renv='{_:"NAME":"value":}' --renv-sets='ci'
builder run|concurrent <task(s)> --renv-file=./JSON_FILE_OF_RENV_VARS.json --renv-sets='ci,qa'
builder run|concurrent <task(s)> --renv-url=https://example.com/JSON_FILE_OF_RENV_VARS.json --renv-sets=''
Right now, builder install
doesn't print anything until it's done, and it's annoying to sit there and wait without the default npm progress bar. I'm assuming it's because npm only prints progress if stderr is a TTY. We can force these options when builder
runs npm (and maybe inherit any values from the environment to be nice).
I think I understand why this happens, but it's still annoying. Maybe we can brainstorm a solution.
After running builder install
, the archetype's deps get installed in your root's node_modules
, and since they're not in your root's package.json
, npm thinks they're extraneous. So commands like npm ls
will print:
npm ERR! extraneous: [email protected] /Users/brianbeck/Projects/Formidable/Builder/victory-animation/node_modules/babel-loader
npm ERR! extraneous: [email protected] /Users/brianbeck/Projects/Formidable/Builder/victory-animation/node_modules/css-loader
npm ERR! extraneous: [email protected] /Users/brianbeck/Projects/Formidable/Builder/victory-animation/node_modules/file-loader
npm ERR! extraneous: [email protected] /Users/brianbeck/Projects/Formidable/Builder/victory-animation/node_modules/style-loader
npm ERR! extraneous: [email protected] /Users/brianbeck/Projects/Formidable/Builder/victory-animation/node_modules/url-loader
npm ERR! extraneous: [email protected] /Users/brianbeck/Projects/Formidable/Builder/victory-animation/node_modules/webpack
Add configurable number of retries for command.
Use case: Selenium / Sauce wonkiness
$ builder run --retry=3 test-func-sauce
$ builder concurrent --retry=3 test-func-01 test-func-02
$ builder concurrent-env --retry=3 \
SAUCE_CHROME=chrome_latest_Windows_2012_R2_Desktop \
SAUCE_FIREFOX=firefox_latest_Windows_2012_R2_Desktop
Let's start this as a brainstorm for supporting multiple builder archetypes
builder add
command that acts normally like run
, but when traversing something like archetype1:package.json:scripts:test = "builder add foo"
and archetype2:package.json:scripts:test = "builder add bar"
, when run from project it would be effectively test: "builder run foo && builder run bar"
.# ROOT/.builderrc
archetypes:
- archetype1
- archetype2
# node_modules/archetype1/package.json
"scripts": {
"foo": "echo foo",
"test": "builder run foo"
}
# node_modules/archetype2/package.json
"scripts": {
"bar": "echo bar",
"test": "builder run bar"
}
So, builder run bar
and builder run foo
work just fine, but builder run test
needs a priority / override resolution of some type...
In .builderrc
:
---
overrides:
"builder-react-component/config/webpack/webpack.config.js": "webpack.config.js"
In all of the builder-react-component
configs that have a:
var base = require("./webpack.config");
import, we would change to something like:
var base = require(builder.resolve("./webpack.config"));
That infers and switches based on overrides
. In this case, ROOT/package.json
wouldn't need a scripts
override.
Perhaps the default behavior should be builder help
as is the case with tools like git, npm and many others.
/Users/<user>/dev/tmp/builder-test/node_modules/builder/lib/task.js:42
throw new Error("Invalid action: " + this._action +
^
Error: Invalid action: undefined - Valid actions: help, run, concurrent, envs
at new module.exports (/Users/ebaer/dev/tmp/builder-test/node_modules/builder/lib/task.js:42:11)
at module.exports (/Users/ebaer/dev/tmp/builder-test/node_modules/builder/bin/builder-core.js:17:14)
at Object.<anonymous> (/Users/ebaer/dev/tmp/builder-test/node_modules/builder/bin/builder.js:24:1)
at Module._compile (module.js:399:26)
at Object.Module._extensions..js (module.js:406:10)
at Module.load (module.js:345:32)
at Function.Module._load (module.js:302:12)
at Function.Module.runMain (module.js:431:10)
at startup (node.js:141:18)
at node.js:977:3
Hit bug on node 0.10 and 0.12 (probably others?)
For a very large webpack output, hitting this error:
$ builder run dev
# ... LOTS OF TEXT ...
[builder:proc:error] Code: 1, Command: builder concurrent server-dev server-test
[builder:builder-core:end:33238] Ended with error: run dev - stdout maxBuffer exceeded.
Likely instant hack solution: add _.extend({ maxBuffer: Infinity }, shOpts)
to runner.js
run
method.
Perhaps better solution is just add .on("data")
or something methods to drain the buffer along the way in a "normal" manner (?)
I love the idea of archetypes and staying true to NPM scripts, but I'm not sold on using another tool / task runner. I'd be really interested in using a tool like builder that instead of replacing npm run
, simply uses archetypes to add entries to my scripts section. The postinstall hooks that have been copied around, many inspired by Radium, are a great example of something that'd be nicer as an installable package. The nice thing about plain NPM scripts is that they are very approachable to newcomers, and don't require learning or installing anything new.
Nice work on pushing towards simpler builds/tasks, I'm psyched to see where this can go!
Need to add a note to remove npm:
from tasks and merge in with rest of root tasks.
An easy footgun with the existing https://github.com/FormidableLabs/builder-react-component/blob/master/package.json is if in:
// ROOT
"scripts": {
"test": "builder run check"
},
// ARCHETYPE
"scripts": {
"check": "builder run lint && builder run test"
},
and then run:
$ npm test
and watch an infinite loop of the same task...
The lint
command works just fine, but builder run test
in the archetype hits the ROOT test
task recursively firing up everything again, and again, and again, ...
TASK: Research a way to detect and deal with:
npm:test
really be something worth of npm test
in ROOT and remove archetype test
commands.test
be an example of overriding commands./cc @coopy @chaseadamsio
On repos like axis that are on builder infrastructure, npm install
will fail on builder postinstall
if these packages include other builder packages.
failing CI in this PR is an example:
FormidableLabs/victory-axis#40
victory-label
is using builder
If my test
command is something like mocha
or wdio
that comes from node_modules
, builder run test
works fine, but builder envs test --envs-path=envs.json
spits out:
/bin/sh: mocha: command not found
Builder should start out with whatever $PATH
additions come standard with builder run
, then merge in stuff from envs
on top of that.
(In case it matters: I'm using Builder just for its task running features, without an archetype.)
It might be nice to have a builder-cli
module like karma-cli and grunt-cli that just points to local builder and warns if it does not find it:
https://github.com/karma-runner/karma-cli/blob/master/bin/karma
I know builder installed globally does look for a local builder. But in some ways I like the fact the karma and grunt cli just fail when they can't find a local install.
Having builder-cli
would establish that local install is preferred. You could still enable builder to be globally installed to save space but it could warn by saying: 'you are using a globally installed version of builder'
Telling the user to update the path makes it harder to teach people how to install builder in a way that works for both Windows and Linux.
Use:
try {
// Require resolve it
var modPath = require.resolve(name);
pkgPath = path.join(modPath, "package.json");
return require(pkgPath);
} catch (err) {
/*eslint-disable no-empty*/
}
Which has problems identified by @exogen in #31 (comment)
Task:
https://github.com/kimmobrunfeldt/concurrently is nice but has some limitations. Ideally we create a:
builder concurrent FOO BAR
Task target that runs conncurrently like concurrently...
We use:
"foo": "NODE_ENV=foo foo bar"
Which works fine in Mac/Linux, but not windows.
TASK
Should add a note near https://github.com/FormidableLabs/builder#project-root with the following information:
ARCHETYPE
and ARCHETYPE-dev
on the backend in Node.js, the frontend via Webpack needs some enhancements.Strawman / example:
// webpack.config.test.js
// ... STUFF ...
// Get Paths to give node_modules by resolving based on assumed presence of
// `package.json`.
var _archNodeModules = function (arch) {
var archDir = path.dirname(require.resolve(path.join(arch, "package.json")));
return path.join(archDir, "node_modules");
};
module.exports = {
// ... STUFF ...
resolve: _.merge({}, prodCfg.resolve, {
modulesDirectories: [
"node_modules",
_archNodeModules("ARCHETYPE_NAME"),
_archNodeModules("ARCHETYPE_NAME-dev")
]
})
// ... STUFF ...
};
Requirements:
PATH
with ROOT/node_modules/.bin
and ROOT/node_modules/BUILDER_ARCHETYPE/node_modules/.bin
And uses local builder instead if present.
Just like npm run
lists available scripts, it would be great if builder run
(omitting the command) had analogous behavior, especially since the list of scripts is buried in the archetype inside the node_modules
tree.
builder-logger
package (for use in builder-init
and maybe builder-support
.argv
values for log messages that are only available after we've imported / run a lot of code.--log-level
or something)npm link
and export USE_LOCAL=true
helpers for dev'ing in a project.When running npm install
from a project that has a dependency on a package using [email protected]
, the install will fail on builder postinstall for that package
I've only seen this when using node ~v4
and npm ~v2
, The problem does not seem to occur for node ~0.10
and npm ~v2
or node ~v4
and npm ~v3
Obviated now that we're moving to the new ARCHETYPE
/ ARCHETYPE-dev
thing for devDependencies
management.
For duplicate dependencies in ROOT/package.json
and ARCHETYPE/package.json
and ARCHETYPE/dev/package.json
we would like to know / error if there is a mismatch in dependencies.
TASK
package.json:scripts
builder check-deps
action and formulate an implementation plancheck-deps
If we're doing this, rough strawman tasks:
ROOT/package.json:dependencies
vs. ARCHETYPE/package.json:dependencies
--no-prod
flag, otherwise check.ROOT/package.json:devDependencies
vs. ARCHETYPE/dev/package.json:dependencies
--no-dev
flag, otherwise check.Change local install scenario to place local node modules first in PATH.
Any drawbacks anyone sees?
TODO
s.A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.