stefanpenner / broccoli-stew Goto Github PK

As advised by @stefanpenner in this stackoverflow question I'd like to report that excluding ember pods from Broccoli trees doesn't appear to be working.

Let say we have the following structure:

• /app/modules/components
• /app/modules/pod1
• /app/modules/pod2
• /app/modules/pod3

And this brocfile:

var EmberApp = require('ember-cli/lib/broccoli/ember-app');
var app = new EmberApp();

var Stew = require('broccoli-stew');
module.exports = Stew.rm(app.toTree(), 'app/modules/pod1', 'app/modules/pod2');

I'd expected the above should exclude app/modules/pod1 and app/modules/pod2 from ember app tree, leaving app/modules/pod3 only but it does not.

Made a repository to repro the issue:
https://github.com/WooDzu/ember-exclude-pod/tree/broccoli-stew

From a slack conversation:

locks [3:00 PM] 
I figured it out, I always get stumped by it not actually logging until consuming
and I was stopping the build before it got consumed

rwjblue [3:01 PM] 
I bet we could detect that
like when the process exits, if there are `stew.debug` or `stew.log` instances created but not "read" we print out an informative error
if you make an issue in broccoli-stew I can guide someone through it...

Using ember-cli 0.1.15. This is my Brocfile.js:

/* global require, module */

var find  = require('broccoli-stew').find;
var log   = require('broccoli-stew').log;

var tree = find('app/**/*.js');
var logTree = log(tree, {output: 'tree'});

module.exports = logTree;

There is no log output at all

https://github.com/stefanpenner/broccoli-stew/blob/master/lib/debug.js#L22-L24

gobble does this, and i think it would be great.

find([
  'lib',
  'apple'
], filter);

cc @Rich-Harris

Stuff to cover

• how to think about broccoli and broccoli-stew
  • figure out a good way to describe the virtual layered FS via symlinks concept, and how stew tries to give us unix inspired helpers to mutate this file system
• some examples, and common patterns
• some real-world example
  • https://github.com/tildeio/rsvp.js/blob/master/Brocfile.js
  • krisselden/dag-map#4
  • BackburnerJS/backburner.js#129

cc @IgorMinar

• if provided a root (via another tree) find(dist, '*.js'})
• if merely a simple expansion (create two funnels and merge) find('{foo.js,bar.js}') but not find('*') or find('{foo.js,*.css}')

/* global require, module */

var find  = require('broccoli-stew').find;
var debug = require('broccoli-stew').debug;

var tree = find('app/**/*.js');
var debugTree = debug(tree, {name: 'tree'});

module.exports = tree;

This produces nothing. To get debug to output anything, I need to completely change my Brocfile, and do, for example:

module.exports = debugTree;

Which I assume triggers the tree. This is counterintuitive (I expected that just by declaring debugTree it would trigger), and inconvenient: a single debug line is not enough; instead, I need to change several places of the Brocfile. For a complex Brocfile, where the problem is not obvious, it means lots of tampering, putting debug trees, and placing them in the export chain correctly, which is in itself error-prone, and means we need to debug the debugging.

What I would expect is that just by declaring the debug it would produce a debug output. If that is not possible for any technical reason, at least something like:

var debugTree = debug(tree, {name: 'tree'}).trigger();

Would be a good compromise. Maybe this already exists, but I can not find it documented.

The same applies to log by the way (which, as filed in a previous issue, is not even triggered when put in the export chain)

• ensure they don't happen
• fail with a useful error message

this would allow someone to quickly cat changed files contents into a function for simple transforms

var tree = cat(tree, function(inputString) { return outputString; })

var dist = cat(tree, function prepend(file) {
  return license + "\n" + file;
});

v1.4.0 is not shown in the github releases.

Log is useful for debugging, but it may also be used to leave in for future debugging sessions.

If we look to debug, we can see the value.

usage:

...
log(lint(tree), { 
  label: 'ember-cli:tree', // label to match on, and print
  debug: true // when true, only log if the label matches process.env.DEBUG (otherwise be a passthrough step
});
...

enabling:

DEBUG=ember-cli:trees* ember build

@chadhietala / @rwjblue thoughts?

The setup for each test is pretty much the same and should be broken out into their own helpers file. Just some thoughts on what this might look like.

var mv = wrapSubject(_mv, fixturePath);

afterEach(function() {
   return cleanupBuilders(function() {
    // some callback 
   });
});

BROCCOLI_ENV doesn't seem to be set when ember --environment production is ran. Instead process.env.EMBER_ENV gets set.

This doesn't ever get called for example https://github.com/tildeio/rsvp.js/pull/349/files#diff-ec6fb87583b2323d013c3e30c0a5084dR38

ember-cli/ember-fetch#7 (comment)

example:

• env

but for all of them

Would be nice to know we pass on windows...

• destDir
• getDestinationPath

• var tree = log(tree) this would log the resulting tree to sdout
• var tree = debug(true, 'name') this would emit the resulting tree to disk at process.cwd() + name/*

I think label should exist independent of debug filtering.

if options.debugOnly is true, that logger will only be enabled if env.DEBUG matches options.label

the goal of debug was to leave loggers in ember-cli, but have them engage only when the DEBUG flag matches.

cc @chadhietala

I'm sure it's some sort of incompatibility, but I'm not getting an error or any output:

var stew = require('broccoli-stew');

var log = stew.log;
var find = stew.find;

log(find('/src/**/*.js'), { output: 'tree' });

• broccoli-stew 0.2.1
• broccoli 0.15.0

Node v0.10.35

the idea would be for broccoli-stew to decompose stuff like

find('node_modules/mocha/*.{css.js}')
find('node_modules/{foo,bar}/lib/*.{css.js}')

by detecting reasonable globing patterns, into:

new Funnel('node_modules/mocha', {
  include: ['*.{css.js}']
});

new Funnel('node_modules', {
  include: ['{foo,bar}/lib/*.{css.js}']
});

// or maybe for optimizations (but only if we need to)

var a = new Funnel('node_modules/foo', {
  include: ['lib/*.{css.js}']
});

var b = new Funnel('node_modules/bar', {
  include: ['lib/*.{css.js}']
});

return merge([a,b]);

• depends on broccolijs/broccoli-funnel#9
• implement glob detection in stew

It seems that this plugin (or at least .log) uses the .read/.rebuild API which is no longer supported as of Broccoli 1.0, based on the error message.

We should add a default description to all of our generated trees to make the slow tree output more useful. Broccoli tries to grab the description based on the constructors name, but that is somewhat error prone and does not lead to really nice names. We can do better. 😃

Examples:

ef4/broccoli-sourcemap-concat#1
ember-cli/broccoli-es6modules#2
ember-cli/ember-cli#2857

In nearly all cases, stew.find is better served by just using Funnel. I think it was a good experiment. We should confirm the above, and then find a good path forward.

getting errors when using your library saying that node can't find module walk-sync

Given 2 arrays of trees with the same inputPaths, zip them together using mergeTrees from right to left leaving one array of N merged trees.

var zippedTrees = stew.zip(['tree1', 'tree2'], ['tree1', 'tree2']);
// merged ['tree1', 'tree2']

given:

- node_modules
  |--foo
      |-- app
           |--baz.js
           |--bar.js

mv(tree, 'node_modules/foo/app/', 'foo')

should result in:

- foo
   |--baz.js
   |--bar.js

also, in this scenario it should just create a single symlink node_modules/foo/app/ -> app/

as i was thinking about docs, linking to the API docs, but then sharing some handy recipes in the readme might be a good way going forward. Thoughts?

mv might be confusing because it only works at the directory level. If the mv is kept, maybe mv and rename should be combined to keep with bash mv feature parity.

Given:

var globs = ['dummy/**/*.js', 'foo/bar/*.js'];
var app = find(tree, { include: globs, exclude: ['dummy/tests']})

env('!production', function() {
  app = find(tree, {include: globs})
}) 

The second find will yak because globs is now an array of Minimatch objects.

This doesn't play nicely with binary files:

stew.map(tree, function replacer(content) {
  return content;
});

https://github.com/eviltrout/ember-performance/blob/186752c6de32331ca8356d544410ef8e33de7770/Brocfile.js#L39

Serving on http://0.0.0.0:4200/
[ 'baseline-handlebars-list/index.html',
  'baseline-render-list/index.html',
  'ember-get/index.html',
  'ember-get/primed/index.html',
  'ember-set/index.html',
  'ember-set/primed/index.html',
  'object-create/baseline/index.html',
  'object-create/index.html',
  'object-create/view/index.html',
  'render-complex-list/index.html',
  'render-list/index.html',
  'render-list-unbound/index.html' ]
ENOENT, no such file or directory 'baseline-handlebars-list/index.html,baseline-render-list/index.html,ember-get/index.html,ember-get/primed/index.html,ember-set/index.html,ember-set/primed/index.html,object-create/baseline/index.html,object-create/index.html,object-create/view/index.html,render-complex-list/index.html,render-list/index.html,render-list-unbound/index.html/'
Error: ENOENT, no such file or directory 'baseline-handlebars-list/index.html,baseline-render-list/index.html,ember-get/index.html,ember-get/primed/index.html,ember-set/index.html,ember-set/primed/index.html,object-create/baseline/index.html,object-create/index.html,object-create/view/index.html,render-complex-list/index.html,render-list/index.html,render-list-unbound/index.html/'

https://github.com/tildeio/rsvp.js/blob/master/Brocfile.js#L58

Hey,

hate to file a crazy issue like this, but in theory, shouldn’t these two Brocfile.js excerpts work exactly the same except for the console outputting a tree?

// No log

function processEyeglass(tree, options) {
  return new BroccoliEyeglass(tree, { /* opts */ });
}

// with log

function processEyeglass(tree, options) {
  var loggedTree = stew.log(tree, { output: 'tree', label: 'brocEyeglassLogTree' });

  return new BroccoliEyeglass(loggedTree, { /* opts */ });
}

But for some reason I don’t understand, the tree itself seems to change significantly. Instead of attempting to compile my Sass, now BroccoliEyeglass seems to return the input tree. Exactly.

any idea what’s going on?

I'd like to place l DEBUG- files to be in a single ignorable subdirectory.

BTW thanks a lot for this, extremely useful

I'm trying to use Broccoli to automatically populate source scripts in an index.html file. I'm new to Broccoli and noticed that it is quite tricky to get the list of files contained in a tree. I haven't been able to find a plugin that could do this in Broccoli. Then I found your plugin and I think it could help.

I noticed that .log() outputs the matched files in the tree/node directly to stdout. Is it possible to include an option in .log() to return the output as an array of strings containing the matched files, for example:

var Funnel = require('broccoli-funnel');
var Log = require('broccoli-stew').log;

var appFiles = new Funnel('src/app', { include: ['**/*.js'] });
var fileArray = Log(appFiles, { output: 'array' });

console.log(fileArray);
// fileArray = ['src/app/file1.js', 'src/app/file2.js', ... ]

There are situations where we only want to work with the list of files that matched a tree instead of the full tree object.

https://inch-ci.org/github/stefanpenner/broccoli-stew

https://github.com/eviltrout/ember-performance/blob/master/Brocfile.js

• mv should support ../../ -> #36
• find should support mergeTree case correctly
• cat will cleanup the various concat steps -> #5
• log needs to work better.