observing / square Goto Github PK

Add Temper as replacement for the dropped jade template inclusion support.

The plugin interface is "okay" but it's to molded to much around a middleware interface that is supported by connect / express et all..

To make the core more clear we should extract some logic.

var square = require('square/plugin'); // some plugin tools

exports.configuration = {
    key: value,
    setting: 13131
}

exports.description = "bla bla bla"
exports.name = "bla bla bla bla"
exports.type = square.modifier; // type of plugin, modifies content, or something else.
exports.version = "1.0.0"; // plugin version

exports.init = function (configuration, fn) {

};

or something simple like that

The minifier appears to convert my multibyte characters to a question mark (?) for each byte in that multibyte. This has been confirmed to be specifically the case when the code is being minified. The dev version of the JavaScript has proper encoding and keeps it's characters as it should.

(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.
Trace: 
    at StatWatcher.<anonymous> (events.js:139:15)
    at Object.watchFile (fs.js:762:8)
    at /home/arnout/repos/square/vendor/watch.js:69:10
    at Object.oncomplete (/home/arnout/repos/square/vendor/watch.js:83:19)

Happend after a long day of work.

The --bundle parameter takes the path or directory it needs to search for and defaults to finding square/bundle.json files by default. If you want to have it named differently you should supply a --filename

Instead of using 2 parameters we could just ditch the --filename flag and have users specify the filename in the --bundle instead:

square --bundle square.json
square --bundle ./directory/pewpew.json
square --bundle

Could a feature be added to control what files are generated? For example, would be nice in square.json if we could specify:

"configuration": {
"dist": {
"dev": false
}
}

Which would stop it generating the dev version of the files. I'm just interested in the min versions.

similar as the report @doenietzomoeilijk made

The module should only be using async calls so it can be deployed as a dynamic build system on servers.

Compression is one of the most important things in a build system and I have the feeling that this is currently one of the weakest parts of our system. There are a couple of issues that I would like to see addressed before we move this project forward towards the glorified 1.0 release of the development branch.

YUI compressor

YUI has recently announced the deprication of their YUI compressor project. They are moving away from their Java based compressor and are embrascing opensource as the base of their compressor which they have named yUglify as it's based upon the excellent uglifyjs compressor. The new version add's some additional fixes upon the regular uglify compressor. Some of these fixes are already in our code base as well, such as appending an extra ; so it's easier to concatenate files after minification.

The CSS compressor has been replaced by CSSMin which is included in their yUglify package. The version bundled is a forked version and maintained version of CSSMin.

UglifyJS

UglifyJS has gained a new brother, UglifyJS2 this is the successor of the current uglify parser and it should be able to compress more then the current stable version.

JSMin

We should also support JSMin as some people seem to fancy that, it's also available with sourcemaps

CSS compression

The only CSS compressor that we are currently supporting is the one bundled in the YUIcompressor.jar, we should also different CSS compressors such as sqwish

Closure compiler

Closure compiler is awesome, but it's depends that Java is installed on your system. I would love to get rid of this dependency and only require node to be installed on the system. Google created a closure compiler service that runs on app-engine. We could leverage this service to compress the data if java is not installed.

The level option

We currently allow people to configure the compressor using a level option, this is just silly and doesn't provide any useful information about what it actually does and is overly complicated to understand for new comers. Instead we should just allow a comma separated list of compressors they want to use on their code base.

Compression detection

Less, important but nice to have.. Iterate over all the available compressors to figure out which compressor is the most effective for the given code base. As there are cases where closure compiler is more effective then uglifyjs. By checking out the best compilation we could advice users on compression levels. Also the order of compression is important, for example use closure first as it remove dead code and then uglify to save a couple more extra bytes.

Verification

We should verify if the compressed code is actually smaller when gzipped, if this is not the case, we should return the original source code.

Source maps

We need this.

TL;DR

• Remove yui compressor, and upgrade to yuglify
• Support more CSS compressors
• Detect if java is available on the system, and use the closure.jar to minify. When it's not available we should use the close compiler service API to minify the data.
• All the compressing should be delegated to a child process so the operations do not block our event loop. This crush cluster should take care of all minification related procedures and use the node cluster to distribute the workload to it's child processes.
• Add compression algorithm detection
• Don't use level but add new engine or compressor option
• Verify these results.
• Add source map support

Somehow the require('less') in the try {} seems to fail even if less is globally installed. In addition, after the installation by the npm.install call the content seems to be not parsed correctly. In comparison to the others, aka stylus etc. the content is provided by the variable content while less uses output.content?

Lastly less seemed to install the itself in a new node_modules directory relative to the Makefile... But my fiddling might be the cause of that.

With my limited node skills I could not yet pinpoint the problem correctly ;) hence no pull request... I need a short headsup/readup about the generated errors. Perhaps I can free some time.

We should not expect our users to have java installed on their systems. The compressors that do require this functionality should only be enabled when java is detected on the users system. We can probably use https://github.com/isaacs/node-which to detect if Java is installed.

Currently the CSS compressor is the only compressor that requires Java to be installed as we already have other fallbacks for JS compression in the form of uglify.

We could add alternate CSS compressors to the stack, for example:

• https://github.com/ded/sqwish

When an error is triggered during building, it's possible that the writes are queued up.

notice:   changes detected, refreshing hotelpage.styl
 critical: Failed to concat all dependencies Error: hotelpage.styl:1468
   1464|       color: $link
   1465|       font-weight: 500
   1466|       margin: 0 0 0 10px
   1467|       text-decoration: underline
 > 1468| 

Failed to @extend ".sprite"

    at /home/arnout/.square/node_modules/stylus/lib/visitor/normalizer.js:158:24
    at Array.forEach (native)
    at Normalizer.extend (/home/arnout/.square/node_modules/stylus/lib/visitor/normalizer.js:156:17)
    at Normalizer.visitGroup (/home/arnout/.square/node_modules/stylus/lib/visitor/normalizer.js:137:8)
    at Normalizer.visit (/home/arnout/.square/node_modules/stylus/lib/visitor/index.js:28:39)
    at Normalizer.visitBlock (/home/arnout/.square/node_modules/stylus/lib/visitor/normalizer.js:110:19)
    at Normalizer.visit (/home/arnout/.square/node_modules/stylus/lib/visitor/index.js:28:39)
    at Normalizer.visitGroup (/home/arnout/.square/node_modules/stylus/lib/visitor/normalizer.js:139:22)
    at Normalizer.visit (/home/arnout/.square/node_modules/stylus/lib/visitor/index.js:28:39)
    at Normalizer.visitRoot (/home/arnout/.square/node_modules/stylus/lib/visitor/normalizer.js:74:23)

 notice:   changes detected, refreshing hotelpage.styl
 critical: Failed to concat all dependencies Error: hotelpage.styl:1468
   1464|       color: $link
   1465|       font-weight: 500
   1466|       margin: 0 0 0 10px
   1467|       text-decoration: underline
 > 1468| 

Failed to @extend ".sprite"

    at /home/arnout/.square/node_modules/stylus/lib/visitor/normalizer.js:158:24
    at Array.forEach (native)
    at Normalizer.extend (/home/arnout/.square/node_modules/stylus/lib/visitor/normalizer.js:156:17)
    at Normalizer.visitGroup (/home/arnout/.square/node_modules/stylus/lib/visitor/normalizer.js:137:8)
    at Normalizer.visit (/home/arnout/.square/node_modules/stylus/lib/visitor/index.js:28:39)
    at Normalizer.visitBlock (/home/arnout/.square/node_modules/stylus/lib/visitor/normalizer.js:110:19)
    at Normalizer.visit (/home/arnout/.square/node_modules/stylus/lib/visitor/index.js:28:39)
    at Normalizer.visitGroup (/home/arnout/.square/node_modules/stylus/lib/visitor/normalizer.js:139:22)
    at Normalizer.visit (/home/arnout/.square/node_modules/stylus/lib/visitor/index.js:28:39)
    at Normalizer.visitRoot (/home/arnout/.square/node_modules/stylus/lib/visitor/normalizer.js:74:23)

 notice:   changes detected, refreshing hotelpage.styl
 info:     writing /www/hotelsnl/html/css/hotelpage.min.css
 info:     writing /www/hotelsnl/html/css/hotelpage.dev.css
 info:     writing /www/hotelsnl/html/css/hotelpage.min.css
 info:     writing /www/hotelsnl/html/css/hotelpage.dev.css
 info:     writing /www/hotelsnl/html/css/hotelpage.min.css
 info:     writing /www/hotelsnl/html/css/hotelpage.dev.css

Currently we only allow .json files to be used for square configuration we should be able to support .js files as well and use the node.js require function to fetch it. This will allow users to use nodejs and env variables to build up the configuration.

/// square.js
module.exports {
  configuration: {
    dist: '/hello/world.js',
    vars: process.env
  }
}

This might also unlock the capability of using custom square plugins..

/// square.js
module.exports {
  configuration: {
    dist: '/hello/world.js',
    plugins: {
       'smashingpluginname': require('smashingpluginname')
    }
  }
}

So that's a little bit of extra niceness we could use..

Is it possible to add option with output format to jade pre-processor?

Hi,
Not sure if it's just me or not, but I have about 3-5 seconds delays of getting notifications when file has changed, is it normal?
OS: ubuntu 12.04
Cheers,
Tadek

{
  "bundle": {
    "jquery.js": {
      "description": "jQuery update based on version number in file",
      "version": "1.3",
      "latest": "http://code.jquery.com/jquery.js"
    }
  }
}

Can square download the latest if jquery.js doesn't exist as file?

Stacktrace:

/home/arnout/repos/square/lib/watch.js:92
    if (fstat.isDirectory()) return true;
              ^
TypeError: Cannot call method 'isDirectory' of undefined
    at EventEmitter.ignore (/home/arnout/repos/square/lib/watch.js:92:15)
    at EventEmitter.filter (/home/arnout/repos/square/lib/watch.js:76:37)
    at EventEmitter.emit (events.js:70:17)
    at /home/arnout/repos/square/vendor/watch.js:113:22
    at Object.oncomplete (/home/arnout/repos/square/vendor/watch.js:81:19)

This was during a build

Stylus allows you to put conditional comments in your css. We should allow developers to use this functionality to create:

• CSS based on distribution type for example include debug css for the dev distribution so you can debug you CSS grids and styling for other debug markup that you might use.
• CSS for different platforms. Allow css that is specifically build for mobile devices to share the same code base.
• Maybe some more use cases.

Convert the doc's folder's markdown pages to MAN pages so they can be used in NPM

Instead of writing to the file system we should also be able to write to something else, like a S3 bucket where we can store the assets.

This is nice ways to integrate the squire build system in existing server architectures.

Square currently only supports writing to:

• File
• STDOUT

Ideally we should add different storage backends so you can for example add your builds to a database, remote server or anything else you want to hook up.

A simular apporach as observing/devnull transports would be suitable for this: https://github.com/observing/devnull/tree/master/transports

var Square = require('square');

var square = new Square();
square.storage(require('square/storage').stdout);
square.storage(require('square/storage/disk');
square.storage(require('square/storage/s3');

Or something simular to this..

(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
    at Square.EventEmitter.addListener (events.js:168:15)
    at Square.build (/Users/V1/Projects/nodejitsu/--/node_modules/square/lib/square.js:511:8)
    at Function.update (/Users/V1/Projects/nodejitsu/--/node_modules/square/bin/square:166:14)
    at ratelimit (/Users/V1/Projects/nodejitsu/--/node_modules/square/lib/watch.js:31:8)
    at Object._.debounce.later [as _onTimeout] (/Users/V1/Projects/nodejitsu/--/node_modules/square/node_modules/underscore/underscore.js:561:30)
    at Timer.list.ontimeout (timers.js:101:19)

derp.

Hello! I use master branch version, and currently stylus output is empty, no erros just empty.

The watch module is currently really dumb, it's only watching files that are in the root directory of the square.json file. If you have a file that is fetched from an other location, it's not watching that for changes. :(

Haven't looked into it yet, but seems to be introduced between 0.9 and 0.10

When we combine a make entry and let square run multiple processes on different bundles, all but the first bundle of the bundles that share dependencies won't get triggered. E.g.

Application a.js has ../common/dep.js in a/bundle.json
Application b.js has ../common/dep.js in b/bundle.json

In the Makefile we set up:

ab.js: a.js b.js

a.js:
  square --bundle a --plugin crush --extension js --watch

b.js:
  square --bundle b --plugin crush --extension js --watch

Then we run: $ make -j2 ab.js

We see two square processes spawn, compile and watch.

Now, we change something in dep.js and save that file. What should happen is that we see two square processes get triggered, but instead we only see the square process for a.js.

Will make a pull request as soon as I get around to fix this. Basically, doing

"css/jquery.mobile.theme.styl": {          
    "pre:stylus": {                        
        "datauri": true                    
    }                                      
  , "description": "Hotels.nl mobile theme"                         
}                                          

triggers the data uri processing, but not when the file is not available, e.g. not found. However there is no error/warning emitted either. Thus you end up with compiled CSS with only the string of the file and no datauri (obviously).

The default crush level 2 brakes underscore.js, I used it in combination with backbone.js and zepto.js, but removing either persisted in the error. Setting the crush level to 5 however, solved the problem.

Possible fix: crush level per file

Config attached for easy reproduction.

{
    "configuration": {
        "dist": "/var/www/temp/{ext}/core.{type}.{ext}"
      , "jshint": {
            "laxcomma": false
        }
      , "plugins": {
            "crush": {
                "level": 2
            }   
        }
    }
  , "bundle": {
        "zepto.js": {
            "description": "Zepto.js"
          , "version": "v1.0rc1"
          , "latest": "http://zeptojs.com/zepto.min.js"
        }
      , "underscore.js": {
            "description": "Underscore"
          , "version": "1.3.3"
          , "latest": "http://code.jquery.com/underscore.js"
        }
      , "backbone.js": {
            "description": "Backbone"
          , "version": "0.9.2"
          , "latest": "http://code.jquery.com/backbone.js"
        }
    }
}

This might be me doing something wrong, but the latest (0.0.8) version of square does not wrap the merged content of type dev. This can easily be recreated with the basic.js in the tests/fixtures, by wrapping it in a anonymous function.

{                                                        
  "configuration": {                                     
    "dist": "/tmp/test.{type}.{ext}",                    
    "plugins": {                                         
      "wrap": {                                          
        "header": "(function() {\n",                     
        "footer": "})();",                               
        "leaks": false                                   
      }                                                  
    }                                                    
  },                                                     
  "bundle": {                                            
    "basic.js": {                                        
      "description": "wrap simple js to test the wrapper"
    }                                                    
  }                                                      
}                                                        

Is this intended, can't think of a reason why?

Currenlty the update plugin just downloads every single version change that happens upstream. This can lead to unsave version jumps. What we should do is change the version: field to work just like the version: field of NPM package.json files so users can specify a version range that they want to receive updates for.

There 2 issues that prevents us from implementing it currently and that is:

• we are using the version field to indicate the current version of file.
  • a solution for this would be reading the file from disk and re-parsing the version
  • store the value in a different key
• not all files use semver compatible versioning
  • we could transform the received version to a semver compatible version
  • ignore the update
  • accept the update
• sha1 keys from Github lookups are also stored in the version key
  • move it to a SHA key instead

Hi,
really like your module, made it working with no troubles, but didn't find good way of using it together with for example exress.js or any other modules. For example I have bundle with naming convention containg md5 hash, how to generate/read file name in view to match file generated by square?
Many thanks,
Tadek

Groups make the internal square code really complex because you needed to be able to generate different files based on the groups key. So you could re-use your base files in a new file without having to re-specify those.

We came up with an alternate solution to this, that would keep our code base lean. Remove groups, in favor of different square.json files for each distribution. In addition to this we will allow a new set of keys in the square.json file:

{
   "configuration": {
       // if you specify an array in the configuration section we will find those files, and merge the bundle key with this file
       // this way can create one "main" file and import it in every square.json file you want. This can be useful for configuration
       // etc
       "import": ["square.json", "other.json"]
   },
  "bundle": {
       // in addition to configuration option we will now allow strings to be set in the bundle. The key is the filename you want
       // import and the value is an `@import` statement that will get the file definition from the supplied `square.json` file.
       // this allows you to import single files from other square.json files
       "filename.js": "@import square.json"
   }
}

The only downside is that it adds more "files" to your project, but so be it.

Hi,
I researched many of asset builders in node and every one of them sucks. Square seems to be the closest I'm looking for, but I'd like to know how do you handle following situations:

1. using square in dev mode - watching for file changes and automatic recompilation, but without concatenatic/minifying files as want to have better debugging capabilities

2. producing some sort of manifest.json which would contain all filenames of compiled files , so could easily use it in express to reference those files.

3. do I have to watch for css and js changes files separately via square --watch ?

4. how to pass aditional options to stylus preprocessor config ?

5. should I use stylus @import or square comments with import directives?
   Cheers,
   Tadek

Let --watch also check for modifications in the bundle.json file

Getting error when trying to square minify using level that contains uglifyjs:

Failed to process content [ [Error: Cannot call method 'parse' of undefined] ]

Feature request, watch for changes in files inside dependancies, e.g. [square] @import

if possible add external tools imports, like stylus @import '...'

Currenlty it's not possible to supply the pre-processor modules with any options, making it less flexible then needed. We can add an extra section to the square.json file where you can specify options that need to be passed in to the pre-processors..

eg:

{
  configuration: {
     pre-processors: {
       stylus: {
          options: 'for stylus' 
      }
    }
  }
}

See #19 for the conversation

Add support for githooks

square hook --bundle . --plugin debug

or with an --hook flag

square --bundle . --plugin crush

It should recompile the code and run tests against it once #9 is implemented

{
  configuration: { plugin: { unit: { framework: 'qunit', phantomjs: 'path' } } }
, bundle: {
    test: "/dir/to/test.file.js"
  } 
}

supported unit tests

• qunit
• mocha
• jasmine

to be decided

• automatic full browser testing through http://browserling.com/ or simular sevices like http://www.browserstack.com/

Add commander.js to the canihaz dependencies as the CLI interface is required optionally.

If I have file structure like this:

dir1
  square.json
  dir2
  dir3
  libs
    square_source_code

square -b . output is critical: Failed to parse the bundle SyntaxError: Unexpected token # When I move square_source_code from dir1 everything is fine ;)

Perhaps automatically create the files that are included in the bundle if they do not exist yet, aka @ first build. Point to discuss were to store them, as there is no path required.

Feature request, title explains itself, otherwise you get a lot of duplicate content in the stdout

When a file with //[square] directive comments is parsed we should check if that file also contains directives and so forth.

After all files have been included it should be send to the pre-processor as a whole.