Why do we build?
- Only send the client the bits they need
- Develop in a more powerful syntax
- Keep 3rd party libraries out of your source tree
- Apply transformations like minification
- package.json & npm init
- devDependencies
- src and dist
- version control & .gitignore
- package versioning strategies
- npm-check-updates
- npm shrinkwrap
Note there is an up-and-coming new dep mananagment layer on top of npm/replacing npm coming - yarn.
It's good to both get advised on when particular of your packages have security issues, and also to keep up to date with the latest versions of your dependencies (although you'll need to manually vet if an upgrade has breaking changes to your application, usually by looking at their release log - hopefully you have automated tests to help detect regressions). We can add some tooling to assist us manage our project dependencies.
In a terminal, in your project folder:
npm install --save-dev npm-check-updates retire
In the scripts object in package.json
:
"deps:check": "./node_modules/.bin/npm-check-updates && ./node_modules/.bin/retire",
"deps:update": "./node_modules/.bin/npm-check-updates -u && ./node_modules/.bin/npm-check-updates -a && npm install && ./node_modules/.bin/retire"
npm run deps:check
will inform you if particular of your packages are now out
of date, or have security warnings published.
npm run deps:update
will automatically update your package.json
file to the
latest versions (you want to be more judicious about running this command,
particularly for established projects), install those updates, and rerun the
security check to see if any of those new versions have issues and/or is
upgrading resolved issues you had previously.
If there are packages that retire
indicates have security issues but you have
determined that those security issues don't affect you, you can create a
retireignore.json
file in your project root which documents exceptions. E.g.:
[
{
"path" : "node_modules/webpack-dev-server",
"justification" : "Only used in dev. This avoids the reporting of some minor jQuery issues."
},
{
"path" : "node_modules/tether",
"justification" : "Only used in dev (by webpack-dev-server). This avoids the reporting of some minor jQuery issues."
}
]
Your frontend build pipeline should limit itself to just the frontend build. If
your project has a deployment story, you just need to kick off the f/e build as
part of that process (e.g. ansible or fabric etc. would run npm install
and
npm build
). It is also totally possible to pass in project-level configuration
via command line arguments to your build process using a npm module like
yargs.
- How complex is your project?
- Is there existing tooling?
- Do you want to use ES6/ES2015?
Two choices we will explore today:
Gulp for simple .scss
and jQuery projects.
Webpack for more complex projects using JS modules and ES6.
If you have your own github account already, you might prefer to fork this repository and clone that instead. If you don't just run the following commands on a terminal or command line interface (assuming that your machine already has git available):
git clone https://github.com/jenofdoom/js-build-pipelines-training.git
cd js-build-pipelines-training
First, we need to install node.js and its package manager, npm.
Ubuntu/Debian/Mint instructions
If you have a favourite code editor feel free to use that, but I recommend Atom.
In Atom, right click in the left panel, select Add Project Folder
and open the
js-build-pipelines-training
folder.
In our example project folder gulp-tutorial
, we want to install a dependency
into our project manifest (if you were beginning from scratch and didn't have an
existing package.json
file you'd start by running npm init
and answering the
prompts). From a terminal, in the gulp-tutorial
folder:
npm install --save-dev gulp
To start, for all files other than our JS and SCSS files (which we need to
perform special extra steps on) we just want to directly copy over with no
transformations. It would be easy to extend this later and break out particular
file types to have special tasks associated (for example, to compress .png
files).
Create a file in the root of the gulp-tutorial
folder called gulpfile.js
.
This will contain all our build pipeline configuration. Open that file in an
editor and add the following:
const gulp = require('gulp');
gulp.task('copy-files', () => {
return gulp.src('./src/**')
.pipe(gulp.dest('./dist/'));
});
gulp.task('build', ['copy-files']);
In package.json
, modify the scripts to include a new command:
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1",
"build": "./node_modules/.bin/gulp build"
},
In a terminal from the gulp-tutorial
folder you should now be able to run:
npm run build
All the src files should have been copied over to the the dist folder. Open a
file browser and locate the dist/index.html
file, right click on it and open
it in a web browser. Everything will be pretty broken right now until we get
SCSS compilation working,and integrate some JS dependencies.
This will make the configuration more centralised and prevent repetition later.
const PATHS = {
'src': {
'root': './src/**',
'scss': './src/scss/**/*.scss',
'js': './src/js/**/*.js'
},
'dist': {
'root': './dist/',
'css': './dist/css/',
'js': './dist/js/'
}
}
gulp.task('copy-files', () => {
return gulp.src(PATHS.src.root)
.pipe(gulp.dest(PATHS.dist.root));
});
gulp.task('copy-files', () => {
return gulp.src([PATHS.src.root, '!' + PATHS.src.scss, '!' + PATHS.src.js])
.pipe(gulp.dest(PATHS.dist.root));
});
This will write out empty folders as it's excluding the contents of the folders not the folders themselves, but this doesn't matter a huge amount and we'd need separate entries in the PATHS object to sort it out, without the
/\**
suffix.
npm install --save-dev gulp-sass
const sass = require('gulp-sass');
gulp.task('scss', () => {
return gulp.src(PATHS.src.scss)
.pipe(sass()
.on('error', sass.logError)
)
.pipe(gulp.dest(PATHS.dist.css));
});
gulp.task('build', ['copy-files', 'scss']);
npm install --save-dev gulp-postcss gulp-sourcemaps autoprefixer postcss-flexbugs-fixes cssnano
const postcss = require('gulp-postcss');
const sourcemaps = require('gulp-sourcemaps');
const autoprefixer = require('autoprefixer');
const flexfixes = require('postcss-flexbugs-fixes');
const cssnano = require('cssnano');
gulp.task('scss', () => {
return gulp.src(PATHS.src.scss)
.pipe(sass()
.on('error', sass.logError)
)
.pipe(sourcemaps.init())
.pipe(postcss([
autoprefixer({
browsers: ['last 2 versions'],
cascade: false,
remove: false
}),
flexfixes(),
cssnano()
]))
.pipe(sourcemaps.write('.'))
.pipe(gulp.dest(PATHS.dist.css));
});
We are going to use gulp-concat to squish together all our JS files, and gulp-uglify for minification.
npm install --save-dev gulp-concat gulp-uglify
const concatjs = require('gulp-concat');
const uglifyjs = require('gulp-uglify');
gulp.task('js', () => {
return gulp.src(PATHS.src.js)
.pipe(sourcemaps.init())
.pipe(concatjs('bundle.js'))
.pipe(uglifyjs({ mangle: false }))
.pipe(sourcemaps.write('.'))
.pipe(gulp.dest(PATHS.dist.js));
});
gulp.task('build', ['copy-files', 'scss', 'js']);
This gets all our own JS sorted, but our project has a dependency of jQuery, so we'll need to pull that in too.
npm install --save-dev jquery
We need to modify the gulp.src
statement to take an array of locations rather
than just our local JS folder (note we get the unminified version as we minify
ourselves):
return gulp.src([
'./node_modules/jquery/dist/jquery.js',
PATHS.src.js
])
At the bottom of the gulpfile:
gulp.task('watch', () => {
gulp.watch(PATHS.src.root, ['build']);
});
gulp.task('default', ['build', 'watch']);
In the package.json
scripts:
"start": "./node_modules/.bin/gulp"
In a terminal:
npm start
You can leave that running and as you save changes within the src
folder, the
project will rebuild.
(Obviously, if you didn't want to integrate Bootstrap you'd not do this bit!)
We can use gulp-sass
to import the SCSS root file for Bootstrap.
Install Bootstrap (check the downloads page to check you're getting the most up to date version):
npm install --save-dev [email protected]
In your gulpfile's scss task, alter the sass() function with a new config object:
.pipe(sass({
includePaths: [
'./node_modules/bootstrap/scss/'
]
})
.on('error', sass.logError)
)
In your main.scss
file, add:
@import 'bootstrap';
If we wanted to use Bootstrap's JS file, you'd import it into the
js
task in the same way that we did for the jQuery dep.
Refer to node_modules/bootstrap/scss/_variables.scss
to see what variables can
be customised. Make a new file in your src/scss/
folder,
_custom-boostrap.scss
, and add in your variables there. Then set up your
main.scss
file to import the variables file: it must be imported before the
main bootstrap file (due to the way that the !default declaration works):
@import 'custom-bootstrap';
@import 'bootstrap';
We can add a rule to make background colour something other than white in
_custom-bootstrap.scss
:
$body-bg: #cef1ff;
Usually, you'll be starting from an example Webpack configuration rather than building all of this out from scratch, but my aim here is to teach you from scratch so you'll know what all of the parts of the config file are doing in case you need to tweak them.
Note that we're using version 2 of Webpack (docs here), not version 1 - if you're later looking up tutorials etc. make sure you're looking at the right version, the syntax differs.
In our example project folder webpack-tutorial
, first run npm install
by
itself to get the already-specified project dependencies. Then:
npm install --save-dev webpack
(From this point on, assume any
npm install
instructions should be carried out in a terminal from inside the~/js-build-pipelines-training/webpack-tutorial
folder)
Make a file (at the same level as the package.json
file) called
webpack.config.js
with the following contents:
const config = {
};
module.exports = config;
Inside the config
object (from this point on everything we add, unless
otherwise specified, goes into the config object) in webpack.config.js
:
entry: './src/index.jsx',
At the top of the file:
const path = require('path');
In webpack.config.js
:
output: {
path: path.resolve(__dirname, 'dist'),
filename: 'bundle.js'
},
resolve: {
extensions: ['.js', '.jsx', '.json', '.css', '.scss'],
modules: [path.resolve(__dirname, 'src'), 'node_modules']
}
We use a absolute path so only this folder called
src
will be searched, not any ancestor folders that are calledsrc
.
Natively, Webpack only understands JavaScript, but if we want to get to a point where we can delegate all of our bundling to Webpack we should utilise some loaders in order to be able to also process:
- es6 aka es2015
- jsx (if we're using React)
- images and other files
- css/scss/less
We use Babel to transpile our modern JS (es6) to JS that is supported across all browsers.
npm install --save-dev babel-loader babel-core babel-preset-env babel-preset-react
Note that you only need babel-preset-react
for React projects.
module: {
rules: [
{
test: /\.(js|jsx)$/,
exclude: /node_modules/,
loader: 'babel-loader'
}
]
}
We need to provide some configuration for our Babel loader - this could go in
the webpack config but it's more typical to create a dedicated configuration
file. This is also where we can specify (if we're using React) that we want to
convert jsx to js too. Create a file called .babelrc
at the project root:
{
"presets": [
"react",
[
"env",
{
"targets": {
"uglify": true
},
"modules": false
}
]
]
}
Obviously you'd only include react
for a React project.
The env
preset has configuration setting to disable Babel's module imports,
because Webpack 2 takes care of those for us.
If you need to support older browsers you might want to configure which browsers
env
targets, see the docs for
env.
You could also add additional presets, like the stage-x presets for upcoming JavaScript features, for example preset-stage-3 includes the object destructuring spread operator.
In package.json
:
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1",
"build": "./node_modules/.bin/webpack"
},
In your terminal:
npm run build
It doesn't matter if our file compiles OK if we don't have a HTML page to load
it from. We could just hand write a file and manually add a script tag to our
bundle, but it's usually better to use a Webpack plugin called
HtmlWebpackPlugin to take
care of this for us, as it has some nice extra functionality. We already have a
simple src/index.html
file that we can use.
npm install --save-dev html-webpack-plugin
In webpack.config.js
:
plugins: [
new HtmlWebpackPlugin({
template: './src/index.html',
hash: true
})
],
Because this is a plugin, we need to include it at the top of the file:
const HtmlWebpackPlugin = require('html-webpack-plugin');
We can use webpack-dev-server to provide a development webserver for us.
npm install --save-dev webpack-dev-server
In package.json
:
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1",
"build": "./node_modules/.bin/webpack",
"start": "./node_modules/.bin/webpack-dev-server --hot --inline"
},
In webpack.config.js
we should make sure that our dev server supports HTML5
push state:
devServer: {
historyApiFallback: true
},
Now in a terminal we can leave the dev server running, and it will automatically rebuild when it sees changes are saved to the project.
npm start
The development server runs on a port on localhost (8080 by default, but you can configure what the port number is). If you need to be developing against a backend API that's normally supposed to be on the same domain, which doesn't have
Access-Control-Allow-Origin
set, you're going to run into issues with CORS. You can look at setting up a reverse proxy with nginx or Apache to sidestep this, but this will prevent the hot reload from working. Read more about SOP problems here (and avoid going down the jsonp route for a solution).
Some modern JavaScript we might want to use (beyond just ES6 syntax) is not supported in IE and some other browsers. For example, the new specification for AJAX requests that supersedes XHR, fetch, needs a polyfill for most browsers.
When we have a polyfill we want our code to include, we can normally npm install
it and the include it in our Webpack build by adding an extra entry point to our
app, so for example entry: './src/index.jsx'
might become entry: ['whatwg-fetch', './src/index.jsx'],
where
whatwg-fetch is the name of the fetch
polyfill package.
For more general polyfills, look at
babel-polyfill which pulls in
core-js. Right now our project uses
Object.values()
in the search functionality, which is not yet supported in
older browsers - we can fix this:
npm install --save-dev babel-polyfill
In webpack.config.js
, change the entry point to an array:
entry: ['babel-polyfill', './src/index.jsx'],
Any images that are referenced from within our JSX, or (later on) files that get imported through our CSS (for example, webfont files) will also be pulled into the build by Webpack. We need to tell Webpack how to deal with these files, by using the url-loader:
npm install --save-dev url-loader file-loader
In webpack.config.js
, in the module rules, underneath our JSX rule:
{
test: /\.(png|svg|jpg|gif|woff|woff2|eot|ttf)(\?v=[0-9]\.[0-9]\.[0-9])?$/,
exclude: /node_modules/,
loader: 'url-loader?limit=10000'
}
url-loader will insert a data URI rather than a separate file if the filesize is less than 10kb (as we're configured it) otherwise it automatically uses file-loader to write the file out as a separate asset.
If we want to include images or other files, we should store them somewhere on
our src
tree. In the example project I've already created a src/assets/img
folder and put a couple of images in it (photo
credit, pattern
credit).
We can now use the photo image in our JSX as an image tag (just add it at the
bottom of src/components/about/about.jsx
):
<img src={photo} />
and at the top of the file uncomment:
import photo from 'assets/img/banks-peninsula.jpg';
Optional: some files might need directly copying over to your dist folder because they aren't imported from your normal source tree (e.g. a
robots.txt
file). Look at CopyWebpackPlugin for this functionality.
Optional: if you want to set up favicons for your site you can either directly copy them over using CopyWebpackPlugin, or you can use a favicon plugin, FaviconsWebpackPlugin
To get just CSS working all we need to do is:
npm install --save-dev style-loader css-loader
And in the module rules in webpack.config.js
, below the jsx
test:
{
test: /\.css$/,
exclude: /node_modules/,
use: ['style-loader', 'css-loader']
},
Now we can include our CSS file from the project index (we could include
multiple css files, if we had them). In index.jsx
, uncomment the // import 'styles.css';
line.
File paths in the CSS (for background images) will use the same url-loader
process as images, although note that they are relative to the entry point (if
you want to do imports relative to the .scss
file, look at
resolve-url-loader). So we
can now add a background image in the styling for the body tag in
background: url(assets/img/crossword.png);
First we need to delete the reference to our styles.css
file that we don't
need any more, so we can swap to the .scss
files I've already preprepared in
the example. In index.jsx
, uncomment the // import 'index.scss';
line and
delete the import 'styles.css';
line.
We also need to uncomment the .scss
imports in several other files:
src/components/row/row.jsx
src/components/search/search.jsx
src/components/table/table.jsx
We use the sass-loader to
transform our .scss
files:
npm install --save-dev sass-loader node-sass
In the module rules in webpack.config.js
, alter the css
test:
test: /\.scss$/,
exclude: /node_modules/,
use: [
'style-loader',
'css-loader',
{
loader: 'sass-loader',
options: {
includePaths: [path.resolve(__dirname, 'src')]
}
}
]
The includePaths option means that we can split our files up, so we can both
take advantage of normal SCSS imports, and also do direct imports of .scss
files.
As well as regular SCSS it's good to post-process our CSS to use an autoprefixer (and in this instance we're also going to use postcss-flexbugs-fixes because Bootstrap requires it).
npm install --save-dev postcss-loader autoprefixer postcss-flexbugs-fixes
At the top of the configuration file:
const autoprefixer = require('autoprefixer');
const flexfixes = require('postcss-flexbugs-fixes');
Inside the use
object in the scss
test, after css-loader
and before
sass-loader
:
{
loader: 'postcss-loader',
options: {
plugins: [
autoprefixer({browsers: ['last 2 versions']}),
flexfixes()
]
}
},
(Obviously, if you didn't want to integrate Bootstrap you'd not do this bit!)
We can use sass-loader
to import the SCSS root file for Bootstrap.
Install Bootstrap (check the downloads page to check you're getting the most up to date version):
npm install --save-dev [email protected]
In your main .scss
file, uncomment:
@import "~bootstrap/scss/bootstrap";
The tilde tells the importer not to use a relative path, so it will then resolve
from the node_modules
folder.
Refer to node_modules/bootstrap/scss/_variables.scss
to see what variables can
be customised. We can use the already set up
src/base-styles/_custom-boostrap.scss
, and add in your variables there. Then
set up your base-styles/_variables.scss
file to import the default Bootstrap
variables file (just uncomment the exisiting line):
@import "~bootstrap/scss/variables"; // bootstrap variables
We can add a rule to make our primary colour green not blue in our
src/base-styles/_custom-bootstrap.scss
:
$brand-primary: #48a843;
Because of its good support of ES2015 and React, we will use ESLint for our code linting. We can build the linter into our build process so we'll see any problems in the compile process immediately.
npm install eslint --save-dev
ESLint has a configuration tool we can use to get set up quickly:
./node_modules/.bin/eslint --init
I'd suggest giving the following answers (for this project, for your own you can set up as you please):
- How would you like to configure ESLint? Answer questions about your style
- Are you using ECMAScript 6 features? Yes
- Are you using ES6 modules? Yes
- Where will your code run? Browser
- Do you use CommonJS? No
- Do you use JSX? Yes
- Do you use React? Yes
- What style of indentation do you use? Spaces
- What quotes do you use for strings? Single
- What line endings do you use? Unix
- Do you require semicolons? Yes
- What format do you want your config file to be in? JSON
This will create an .eslint.json
file for you - open it in a code editor and
change the indent to 2
on line 20 to match our current project. We also want
to change line 6 from "extends": "eslint:recommended",
to:
"extends": [
"eslint:recommended",
"plugin:react/recommended"
],
Optional: there are a bunch of other configurations you can make, for example I prefer the
"no-console"
check to "warn" rather than "error". See the rules list for more information.
Now we can add the linter to our build process. First, install the ESlint loader:
npm install --save-dev eslint-loader
In the module rules in the config object in webpack.config.js
, at the top
before the babel-loader
:
{
enforce: 'pre',
test: /\.(js|jsx)$/,
exclude: /node_modules/,
loader: 'eslint-loader',
},
Webpack has a shortcut for the most common needed configuration for production
builds, by adding -p
to the
command line. We can add this now to our build command in package.json
:
"build": "./node_modules/.bin/webpack -p",
We should also make sure we don't end up with any cruft in our build folder, so it's a good idea to always delete the old build before creating a new one:
"build": "rm -rf ./dist && ./node_modules/.bin/webpack -p",
You should see that after applying the prod flag, the bundle.js
file is
substantially smaller in size (from about 1.49 MB to 375 kB)!
It's good to have source maps enabled in production, for debugging and performance profiling. Webpack has many different schema for generating source maps, some of which are better for dev than prod.
I recommend cheap-module-source-map
for production and
cheap-module-eval-source-map
for development (based on this
advice).
There is a variable available at run time that contains the name of the npm
script (as defined in package.json
) that invoked webpack, called
process.env.npm_lifecycle_event
. So we can base our decision about what source
maps schema to use on the value of that variable - if it's "build" then we
should use the cheap-module-source-map
schema. The config key we need to
change is the rather misleadingly-named devtool
.
In webpack.config.js
, in the config object:
devtool: process.env.npm_lifecycle_event === 'build' ? 'cheap-module-source-map'
: 'cheap-module-eval-source-map',
The above should all be on one line
.map
files should now be output by the build.
Optional: rather than this crude method of having production specific configuration, we could have a common config which can be extended for different environments using webpackMerge. Or you can just have two config files, but this leads to a lot of duplication so I don't recommend it.
In order to avoid flash of unstyled content, we should use the ExtractTextPlugin to split all the compiled CSS out into a proper old-fashioned CSS file. It's fine for this to be in place for our development builds too.
npm install --save-dev extract-text-webpack-plugin
At the top of webpack.config.js
:
const ExtractTextPlugin = require('extract-text-webpack-plugin');
In the plugins array:
new ExtractTextPlugin('bundle.css')
In our module rules, we need to replace the exisiting .scss
test with one that
is wrapped with ExtractTextPlugin. So our exisiting rule:
{
test: /\.scss$/,
use: [
'style-loader',
'css-loader',
{
loader: 'postcss-loader',
options: {
plugins: [
autoprefixer({browsers: ['last 2 versions']}),
flexfixes()
]
}
},
{
loader: 'sass-loader',
options: {
includePaths: [path.resolve(__dirname, 'src')]
}
}
]
},
becomes:
{
test: /\.scss$/,
use: ExtractTextPlugin.extract({
fallback: 'style-loader',
use: [
'css-loader',
{
loader: 'postcss-loader',
options: {
plugins: [
autoprefixer({browsers: ['last 2 versions']}),
flexfixes()
]
}
},
{
loader: 'sass-loader',
options: {
includePaths: [path.resolve(__dirname, 'src')]
}
}
]
})
},
You should now get a separate CSS file written. The downside of the plugin is that although the bundle still recompiles automatically (when using the dev server), the webpage will no longer refresh automatically on style changes. For the reason you might want to consider reserving ExtractTextPlugin for your production builds only.