throw an error if a requested module does not include an origami.json

I'd like to explore using build service for loading a few very stable origami components e.g. o-errors. In order for next's application code to wait for the build service script to load it'd be good to have either a callback or an event fired.

Pros of callback

• very backwards compatible
• could be passed the exported bundle so that the application code can store a reference to it as it wishes

Pros of event

• events feels like a better paradigm to use
• doesn't require polluting window

So that I can use the crossorigin=anonymous attribute on script tags to get better error info.

Also o-cookie-message uses local storage so doesn't always work with the build service.

for Build Service v3 we should rename the modules parameter to components, since we refer to components as components -- if we make it it’s not possible to use the build service to include other packages anymore

https://origami-npm-registry-prototype.herokuapp.com is not used as the NPM_REGISTRY_URL in production, https://registry.npmjs.com is used. That could lead to errors being missed by our tests.

To catch a similar error in the future, remove https://origami-npm-registry-prototype.herokuapp.com from integration tests when components which follow the proposed Origami v2 specification are released to the official npm registry.

For a bundle that has previously been built by the build service, I never want to have to wait for a synchronous build of it again, regardless of how frequently it is requested.

Currently, for image sets to appear in the registry, they must have a file could ImageList.json in their root which lists all of the images to show in the registry. See https://github.com/Financial-Times/fticons for the schema.

This imageList has to be kept up to date in order for the registry demo to match up with what's live. This is normally done by people remembering to run the gulp task that creates the list.

This is bad because (1) people forget (2) we have a gulp task across multiple copied across multiple repos (3) we introduce a dependency on gulp which a lot of people who contribute to various image sets don't have installed.

A good solution would be to move this behaviour (generating an index of images in the /src/ folder and putting it in the root) to the build service.

• Log bundle conflicts (409) to Graphite
• Add a graph to Grafana
• Maybe alert us when it exceeds a certain number?

We should run the breaking cascade on our own documentation sites.

[originally reported by @AlbertoElias]

OBT has a bug where it builds demos indentically when building multiple of them that share the same template. Build service shouldn't have this issue as it only compiles one demo at a time, but with o-header, it generates the core demo for the enhanced one.

For consistency with bundles, should we add the following redirects?

/files/* > /v2/files/*
/v1/files/* > /v2/files/*
/modules/* > /v2/modules/*
/v1/modules/* > /v2/modules/*

The code that handles these is exactly the same. Could we see this causing any issues, apart from the additional load of the redirects. Can we get logs for these routes?

Need clarification on this:

polyfills (Optional) If present and set to 'none', does not add polyfills to the output. Use this if your bundle is conflicting with other polyfills (e.g. through the Polyfill Service).

Which polyfills are we talking about? What features do I need to get from polyfill service if I set polyfills=none on the build service?

n-notification demos aren't working because..?

If I have this script tag:

<script async src="https://origami-build.ft.com/v2/bundles/js?modules=o-tracking"></script>

...then later I dynamically add another script tag to load another Origami module:

<script async src="https://origami-build.ft.com/v2/bundles/js?modules=o-expander"></script>

...does the second one overwrite the Origami global exported by the first one? Or does each script check for an existing Origami global and just append its own new properties if found? i.e. will I end up with a single global called Origami with two properties 'o-expander' and 'o-tracking'?

What

Reading about pika web got me to thinking about how similar moves could make the build service esm module ready, and reduce the need for bundlers.

Description

e.g. could import { oThingy, oGoody } from 'https://build.origami/.... in your script

Raises the question of whether there should be convenience urls for individual modules e.g. https://www.ft.com/__origami/service/build/v2/esm/o-thingy@semver, and that all of these import their deps rather than bundling

Just an idea for discussion rather than a serious feature request at this stage. Though I would happily be a beta tester

• Integrate with Biz-Ops via the NodeJS API Client -- https://github.com/Financial-Times/biz-ops-client

Originally posted by @chee in #391 (comment)

We currently have a parameter polyfills which can be used to disable the adding of Babel polyfills. We'd like to make this default behaviour, but this would be a breaking change to the API.

We talked about generating a whitelist based on our logs of users currently using the unversioned build service url (/bundles instead of /v2/bundles) and not allowing any further users to use the unversioned one, forcing them to add the explicit version to the url

What

Let's make the build service work for spec v2 origami components, which are only install-able from npm and not from bower.

Here is the proposal -- https://github.com/Financial-Times/origami/pull/86/files
Here is the spec v2 draft -- Financial-Times/origami-website#273

Details

We need to have the build service work for our components when they all update to be compliant with the v2 spec.

Currently the build service installs only from bower, we will need to make it be able to install from npm as well.

Additional information

npm installs are trickier than bower install because the npm install can run scripts from the installed packages (and does so by default) which is a potential security risk for the build service. when we run npm install we need to ensure we run it like this:

npm install --production --no-audit --ignore-scripts 

That will only install production dependencies, which should lead to a faster completion of the install. It will also not run the scripts from the packages, which protects us from the potential security risk. Finally it will not run npm audit at the end of the install phase, which should lead to a faster completion of the install.

Currently the project is using a fixed this.This probably shouldn't exist as a fixed list.
It should use an origami-repo-data api call to get the list similar to origami-component-converter and fallback to a fixed list if the origami-repo-data api call fails.

We can reuse some code in origami component converter which does a similar thing --
https://github.com/Financial-Times/origami-component-converter/blob/10990dac98e980594a329f0cd68537b0e51a6560/lib/npm.js#L8-L17

Originally posted by @JakeChampion in #391 (comment)

Currently OBS uses a Github username and password in order to access any private repositories. It might be a safer approach to use an OAuth token which can be limited to just reading private repositories rather than using an account which may have more privileges than necessary.

Whitelist the components allowed to have demos compiled (Only Origami components). We can probably do this by checking that the component exists on the Origami Bower registry.

Currently it is possible to compile demos for any component on the global Bower registry.

Let's be explicit and remove the idea of a default brand.

Currently we do not enforce these two spec rules:

• Consecutive dashes or dots not allowed.
• 50 characters or less.

We should add those rules to the regex here

This is something the image service does and has proved useful when finding the product owners behind requests.

Example urls:
https://www.ft.com/origami/service/image/v2/images/raw/http://www.elblogdelinfo.com/wp-content/uploads/alb_elias.jpg?width=400

https://www.ft.com/origami/service/image/v2/images/raw/http://www.elblogdelinfo.com/wp-content/uploads/alb_elias.jpg?width=400&source=alberto

• The main usecase for the files endpoint is to include font files from o-fonts-assets.
• Other uses appear to be old component assets (file endpoint Splunk statistics).
• No current component version includes assets in this way. None will in the same way going forward, as it will not work with npm. There's a proposal for handling component assets in the future.
• If we require a system code to include fonts, and pass through the system code of the project consuming Origami, the client won't cache fonts between pages of the ft (pretty major, fonts are heavy)
• Instead we decided it would be better to remove the file endpoint from v3 and host fonts in S3. We could do this just for o-fonts-assets in whatever way is easiest, or perhaps we go ahead and implement the asset proposal for all components?

Relates to: #408

All information can be found at http://healthcheck.ft.com/validate?server=origami-buildservice-eu.herokuapp.com&host=origami-buildservice-eu.herokuapp.com&protocol=https

If you can't view that, I've pasted the issues below:

Check #1: businessImpact is required
Check #2: businessImpact is required
Check #4: businessImpact is required
Check #8: businessImpact is required
Check #9: businessImpact is required
Check #10: businessImpact is required

After undergoing the npm install on OBS build on a pretty old Mac mini which took 12 minutes. I think a notification either by the terminal or via OS X to inform you when the task has been completed would be incredibly useful to have.

How do I make a Mac terminal pop up alert via AppleScript

Currently the README advocates storing confidential environment variables inside docker-compose.yml for local development. This becomes a bit messy because that file is included within git but will become personal to each developer's set-up (API keys, auth tokens, database names etc). A developer may accidentally commit their environment variables into the git history (which is a bad thing tm).

Have you thought about using dotenv instead of placing the environment variables within docker-compose.yml?

docker-compose.yml is commited into the repo whereas .env can be ignored in .gitignore. This means confidential env vars can be stored in it safely and because the file is ignored by git it won't be deployed to production.

This would be a breaking change, would need to be a new OBS v3 endpoint.

It loses whitespace, so it looks like this:

The HTML source is easier:

Could we put it in a PRE instead?

Currently we do not run integration tests against the Heroku applications, this makes it hard to have confidence in the deployed code to actually work as we expect it to.

If/when github goes down, I still want to be able to request any bundle that has previously been built by the build service.

From: Financial-Times/ft-origami#438

Recent stuff like http://git.svc.ft.com/projects/OT/repos/build-service/pull-requests/58/overview has me thinking: it's quite hard to get people to implement a CTM test, particularly so when they are using the build service. If performance is critical to your application you won't be using the OBS anyway because you will be wanting to custom-compile the components with your own JS and CSS, so...

Could we add an endpoint to the build service that layers a CTM on top of the JS bundle call? So you'd do this in your page:

<script src="//build.origami.ft.com/v2/bundles/js?modules=o-fonts,o-header,...&amp;ctm=ft1"></script>

Now instead of returning the bundle you requested, OBS instead returns this, with very generous cache headers:

(function() {
var cutsTheMustard = ('querySelector' in document && 'localStorage' in window && 'addEventListener' in window);
if (cutsTheMustard) {
   document.documentElement.className = document.documentElement.className.replace(/\bcore\b/g, 'enhanced');
   var o = document.createElement('script');
   o.async = o.defer = true;
   o.src = "//build.origami.ft.com/v2/bundles/js?modules=o-fonts,o-header,...";
   var s = document.getElementsByTagName('script')[0];
   s.parentNode.insertBefore(o, s);
}
}());

Problem

I want to use polyfill from the Polyfill Service and not the ones given by OBS. Now that the Polyfill Service is adding more ES6 polyfills we are seeing clashes between the polyfills from OBS and the ones from the Polyfill Service. Currently we can sidestep this issue by excluding these polyfills from the Polyfill Service and using the polyfills from OBS however this means that our end-users lose the benefit of the polyfill service as every single user will receive the polyfills given by OBS instead of only the users which require them.

Solution

Have a query parameter which when set, disables the babel-runtime set of polyfills. OBT has had a flag to disable the babel-runtime for a few months, meaning this problem doesn't exist for developers who use OBT directly.

It looks like we are not explicitly setting a cache time for the deprecated bundle redirects. We can probably set a long time to cache the redirect for as we know for certain that bundle will always be redirected.

Follow up of: #188

If no errors are captured by Raven:

• Throw a real error.
• Add integration tests.

Else:

• Remove code. Find other solution.

We seem to have picked up some memory issues between the 3.0.4 and 3.0.5 production deploys. Version 3.0.4 was deployed to production on 2016-03-15, and 3.0.5 was deployed on the 2016-03-18. This graph shows the change in memory usage from 12th–22nd:

The drops in usage correspond to Heroku restarting the dynos, I imagine the usage would continue to trend upwards if the restarts weren't happening.

[suggestion from the origami expert user forum]
Have a page which throws together the latest versions of components, with the latest version numbers, that a user can select to generate a url which they can then copy/paste into their application. Similar to the image service docs. (suggestion from @wheresrhys )

origami-build-tools has got to version 10 and we can not upgrade it.
we made obt5fork which is a stripped down version of obt5 with only the features required for the build service

https://github.com/Financial-Times/obt5fork

Easy to do with circle

Ideally the tests should be runnable in either the container or the developer's machine. Currently they're pretty much only runnable on the dev machine. In CI we probably want to run the integration tests against the container.

[originally reported by @AlbertoElias]

which is returned as a comment in bundles

We need to do a bit of out-reach about shrinkwrapping and why it's important.

The way that shrinkwrap URLs are calculated doesn’t account for non-semver versions in the bower config (e.g. a git URL or GitHub username/repo).

For example: o-techdocs includes highlight.js, but from our shim version: https://github.com/Financial-Times/o-techdocs/blob/master/bower.json#L7

Currently when the shrinkwrap URL is generated it references highlight.js@undefined rather than Financial-Times/highlight.js-shim@<commit-hash> (which works).

I'm looking into whether I can fix this without breaking other shrinkwrap URLs. Unfortunately this means that current shrinkwraps will never work, unless I add in a specific case to always serve our version of highlight.js (which would be super hacky).

Steps to reproduce:

1. Visit https://build.origami.ft.com/v2/bundles/css?modules=o-header@^4,o-techdocs@^5,o-colors@^3,o-fonts@^2,o-forms@^2,o-buttons@^3 – this should output a working bundle
2. Copy the shrinkwrap URL (should be something like this) – this should output a 404 error, outlining that danielditgens/highlight.js doesn't exist

Workaround (temporary)

Replace highlight.js%40undefined in the shrinkwrapped URL with Financial-Times/highlight.js-shim@5eb4727ba777e7683140de72ba01468e81639827 (like this) – this should output a working bundle.

The module is unreliable and it gets free space values by parsing a table output by the df command. The df command output is different on different Linux distributions, which caused some issues when moving to Alpine.

Also we currently maintain a forked version of the module and so have to install it through git. We could remove git as a dependency and reduce the container size if we find a good npm-hosted module.

Currently the tests can fail quite easily for unrelated reasons if, for example, a new version of a component is released. We should probably use Regular Expressions for testing some of the output to ensure that this doesn't happen.

There are probably some other improvements we can make (I'll list here when I think of them)