Giter VIP home page Giter VIP logo

boomcatch's Introduction

boomcatch

A standalone, node.js-based beacon receiver for boomerang. Read more.

Build Status

  • boomcatch version: 3.2.5
  • node.js versions: 0.10 and later

Installation

At the system level

First you must install node.

You can then install boomcatch via npm:

npm install -g boomcatch

Local to a node.js project

Add boomcatch to the dependencies in your project's package.json, then run:

npm install

Usage

From the command line

To see the list of command line options run:

boomcatch --help

Available options are:

  • --host <name>: Host name to accept HTTP connections on. The default is 0.0.0.0 (INADDR_ANY).

  • --port <port>: Port to accept HTTP connections on. Defaults to 80 for HTTP or 443 for HTTPS.

  • --https: Start the server in HTTPS mode.

  • --httpsPfx: PFX/PKCX12 string containing the private key, certificate and CA certs for HTTPS mode.

  • --httpsKey: Path to private key file for HTTPS mode. Ignored if --httpsPfx is set.

  • --httpsCert: Path to certificate file for HTTPS mode. Ignored if --httpsPfx is set.

  • --httpsPass: Passphrase for --httpsPfx or --httpsKey options.

  • --path <path>: URL path to accept requests to. The default is /beacon.

  • --referer <regex>: HTTP referers to accept requests from. The default is .*.

  • --origin <origin>: Comma-separated list of URL(s) for the Access-Control-Allow-Origin header. The default is * (any origin), specify 'null' to force same origin.

  • --limit <milliseconds>: Minimum elapsed time to allow between requests from the same IP address. The default is 0.

  • --maxSize <bytes>: Maximum body size to allow for POST requests. The default is -1 (unlimited).

  • --silent: Prevent the command from logging output to the console.

  • --syslog <facility>: Use syslog-compatible logging, with the specified facility level.

  • --workers <count>: The number of worker processes to spawn. The default is -1 (one worker per CPU).

  • --delayRespawn <milliseconds>: The length of time to delay before respawning worker processes. The default is 0.

  • --maxRespawn <count>: The maximum number of times to respawn worker processes. The default is -1 (unlimited).

  • --validator <path>: Validator used to accept or reject request data. The default is permissive (always accept requests).

  • --mapper <path>: Data mapper used to transform data before forwarding, loaded with require. The default is statsd.

  • --prefix <prefix>: Prefix for mapped metric names. The default is the empty string (no prefix).

  • --forwarder <path>: Forwarder used to send data, loaded with require. The default is udp.

  • --fwdHost <name>: Host name to forward mapped data to. The default is 127.0.0.1. This option is only effective with the UDP forwarder.

  • --fwdPort <port>: Port to forward mapped data on. The default is 8125. This option is only effective with the UDP forwarder.

  • --fwdSize <bytes>: Maximum packet size for forwarded data. The default is 512. This option is only effective with the UDP forwarder.

  • --fwdUrl <url>: URL to forward mapped data to. This option is only effective with the HTTP forwarder.

  • --fwdMethod <method>: Method to forward mapped data with. The default is GET. This option is only effective with the HTTP forwarder.

  • --fwdDir <path>: Directory to write mapped data to. This option is only effective with the file forwarder.

From a node.js project

var path = require('path');
var boomcatch = require('boomcatch');

boomcatch.listen({
	host: 'rum.example.com',                  // Defaults to '0.0.0.0' (INADDR_ANY)
	port: 8080,                               // Defaults to 80 for HTTP or 443 for HTTPS
	https: true,                              // Defaults to false
	httpsKey: 'foo.key',
	httpsCert: 'bar.cert',
	httpsPass: 'baz',
	path: '/perf',                            // Defaults to '/beacon'
	referer: /^\w+\.example\.com$/,           // Defaults to /.*/
	origin: [                                 // Defaults to '*'
	'http://foo.example.com',
	'http://bar.example.com'
	],
	limit: 100,                               // Defaults to 0
	maxSize: 1048576,                         // Defaults to -1
	log: console,                             // Defaults to object with `info`, `warn` and `error` log functions.
	workers: require('os').cpus().length,     // Defaults to 0
	validator: path.resolve('./myvalidator'), // Defaults to 'permissive'
	mapper: path.resolve('./mymapper'),       // Defaults to 'statsd'
	prefix: 'mystats.rum.',                   // Defaults to ''
	forwarder: 'http',                        // Defaults to 'udp'
	fwdUrl: 'https://stats.example.com/',     // No default
	fwdMethod: 'POST'                         // Defaults to 'GET'
});

Extensions

Boomcatch implements four extension points to control how beacon requests are handled: validators, filters, mappers and forwarders. These extension points are invoked as a pipeline when a beacon request is received.

boomcatch extensions' diagram

Several extensions are shipped as part of Boomcatch, or you can implement your own custom extensions.

You can read more details in the documentation about extensions.

Development

Before submitting any pull requests, please ensure that you have adhered to the contribution guidelines.

To clone the repository:

git clone [email protected]:nature/boomcatch.git

To set up the development environment:

npm install

To lint the code:

npm run lint

To run the unit tests:

npm test

Change log

History

Support

You can have a look at the Springer Nature Frontend Playbook for an explanation of how we support our open source projects.

License

GPL 3

Copyright © 2014, 2015, 2016 Springer Nature

boomcatch's People

Contributors

alexkilgour avatar dotcode avatar emav avatar hollsk avatar jlprat avatar josebolos avatar marcbachmann avatar philbooth avatar rowanmanning avatar tollmanz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

boomcatch's Issues

Unhandled error in statsd mapper

TypeError: Cannot read property 'type' of undefined
    at /usr/lib/node_modules/boomcatch/src/mappers/statsd.js:73:21
    at Array.map (native)
    at mapRestimingMetrics (/usr/lib/node_modules/boomcatch/src/mappers/statsd.js:69:17)
    at /usr/lib/node_modules/boomcatch/src/mappers/statsd.js:61:23
    at Array.forEach (native)
    at map (/usr/lib/node_modules/boomcatch/src/mappers/statsd.js:51:23)
    at send (/usr/lib/node_modules/boomcatch/src/index.js:656:22)
    at IncomingMessage.emit (events.js:92:17)
    at _stream_readable.js:943:16
    at process._tickCallback (node.js:419:13)

Improve the documentation

The readme is pretty sparse for anyone looking for serious detail. We need to add proper docs for each of the mappers & forwarders, how all of the different command line options interact and the reasoning behind them and probably a bunch of other stuff too.

unable to get the beacon results in boomcatch

Hi Team @tek @pixeltrix @robd @isaiah

Suddenly Am not able to get the beacon results in boomcatch error like " POST https://boomcatch-demo-indhusiva.c9users.io/beacon 503 (Service Unavailable)" .
But seems Boomcatch is running fine .and i have defined my beacon explicitly using BOOMR.init() method.
I have ran the boomcatch in forerver mode ,using below mentioned command
sudo forever -sa
-l /var/log/boomcatch/boomcatch-forever.log
-o /var/log/boomcatch/boomcatch-access.log
-e /var/log/boomcatch/boomcatch-error.log
/home/ubuntu/.nvm/versions/node/v4.7.3/bin/boomcatch
--port 8888
--host 127.0.0.1
--fwdPort 8089
> /dev/null 2>&1 &

Could you please help me to fix this issue!

Thanks
Siva.P

auto_xhr causes errors

When auto_xhr plugin is activated, there is an error while parsing the request because most of the RT vars are not available or set to 0.
Especially
t_resp=0
t_page=0
t_done=0

boomcatch logs

2016-06-03 15:06:06 INFO boomcatch: referer= user-agent=Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2756.0 Safar
i/537.36 address=127.0.0.1[] method=GET url=/beacon?rt.start=manual&v=1464957067&vis.st=visible&ua.plt=Win32&ua.vnd=Google+Inc.&dom.doms=1&mem.total=12700000&me
m.used=10000000&scr.xy=1536x864&scr.bpp=24%2F24&scr.orn=0%2Flandscape-primary&scr.dpx=2.5&cpu.cnc=4&scr.mtp=10&bat.lvl=1&dom.ln=15&dom.sz=1719&dom.img=0&dom.scr
ipt=3&dom.script.ext=1&pgu=http%3A%2F%2Flocalhost%3A8080%2Findex.html&restiming=%7B%7D&rt.tstart=1464959165911&rt.bstart=1464959126858&rt.end=1464959165911&t_re
sp=0&t_page=0&t_done=0&r=http%3A%2F%2Flocalhost%3A8080%2Findex.html&r2=&u=http%3A%2F%2Flocalhost%3A8080%2Fbetclic-logo-2.png&http.initiator=xhr&nt_fet_st=146495
9128413.81&nt_dns_st=1464959128413.81&nt_dns_end=1464959128413.81&nt_con_st=1464959128413.81&nt_con_end=1464959128413.81&nt_req_st=1464959128413.81&nt_res_st=14
64959128413.81&nt_res_end=1464959128413.81&nt_domint=1464959165911&nt_load_st=1464959165911&nt_load_end=1464959165911&dom.res=2

2016-06-03 15:06:06 ERROR boomcatch: 400 Invalid data
2016-06-03 15:06:06 ERROR boomcatch: Error: Invalid number
at assertImpl (C:__git\boomcatch\node_modules\check-types\src\check-types.js:643:19)
at assertPredicate (C:__git\boomcatch\node_modules\check-types\src\check-types.js:638:9)
at Object.positive (C:__git\boomcatch\node_modules\check-types\src\check-types.js:632:13)
at normaliseRtData (C:__git\boomcatch\src\normalise.js:47:24)
at normalise (C:__git\boomcatch\src\normalise.js:28:13)
at map (C:__git\boomcatch\src\mappers\statsd.js:49:12)
at send (C:__git\boomcatch\src\index.js:656:22)
at emitNone (events.js:80:13)
at IncomingMessage.emit (events.js:179:7)
at endReadableNT (_stream_readable.js:906:12)

MSIE 11 returns 400 Invalid data

Mostly all of these useragents returns 400 invalid data. Is it possible do more debug on the data?

2015-06-04 15:12:27 INFO boomcatch: referer=http://www.xxxxxxx.no user-agent=Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko address=127.0.0.1[84.x.x.x] method=POST url=/beacon
2015-06-04 15:12:27 ERROR boomcatch: 400 Invalid data
# boomcatch --version
2.2.1

Expose data type from mappers

Currently the file forwarder automatically sticks a .json extension on generated file names. Instead, we should expose the data type from the mappers (in the same way we expose separator strings), so that the file forwarder can choose a suitable extension. The same information could also be useful elsewhere, e.g. in the HTTP forwarder for setting a Content-Type request header.

Expose all the data

Currently we're only exposing a subset of the available data, based purely on what we happen to be graphing at Nature. We need to expose everything.

Error: events.js:72 \ throw er; // Unhandled 'error' event

I have installed Boomcatch 1.3.6 on Ubuntu 13.10 with Node.js 0.10.28. When trying to start the Boomcatch server (i.e., running boomcatch), I get the following:

2014-05-10 15:23:31 INFO boomcatch: listening for 0.0.0.0:80/beacon

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: listen EADDRINUSE
    at errnoException (net.js:904:11)
    at Server._listen2 (net.js:1042:14)
    at listen (net.js:1064:10)
    at net.js:1146:9
    at dns.js:72:18
    at process._tickCallback (node.js:419:13)
    at Function.Module.runMain (module.js:499:11)
    at startup (node.js:119:16)
    at node.js:906:3

Any thoughts?

Are we handling aborted loads sanely?

When I wrote the normalisation code, I hadn't considered the case of sessions that don't make it as far as the load event. It was pointed out at Velocity that boomerang also listens for beforeunload and sends a beacon request then, if it hasn't already been sent. Those beacon requests will obviously not contain load data and I think I may be unwittingly discarding them as a result. The data is pertinent and should be propagated to the extension pipeline.

Consider exposing metadata derived from user agent string and IP address

There are use cases for filtering/grouping data by browser or geographical location.

To some extent, this information can be derived from the user agent string and the connecting IP address, although it will often be inaccurate because of user agent spoofing, proxying and so on.

We need to consider whether we should expose this information to the data mappers (in particular, the HAR spec [1][2] has fields for this).

[1] http://www.softwareishard.com/blog/har-12-spec/
[2] #9

Missing callback in forwarder

As of 1.7.0, we're seeing boomcatch processes using the statsd-npg mapper fall over intermittently with the following error:

Dec  2 04:25:07 rum-data boomcatch[7243]: unhandled exception
TypeError: undefined is not a function
    at Object.cb (/usr/local/lib/node_modules/boomcatch/src/forwarders/udp.js:65:13)
    at Object.afterSend [as oncomplete] (dgram.js:322:9)

Empty mapped data should not cause 400 response

Currently, there is a condition that tests the result from the mapper and returns 400 if it is empty.

I thought that was a good idea at the time. It wasn't.

Instead those requests should just return early with a 204 response. It is entirely legitimate for a mapper to ignore data without failing the request.

Requests from the round-trip plugin are being incorrectly rejected

Jun  5 04:27:39 rum-data boomcatch[2357]: referer=http://test-www.nature.com/nmstr/about/aims address=127.0.0.1[31.221.90.4] method=GET url=/beacon?rt.start=none&rt.tstart=&rt.bstart=1401956859257&rt.end=1401956859818&v=0.9.1400768954&u=http%3A%2F%2Ftest-www.nature.com%2Fnmstr%2Fabout%2Faims
Jun  5 04:27:39 rum-data boomcatch[2357]: 400 Invalid data

missing restiming data in statsd

I understand this is not a boomerang issue --- but still wanted to post here to see if any help can be made available.
We are using boomcatch -- to capture the beacon data and then forward to statsd... we are seeing the navtiming and rt data in statsd but not the restiming....

Implement branching

Currently, users can only specify one instance of each extension point. However, it is reasonable to split the data so that, for instance, navtiming goes to statsd and restiming goes to waterfall charts stored on disk.

This could either be implemented as part of the filter step or in a dedicated new extension point.

Implement forwarders/file

This could be useful for the HAR mapper and also if someone just wants to write to the disk as a quick and dirty place to keep data.

Should support options to control one file vs many files, output path, naming scheme for the many files case (allowing directory creation as part of the name).

Support an `ignore` parameter on beacon requests

In the nagios settings for our boomcatch instance, we have a check_http rule that regularly hits the beacon with valid data to ensure that everything is running okay. This means that our data is getting polluted by a made-up timing from Nagios.

Admittedly it is not a big deal in our case because we're graphing averages and the timing value is a typical one, but it would still be better if we had a way to let that request succeed but to ignore that data. I guess an ignore parameter would be the simplest way to do it.

Negative values for navtiming.load

From the logs:

Jul 27 14:01:15 rum-data boomcatch[1766]: referer=http://www.nature.com/articles/npjpcrm201425 address=127.0.0.1[86.142.245.46] method=GET url=/beacon?rt.start=none&rt.tstart=&rt.bstart=1406484073607&rt.end=1406484073625&rt.abld=&nt_red_cnt=0&nt_nav_type=0&nt_nav_st=1406484071419&nt_red_st=0&nt_red_end=0&nt_fet_st=1406484071419&nt_dns_st=1406484071419&nt_dns_end=1406484071419&nt_con_st=1406484071419&nt_con_end=1406484071419&nt_req_st=1406484072228&nt_res_st=1406484072228&nt_res_end=1406484073371&nt_domloading=1406484072228&nt_domint=1406484073398&nt_domcontloaded_st=1406484073398&nt_domcontloaded_end=1406484073415&nt_domcomp=0&nt_load_st=0&nt_load_end=0&nt_unload_st=1406484072231&nt_unload_end=1406484072231&nt_first_paint=1406484072428&v=0.9.1400768954&u=http%3A%2F%2Fwww.nature.com%2Farticles%2Fnpjpcrm201425
Jul 27 14:01:15 rum-data boomcatch[1766]: sending rum.rt.load:18|ms#012rum.navtiming.dns:0|ms#012rum.navtiming.firstbyte:809|ms#012rum.navtiming.domload:1979|ms#012rum.navtiming.load:-1406484071419|ms
Jul 27 14:01:15 rum-data boomcatch[1766]: sent 139 bytes

The value for the load event is -1,406,484,071,419 milliseconds. Needs fixing.

Make unmapped wrapper the default

Currently the default wrapper is statsd, which contains a normalisation step. This makes things kind of confusing for people who just use the default mapper, who wonder why their data has vanished on the way through Boomcatch.

#58 was created to remove the normalisation from the existing mappers, but not from the default package - it would probably help to change the default mapper to unmapped instead, so users get a JSON serialisation of the Boomerang payload. This will give us some breathing space to rewrite the rest of the mappers ( #79 ).

Consider how to filter data before forwarding

Now that I am exposing much more of the data from boomerang, there is a requirement to filter which parts of the data get passed to the forwarders so that users are only storing the data they need in their back-end of choice.

The simplest way to do this (from our point of view) would just be to say that people need to create a custom data mapper if they want to filter out unwanted data. Obviously that places a greater burden on the user.

Another way would be to add an extension point, similar to validators and mappers, which is called as part of the processing pipeline before data is passed to the forwarder.

A third option would be to provide an option that allows filtration by regex or some other pattern matching, perhaps.

The extension point feels like it's probably the right way to do it at the moment but I'm going to mull it over for a bit.

Unhandled exception

I am getting a write ENOTSUP unhandled exception with the following request:

C:\Users\Administrator>boomcatch --port 8090 --host 127.0.0.1
2016-03-16 13:25:54 INFO boomcatch: starting boomcatch in process 5928 with opti
ons:
{
"port": 8090,
"host": "127.0.0.1",
"log": {},
"workers": 1
}
2016-03-16 13:25:54 INFO boomcatch: worker 3868 started
2016-03-16 13:25:54 INFO boomcatch: listening for 127.0.0.1:8090/beacon
2016-03-16 13:26:07 INFO boomcatch: referer=http://localhost:8088/index.htm user
-agent=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0
address=127.0.0.1[] method=GET url=/beacon?rt.start=navigation&rt.tstart=1458134
761909&rt.cstart=1458134762034&rt.bstart=1458134766092&rt.end=1458134767197&t_re
sp=1619&t_page=3669&t_done=5288&t_other=t_domloaded%7C5276&r=http%3A%2F%2Flocalh
ost%3A8088%2Findex.htm&nt_red_cnt=0&nt_nav_type=1&nt_nav_st=1458134761909&nt_red
_st=0&nt_red_end=0&nt_fet_st=1458134762267&nt_dns_st=1458134762267&nt_dns_end=14
58134762267&nt_con_st=1458134762267&nt_con_end=1458134762267&nt_req_st=145813476
2474&nt_res_st=1458134763528&nt_res_end=1458134763531&nt_domloading=145813476356
3&nt_domint=1458134767033&nt_domcontloaded_st=1458134767092&nt_domcontloaded_end
=1458134767185&nt_domcomp=1458134767189&nt_load_st=1458134767189&nt_load_end=145
8134767194&nt_unload_st=1458134763565&nt_unload_end=1458134763697&u=http%3A%2F%2
Flocalhost%3A8088%2Findex.htm&v=0.9&vis.st=visible&ua.plt=Win32&ua.vnd=
2016-03-16 13:26:07 INFO boomcatch: sending rt.firstbyte:1619|ms
rt.lastbyte:5288|ms
rt.load:5288|ms
navtiming.unload:132|ms
navtiming.response:3|ms
navtiming.dom:3626|ms
navtiming.domContent:93|ms
navtiming.load:5|ms

2016-03-16 13:26:07 ERROR boomcatch: unhandled exception
Error: write ENOTSUP
at exports._errnoException (util.js:746:11)
at ChildProcess.target._send (child_process.js:484:28)
at ChildProcess.target.send (child_process.js:416:12)
at sendHelper (cluster.js:676:8)
at send (cluster.js:512:5)
at cluster.js:488:7
at SharedHandle.add (cluster.js:99:3)
at queryServer (cluster.js:480:12)
at Worker.onmessage (cluster.js:438:7)
at ChildProcess. (cluster.js:692:8)

My suspicion is that there must have been an issue with my installation of boomcatch. Please advise.

Remove normalisation from remaining mappers

As discussed in #55 and #58. Data is normalised in three mappers (statsd, statsd-npg, waterfall), which removes a bunch data from each payload. statsd is the default, so most users are probably going to use that first - unless they can work out from the documentation what's going on, they're going to be confused.

We should remove the normalisation step from the remaining mappers, probably prioritising statsd.

Propagate additional request data

Arbitrary values can be added to the beacon request with BOOMR.addVar() for all kinds of reasons. We need to propagate these through the extension pipeline.

Millennium bug

Currently boomcatch rely on qs library to parse beacon data in parseData function and by default qs will only parse up to 1000 parameters.

This is not usually an issue in production but on dev you may have CSS and JS not combined and with the restiming.js plugin you may hit the above limit.

Raising the limit to 10K should solve the issue. See /pull/54

Inconsistent data from Boomerang to Boomcatch

I'd added 2 console.log at the send event ( see Logs here ):

image

As you can see, at the first log ( before data "turns" into mappedData ) there are infos about restiming, mem, dom, cpu and etc... but at the second log, I just have infos about rt and navtiming.

Am I missing something?

Thanks.

Run as a daemon

Do you have a recommended way to run Boomcatch as a daemon? If you happen to have an init.d script, that would be even better.

Thanks!

Expose all the data

Currently we're only exposing a subset of the available data, based purely on what we happen to be graphing at Nature. We need to expose everything.

Ampersands in restiming URLs

Hi, I hope this isn't a rather stupid thing to bring up, but I've been battling with an issue where the site I'm working on would request a url with ampersands in breaking the JSON parsing in boomcatch.

So requesting something like: https://use.typekit.net/af/4713c5/000000000000000000017690/27/l?primer=09b6306563171828e867a80b0e487a34c56b49b8efde4541b63ffd44535e9a56&fvd=n4&v=3

would make boomerang send a timing object to boomcatch like this (looks good): "use.typekit.net/af/4713c5/000000000000000000017690/27/l?primer=09b6306563171828e867a80b0e487a34c56b49b8efde4541b63ffd44535e9a56&fvd=n4&v=3":"3p,2g,28,13,13,b,a,a,2111fw,8720"

However on the boomcatch sode of things it breaks. By the time it gets to the mapper it has been mangled and lookes like:
{"use.typekit.net/af/4713c5/000000000000000000017690/27/l?primer=09b6306563171828e867a80b0e487a34c56b49b8efde4541b63ffd44535e9a56',
fvd: 'n4',
v:
[ '3":"3p" .... etc

After a bit of digging I discovered this was occurring in index.js between lines 693 and 699

   state.body = decodeURIComponent(state.body);

    if (request.headers['content-type'] === 'text/plain') {
        return JSON.parse(state.body);
    }

    return qs.parse(state.body, { parameterLimit: Infinity });

If decoded data is sent to qs.parse it breaks out the ampersand of a nested querystring and completely breaks the JSON at the point where the ampersand would occur in the restiming data. The only way I got it to work is if I moved the decodeURIComponent function into the if statement:

    if (request.headers['content-type'] === 'text/plain') {
        state.body = decodeURIComponent(state.body);
        return JSON.parse(state.body);
    }

    return qs.parse(state.body, { parameterLimit: Infinity });

I can't say if this is the ideal solution, it could break other parts of the application but it seems to work for me at the moment.

Why is the whole body being treated as if its a query string and being sent through qs.parse?

Is there a better way to solve this?

Thanks.

Implement forwarders/console

This could potentially be useful for debugging or if someone just wants to quickly make sure things are running okay by dumping the result of the extension pipeline out to the console.

Conditional extension invocation

Currently a single boomcatch instance executes with a single extension pipeline configuration. However it is unlikely that anyone using multiple boomerang plugins wants to handle all of the data in the same way.

For example, you might want to map navigation timing data as statsd timers and resource timing data as a waterfall chart.

This would require a way to conditionally specify multiple extension configurations.

Doing that on the command line would likely be a bit of a mess, trying to define some arbitrary DSL that handles all use cases perfectly. A better approach could be to expose a meta-extension, a JavaScript extension point that lets you invoke the other extensions however you like.

Incorrect log value for `boomcatch.listen` in README

Using the following as shown in the README.md#L211:

log: console.log,                         // Defaults to function () {}

Causes this error:

PATH_TO_PROJECT\node_modules\boomcatch\node_modules\check-types\src\check-types.js:643
            throw new Error(message || 'Assertion failed');
                  ^
Error: Invalid log object
    at assertImpl (PATH_TO_PROJECT\node_modules\boomcatch\node_modules\check-types\src\check-types.js:643:19)
    at assertPredicate (PATH_TO_PROJECT\node_modules\boomcatch\node_modules\check-types\src\check-types.js:638:9)
    at Object.object (PATH_TO_PROJECT\node_modules\boomcatch\node_modules\check-types\src\check-types.js:632:13)
    at verifyLog (PATH_TO_PROJECT\node_modules\boomcatch\src\index.js:191:24)
    at verifyOptions (PATH_TO_PROJECT\node_modules\boomcatch\src\index.js:169:5)
    at Object.exports.listen (PATH_TO_PROJECT\node_modules\boomcatch\src\index.js:130:9)
    at Object.<anonymous> (PATH_TO_PROJECT\index.js:11:11)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)

Fix is to change the line to:

log: console,                               // Defaults to function () {}

to reflect the jsdoc:

 * @option log {object}          Object with `info` and `error` log functions.

Process thrashing when respawning as a service

Intermittently, our boomcatch server gets into a tight loop of respawning and dying with EADDRINUSE, because another boomcatch process has already bound to the listen address. This problem seems to have become more common since the release of 1.8.0.

Although I think part of the problem is to do with our service configuration, it is also true that boomcatch itself should respawn more intelligently. If EADDRINUSE is the cause, then it should pause before trying to respawn. Additionally, if n attempts fail, it should give up and fail permanently.

Normalisation of round-trip data sometimes produces wildly inaccurate results

eg:

Jun 10 09:43:26 rum-data boomcatch[8416]: referer=http://www.nature.com/articles/sdata20147 address=127.0.0.1[31.221.90.4] method=GET url=/beacon?rt.start=navigation&nt_red_cnt=0&nt_nav_type=0&nt_nav_st=1402393979731&nt_red_st=0&nt_red_end=0&nt_fet_st=1402393979731&nt_dns_st=1402393979731&nt_dns_end=1402393979731&nt_con_st=1402393979731&nt_con_end=1402393979731&nt_req_st=1402393979747&nt_res_st=1402393980426&nt_res_end=1402393980480&nt_domloading=1402393980436&nt_domint=1402393980739&nt_domcontloaded_st=1402393980769&nt_domcontloaded_end=1402393980868&nt_domcomp=1402393981265&nt_load_st=1402393981266&nt_load_end=1402393981368&nt_unload_st=0&nt_unload_end=0&nt_spdy=0&nt_first_paint=1402393980.875492&v=0.9.1400768954&u=http%3A%2F%2Fwww.nature.com%2Farticles%2Fsdata20147&rt.tstart=1402393979731&rt.bstart=1402393980956&rt.end=1402407806377&t_done=13826646&r=http%3A%2F%2Fwww.nature.com%2Farticles%2Fsdata20147&r2=
Jun 10 09:43:26 rum-data boomcatch[8416]: sending rum.rt.load:13826646|ms#012rum.navtiming.dns:0|ms#012rum.navtiming.firstbyte:695|ms#012rum.navtiming.domload:1038|ms#012rum.navtiming.load:1535|ms
Jun 10 09:43:26 rum-data boomcatch[8416]: sent 135 bytes

Remove the data normalisation process

My original thinking behind normalising the data was to make writing extensions easier; the boomerang format is necessarily terse to reduce the size of its payload so I wanted to pass something through that was easier to work with.

However, I now think that was a mistake for the following reasons:

  1. It adds little value because nobody is writing extensions.
  2. It separates extensions from the boomerang data format, which is documented and familiar to people.
  3. It only passes through data from the round trip, navigation timing and resource timing plugins.

Reason number 3 is the really big one. Normalisation also places an extra maintenance burden on this project as it must ensure it matches up to any changes made to boomerang.

Making the normalisation step an extension point can't be done without affecting the reusability of the other extensions, since they expect a uniform data format.

Hence the only sane thing to do, I think, is delete normalisation entirely. Sadly it is a bigger job than simple deletion, because all of the extensions must be updated to use the boomerang data format.

Sanitise the validator extension point

There are two issues with validators: when they are called and the arguments they are passed.

They are called after data is normalised, which only delays the possible failure. Instead they should be the first action after preliminary checks on the request.

They are passed the normalised data, when instead they should be passed either the naked request object, or some subset thereof. The main use for this extension point is nonce validation, which obviously requires access to the query string. Conceivably there could also be a requirement to check other aspects of the request, such as userAgent string or remote IP address (for geolocation).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.