tulios / burrow-stats Goto Github PK

Dashboard for burrow kafka consumer lag checking

JavaScript 83.14% CSS 14.79% HTML 2.07%

burrow-stats's Introduction

Burrow Stats

Dashboard for burrow kafka consumer lag checking

Running locally

yarn install

npm run dev

Remember to create the config file (configs.json)

Production

NODE_ENV=production npm run build
NODE_ENV=production npm start

Docker

docker run \
  -p 8022:8022 \
  -e PORT=8022 \
  -v /path/to/your/configs.json:/opt/burrow-stats/configs.json \
  tulios/burrow-stats:latest

Configs

Burrow-stats requires a configuration file called configs.json. Take a look at configs.json.example for a complete example.

Consumers

{
  "consumers": [
    {
      "name": "consumer_name_1",
      "status": "http://burrow-host:8000/v2/kafka/cluster_name/consumer/consumer_name_1/status",
      "consumer_group_offset": "http://burrow-host:8000/v2/kafka/cluster_name/consumer/consumer_name_1/topic/topic_name_1",
      "topic_offset": "http://burrow-host:8000/v2/kafka/cluster_name/topic/topic_name_1"
    }
  ]
}

Accepts configurations for your consumers, if needed can aggregate data from different burrow instances.

Chart Options

{
  "chartOptions": {
    "scaleOverride": false,
    "scaleSteps": null,
    "scaleStepWidth": null,
    "scaleStartValue": null,
    "scaleFontSize": 14
  }
}

Allows you to change some behavior of the chart.

scaleOverride : Enable scale override with a hard coded values. Default false
scaleSteps : The number of steps in a hard coded scale (Requires scaleOverride true)
scaleStepWidth : The value jump in the hard coded scale (Requires scaleOverride true)
scaleStartValue : The scale starting value (Requires scaleOverride true)
scaleFontSize : Scale label font size in pixels. Default 14

Example using hard coded values for scale. The Y axis will start with 0 and it will contain 20 steps of 10. All scale parameters are connected, you must provide all of them when enabling scaleOverride.

{
  "scaleOverride": true,
  "scaleSteps": 20,
  "scaleStepWidth": 10,
  "scaleStartValue": 0
}

Burrow Stats Options

{
  "burrowStatsOptions": {
    "pollInterval": 60,
    "cacheBinSize": 30,
    "envLabel": "Staging",
    "envLabelColor": "white",
    "envLabelBgColor": "rgba(0, 137, 207, 1)"
  }
}

Configure burrow-stats internals.

pollInterval : Frequency of data update in seconds. Default 60
cacheBinSize : Number of data points stored. Default 30
envLabel : Optional environment label. Default null

envLabelColor: Default white
envLabelBgColor: Default rgba(0, 137, 207, 1)

burrow-stats's People

Contributors

Stargazers

Watchers

Forkers

datafibers chandra-techpassionate libin chornyi dezinsektor barockok trailrunner-taulia intellifylearning doodlescheduling maspling velan6 gwake1 trashguy yiliaofan iq-scm

burrow-stats's Issues

Can you tell me how to Install and configure this with burrow.

Support reading data from kastlex instead of burrow

We've had a situation where Burrow reported stale offsets for the topic, while the consumer groups were still updating, leading to "negative lag". In burrow-stats, this showed as a persistent flatline, since it truncates negative lag to zero.

Meanwhile, we had more reliable lag information in KastleX. It would be worthwhile for us to change the backend of burrow-stats to use KastleX, and we could contribute this upstream, but I'm opening up an issue for discussion first. Obviously, the project name is tied to Burrow as it stands, so maybe this change would be out of scope for the project.

Cannot find module '../../configs.json'

Hi, I got following error:

[root@srv1 burrow-stats]# docker run -p 8022:8022 -e PORT=8022 -v /opt/burrow-stats/configs.json tulios/burrow-stats:latest
Error: Error: Cannot find module '../../configs.json'

configs.json file is just on the same path.

Any idea?

And here is my node version:
[root@srv1 burrow-stats]# npm -v
5.6.0
[root@srv1 burrow-stats]# node -v
v8.11.3

Error while running "npm run dev"

I see below error while running "npn run dev"

$ node -v
v7.0.0
npm -v
3.10.8

Starting child process with 'node bin/server.js'
/Users/tanuj/Documents/burrow-stats/node_modules/babel-core/lib/transformation/file/logger.js:41
throw new Constructor(this._buildMessage(msg));
^

ReferenceError: [BABEL] /Users/tanuj/Documents/burrow-stats/node_modules/burrow-stats/server/index.js: Unknown option: base.Children. Check out http://babeljs.io/docs/usage/options/ for more information about options.

A common cause of this error is the presence of a configuration options object without the corresponding preset name. Example:

Invalid:
{ presets: [{option: value}] }
Valid:
{ presets: [['presetName', {option: value}]] }

For more detailed information on preset configuration, please see http://babeljs.io/docs/plugins/#pluginpresets-options.
at Logger.error (/Users/tanuj/Documents/burrow-stats/node_modules/babel-core/lib/transformation/file/logger.js:41:11)
at OptionManager.mergeOptions (/Users/tanuj/Documents/burrow-stats/node_modules/babel-core/lib/transformation/file/options/option-manager.js:221:20)
at OptionManager.init (/Users/tanuj/Documents/burrow-stats/node_modules/babel-core/lib/transformation/file/options/option-manager.js:374:12)
at File.initOptions (/Users/tanuj/Documents/burrow-stats/node_modules/babel-core/lib/transformation/file/index.js:216:65)
at new File (/Users/tanuj/Documents/burrow-stats/node_modules/babel-core/lib/transformation/file/index.js:139:24)
at Pipeline.transform (/Users/tanuj/Documents/burrow-stats/node_modules/babel-core/lib/transformation/pipeline.js:46:16)
at Object.transformFileSync (/Users/tanuj/Documents/burrow-stats/node_modules/babel-core/lib/api/node.js:138:10)
at compile (/Users/tanuj/Documents/burrow-stats/node_modules/babel-register/lib/node.js:118:20)
at loader (/Users/tanuj/Documents/burrow-stats/node_modules/babel-register/lib/node.js:144:14)
at Object.require.extensions.(anonymous function) [as .js] (/Users/tanuj/Documents/burrow-stats/node_modules/babel-register/lib/node.js:154:7)
Program node bin/server.js exited with code 1

Consumer Lag and Partitons Lag do not display unless all consumers are running

Unless all specified consumers are running, the Consumers Lag and Partitions Lag view are not displaying anything? Status shows consumer groups that are currently active but unless they are all active/available nothing displays in the other two views.

configs.json:


{
  "consumers": [
{ "name": "splash_csv_console_listener_1", "status": "http://localhost:8000/v2/kafka/local/consumer/splash_csv_console_listener_1/status", "consumer_group_offset": "http://localhost:8000/v2/kafka/local/consumer/splash_csv_console_listener_1/topic/splash_csv", "topic_offset": "http://localhost:8000/v2/kafka/local/topic/splash_csv" }
,
{ "name": "splash_json_console_listener_1", "status": "http://localhost:8000/v2/kafka/local/consumer/splash_json_console_listener_1/status", "consumer_group_offset": "http://localhost:8000/v2/kafka/local/consumer/splash_json_console_listener_1/topic/splash_json", "topic_offset": "http://localhost:8000/v2/kafka/local/topic/splash_json" }
,
{ "name": "splash_csv_json_converter", "status": "http://localhost:8000/v2/kafka/local/consumer/splash_csv_json_converter/status", "consumer_group_offset": "http://localhost:8000/v2/kafka/local/consumer/splash_csv_json_converter/topic/splash_csv", "topic_offset": "http://localhost:8000/v2/kafka/local/topic/splash_csv" }
,
{ "name": "splash_json_couchbase_loader", "status": "http://localhost:8000/v2/kafka/local/consumer/splash_json_couchbase_loader/status", "consumer_group_offset": "http://localhost:8000/v2/kafka/local/consumer/splash_json_couchbase_loader/topic/splash_json", "topic_offset": "http://localhost:8000/v2/kafka/local/topic/splash_json" }
  ],
  "chartOptions": {
    "scaleOverride": false,
    "scaleFontSize": 14,
    "scaleSteps": null,
    "scaleStepWidth": null,
    "scaleStartValue": null
  },
  "burrowStatsOptions": {
    "pollInterval": 60,
    "cacheBinSize": 60
  }
}

    {
      fillColor: 'rgba(0, 137, 207, 0.3)',
      strokeColor: 'rgba(0, 137, 207, 1)',
      pointColor: 'transparent',
      pointStrokeColor: 'transparent',
      pointHighlightFill: 'transparent',
      pointHighlightStroke: 'transparent',
    },

Cannot read property 'series' of undefined

while running latest docker image (0.6.1), instantly getting this error:

TypeError: Cannot read property 'series' of undefined
    at t.chartData (app-de86d3a….js:8)
    at t.render (app-de86d3a….js:8)
    at c._renderValidatedComponentWithoutOwnerOrContext (common-de86d3a….js:17)
    at c._renderValidatedComponent (common-de86d3a….js:17)
    at c.performInitialMount (common-de86d3a….js:17)
    at c.mountComponent (common-de86d3a….js:17)
    at Object.mountComponent (common-de86d3a….js:2)
    at Object.updateChildren (common-de86d3a….js:17)
    at _reconcilerUpdateChildren (common-de86d3a….js:17)
    at _updateChildren (common-de86d3a….js:17)

here is my config:

{
  "consumers": [
    {
      "name": "consumer_name_1",
      "status": "https://burrow.defined.tld/v2/kafka/local/consumer/cg/status",
      "consumer_group_offset": "https://burrow.defined.tld/v2/kafka/local/consumer/cg/topic/topic-QA",
      "topic_offset": "https://burrow.defined.tld/v2/kafka/local/topic/topic-QA"
    }
  ],
  "chartOptions": {
    "scaleOverride": false,
    "scaleFontSize": 14,
    "scaleSteps": null,
    "scaleStepWidth": null,
    "scaleStartValue": null
  },
  "burrowStatsOptions": {
    "pollInterval": 10,
    "cacheBinSize": 30,
    "envLabel": "QA"
  }
}

How to increase X-axis range

Is there any way to extend time of X-Axis to keep 2 days data at least?