Giter VIP home page Giter VIP logo

github-monitoring's Introduction

Build Status

A Docker Stack which Monitors your GitHub Repos

Here's a quick start to stand-up a Docker Prometheus stack containing Prometheus, Grafana and github-exporter to collect and graph GitHub statistics.

Pre-requisites

Before we get started installing the Prometheus stack. Ensure you install the latest version of docker and docker-compose on your Docker host machine. This has also been tested with Docker for Mac and it works well.

Installation

Clone the project to your Docker host.

If you would like to change which targets should be monitored or make configuration changes edit the /prometheus/prometheus.yml file. The targets section is where you define what should be monitored by Prometheus. The names defined in this file are actually sourced from the service name in the docker-compose file. If you wish to change names of the services you can add the "container_name" parameter in the docker-compose.yml file.

Configuration

In order to pull GitHub stats consistently it is recommended you create a personal access token inside of GitHub. This token will allow you to query the GitHub API more frequently than a public user. Create GitHub Token. It is only necessary to give the repo scope to the token permission.

Copy the GitHub Token you created and paste into the bottom of the docker-compose.yml file under the metrics service section replacing the GITHUB_TOKEN with your newly created token.

The REPOS variable can also be updated to point to the Repos that you wish to monitor. In my example I monitor freeCodeCamp and Docker.

 metrics:
  tty: true
  stdin_open: true
  expose:
    - 9171
  image: infinityworks/github-exporter:latest 
  environment:
    - REPOS=freeCodeCamp/freeCodeCamp, docker/docker
    - GITHUB_TOKEN=<GitHub API Token see README>
  networks:
    - back-tier

Once configurations are done let's start it up. From the /prometheus project directory run the following command:

$ docker-compose up -d

That's it. docker-compose builds the entire Grafana and Prometheus stack automagically.

The Grafana Dashboard is now accessible via: http://<Host IP Address>:3000 for example http://192.168.10.1:3000

username - admin password - foobar (Password is stored in the config.monitoring env file)

The DataSource and Dashboard for Grafana are automatically provisioned. You can still install the dashboard manually if you choose below.

Manual Install Dashboard

I created a Dashboard template which is available on GitHub Stats Dashboard. Simply download the dashboard and select from the Grafana menu -> Dashboards -> Import

This dashboard is intended to help you get started with graphing your GitHub Repos. If you have any changes you would like to see in the Dashboard let me know so I can update Grafana site as well.

Troubleshooting

It appears some people have reported no data appearing in Grafana. If this is happening to you be sure to check the time range being queried within Grafana to ensure it is using Today's date with current time.

github-monitoring's People

Contributors

katopz avatar shark-h avatar vegasbrianc avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

github-monitoring's Issues

Speedtest for global locations, and ICMP probe

Hello,

Great tool, thanks for packaging it up.

I am wondering if you can help with two issues that I am experiencing:

  1. Speedtest consistently shows low speed and high latency. Is it possible that this is using a US-based node and if yes it is possible for me to configure the node manually?
  2. Could you indicate how an ICMP probe may be added? This is a workaround to a problem that I am facing; I want to be able to capture data points which show a failed HTTP probe, but I don't seem to be able to do this in Prometheus as rather than an event existing with a failure code no event exists. I am considering that perhaps using an ICMP probe would show a failure event.

Thanks in advance.

No Data

Hello! Thank you for putting this together. It's my first foray into Prometheus and Grafana, and it's been fairly painless.

I am stuck on one matter. I have the Grafana dashboard up and running on localhost:3000 and I see metrics are being successfully collected in my terminal and on localhost:9171. But this does not reflect in the dashboard.

The queries are working, as I get this when I Inspect:
{ "request": { "url": "http://localhost:9090/api/v1/query_range?query=github_repo_stars%7Brepo%3D%22gravitational%09eleport%22%7D&start=1562803200&end=1622851200&step=86400", "method": "GET", "hideFromInspector": false }, "response": { "status": "success", "data": { "resultType": "matrix", "result": [] } } }

It looks like the request is successful, but it's not returning any data? Any guidance?

Support for whole organisation

Hi @vegasbrianc, thanks for the great tutorial!

It is working like a charm. I'm one of the maintainer of https://github.com/voxpupuli. Displaying a single repo, or a few, is pretty easy with your template. I would like to see some accumulated value. Like open Issues/PR/Stars for the whole org. It is probably easy to add up all values for an organisation?

Also do you think it is possible to get all the graphs you have right now generated for each repository? Maybe one dashboard for each? Creating them by hand takes a huge amount of time since we have more than 100 repos.

Add "watch" stats

It'd be awesome if we can add a "watch" stats as well to the dashboard.
Stars, issues and forks are working flawlessly.
Is there a way to get the number of people watching the repo as well?

prometheus container won't start

The prometheus container fails to start:

prometheus | time="2017-02-24T12:39:30Z" level=info msg="Starting prometheus (version=1.5.2, branch=master, revision=bd1182d29f462c39544f94cc822830e1c64cf55b)" source="main.go:75"
prometheus | time="2017-02-24T12:39:30Z" level=info msg="Build context (go=go1.7.5, user=root@1a01c5f68840, date=20170210-16:23:28)" source="main.go:76"
prometheus | time="2017-02-24T12:39:30Z" level=info msg="Loading configuration file /etc/prometheus/prometheus.yml" source="main.go:248"
prometheus | time="2017-02-24T12:39:30Z" level=error msg="Error loading config: couldn't load configuration (-config.file=/etc/prometheus/prometheus.yml): open /etc/prometheus/prometheus.yml: no such file or directory" source="main.go:150"
prometheus exited with code 1

Unable to add custom repo details

After editing the docker-compose.yml file with the repo details

 environment:
      - REPOS=kubevirt/kubevirt

And importing the template, Im not seeing any stats and also the older REPO(docker/docker, freeCodeCamp/freeCodeCamp) exists.

Add a License

Hi, I noticed that your repository doesn't have a license, yet many people are forking it and contributing to it, this is actually a pretty big issue.

Why are licenses important

Whenever you publish essentially anything, in this case some code or static content created by you, you automatically gain copyright over that content. This copyright is respected in most developed countries and it imposes big legal restrictions on the content in this repository. Essentially, this means that while you decided to make your software source-available, it is not "open-sourced", because it doesn't have an open source license. This means that while you as the copyright owner allowed others to look at your source, you did not give legal permission to re-use this code or to alter it in any way.

This means that everybody who has forked your repository is actually legally breaking the copyright law. You should really consider adding an open sourced (ideally an FSF approved) license. Doing this will not affect your copyright over the code, it just gives everyone some additional permissions that wouldn't be there without it, a license like this is the only thing that allows others to make derivative projects (projects based on your code), or utilize parts of your code in other places.

License types

Learning about different licenses may be a bit confusing at first, and i would suggest you do your own research. I will describe some basic categories of these licenses and what they mean, but this is NOT a legal advice and again, you should always to your own research. Don't just blindly trust some random guy on the internet.

Copy-Left licenses

These licenses allow others to use your code and built upon it, but they require the use of the same license. This essentially enforces the code to always stay open-sourced as it explicitly forbids sublicensing under any non-copatible license (i.e. a license with different terms).

Most notably, this include the GPL licenses.

Permissive licenses

Permissive licenses give your contributors a lot more freedom because they allow sublicensing. They usually only require stating the original source of the code. This means that if you would decide to use a permissive license, anyone would be able to make a close-sourced project based on your code and even sell it, as long as there is a mention that some parts of this code-base were utilized in it. (If that license requires it)

Most notably, this includes the MIT license and the Apache 2.0 License, another popular set of licenses are the BSD licenses, most notably the BSD 2-Clause license.

Public domain licenses

There are also so called public domain licenses, these licenses are actually a subtype of permissive licenses, however when we talk about permissive licenses, we generally don't really refer to the public domain ones, which is why I separated this category as it is a bit special.

Public domain licenses essentially strip the copyright away and essentially it gives everybody rights to do pretty much anything with the code. They don't require any mentions of the original source for that code and obviously they allow sublicensing. These licenses dont actually "remove" your copyright over that code, they just allow anyome to do anything with the project, essentially disregarding the copyright alltogether.

Most notably, this includes the Unlicense license.

Picking your license

While you aren't required to add a license of any kind, it is a bit weird to have a popular project such as this with a lot of forks without any license. At the moment, all of the contributors are actually breaking the copyright law which really isn't ideal. Even though I'm sure you don't have the intention to sue these people, you technically could even though they just wanted to contribute to your project. That's a bit unexpected for the contributors and it could discourage many people from contributing to or even using this project.

Some great websites;

  • There is a page from GitHub that may help you pick the correct license for your program: https://choosealicense.com/.
  • You can also check out their post about licensing here.
  • Another wonderful site you should check out is https://tldrlegal.com, it can quickly show you the details about what license requires from others when they use your code, and what rights it gives them.

Limitations:
While you can license your code under any license whatsoever, if your code is utilizing some copy-left licensed parts (whether by a library-like dependency or directly by having bits of code written by others with that license), you will need to respect the terms of that license and license your code accordingly. This means that if you have a copy-left licensed dependency that you're utilizing in your project, you will need to adhere to it's terms or remove that dependency, otherwise you, as the owner of that code are yourself violating the copyright law by doing something prohibited by copyright and not explicitly allowed by the license of the software you depend on.

prometheus stack fails to launch (docker mac)

docker-compose up -d
d831b5055bfd        grafana/grafana                        "/run.sh"                8 seconds ago       Up 6 seconds               0.0.0.0:3000->3000/tcp   githubmonitoring_grafana_1
094980593e3c        prom/prometheus                        "/bin/prometheus -co…"   8 seconds ago       Exited (1) 7 seconds ago                            prometheus
95894bdb1843        infinityworks/github-exporter:latest   "/bin/main"              9 seconds ago       Up 8 seconds               0.0.0.0:9171->9171/tcp   githubmonitoring_metrics_1

Docker version 17.12.0-ce, build c97c6d6

logs from prometheus stack:

Error parsing commandline arguments: unknown short flag '-c'
prometheus: error: unknown short flag '-c'

Tried 2.2 and 3.3 received the same errors.

update

format changed from - to -- so updating to --config.file=/etc/prometheus/prometheus.yml worked but new errors with storage: --storage.tsdb.path=/prometheus (per docs throws new errors:

level=info ts=2018-01-13T00:44:59.127037413Z caller=main.go:215 msg="Starting Prometheus" version="(version=2.0.0, branch=HEAD, revision=0a74f98628a0463dddc90528220c94de5032d1a0)"
level=info ts=2018-01-13T00:44:59.127111605Z caller=main.go:216 build_context="(go=go1.9.2, user=root@615b82cb36b6, date=20171108-07:11:59)"
level=info ts=2018-01-13T00:44:59.127125741Z caller=main.go:217 host_details="(Linux 4.9.60-linuxkit-aufs #1 SMP Mon Nov 6 16:00:12 UTC 2017 x86_64 241d1e6236d7 (none))"
level=info ts=2018-01-13T00:44:59.12935431Z caller=web.go:380 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2018-01-13T00:44:59.129362349Z caller=main.go:314 msg="Starting TSDB"
level=info ts=2018-01-13T00:44:59.129373268Z caller=targetmanager.go:71 component="target manager" msg="Starting target manager..."
level=info ts=2018-01-13T00:44:59.134681785Z caller=main.go:326 msg="TSDB started"
level=info ts=2018-01-13T00:44:59.13471518Z caller=main.go:394 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=error ts=2018-01-13T00:44:59.141352462Z caller=manager.go:485 component="rule manager" msg="loading groups failed" err="yaml: unmarshal errors:\n  line 1: cannot unmarshal !!str `ALERT s...` into rulefmt.RuleGroups"
level=error ts=2018-01-13T00:44:59.141401118Z caller=main.go:413 msg="Failed to apply configuration" err="error loading rules, previous rule set restored"
level=error ts=2018-01-13T00:44:59.141417474Z caller=main.go:356 msg="Error loading config" err="one or more errors occurred while applying the new configuration (--config.file=/etc/prometheus/prometheus.yml)"

Update 2

Looks like the error is with the rule_files config in prometheus.yml:

rule_files:
  - "alert.rules"
  # - "first.rules"
  # - "second.rules"

I commented out:

rule_files:
  # - "alert.rules"
  # - "first.rules"
  # - "second.rules"

And it runs, data is now flowing in. I'm not sure what the side effect is of removing this but the dashboard does display.

In terms of metrics, it would be great if npm stats could be incorporated as well as travis-ci build statuses! Thanks for the project.

no data points

hi , I am following this guide https://github.com/vegasbrianc/github-monitoring and no data points in github stats dashboard
go to http://localhost:9090/targets and shows

Endpoint State Labels Last Scrape Error
http://localhost:9090/metrics up instance="localhost:9090" 14.744s ago  
http://metrics:9171/metrics up instance="metrics:9171" 7.262s ago  
http://node-exporter:9100/metrics down instance="node-exporter:9100" 14.295s ago Get http://node-exporter:9100/metrics: dial tcp: lookup node-exporter on 127.0.0.11:53: server misbehaving

i also checked the date looks good

any idea ?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.