voxpupuli / puppet-prometheus Goto Github PK
View Code? Open in Web Editor NEWPuppet module for prometheus
Home Page: https://forge.puppet.com/puppet/prometheus
License: Apache License 2.0
Puppet module for prometheus
Home Page: https://forge.puppet.com/puppet/prometheus
License: Apache License 2.0
Please tag your 0.1.1 release.
Thanks!
I notice the acceptance tests tend to fail in travis and they seem to be inconsistent/flaky. This should be investigated. I talked with @bastelfreak and he looked into it but was unable to reproduce it locally.
class { 'prometheus':
install_method => 'package',
...
Sep 27 20:30:54 hostname systemd[2208]: Failed at step EXEC spawning /usr/local/bin/prometheus: No such file or directory
If is "package" installation chosen, should be also used systemd service distributed with package.
If 'package' method is chosen, module still using own service which make prometheus not run. And without modules own service 'prometheus.yam' is not loaded because default is 'prometheus.yml'. So, i'm missing something or is this not supported use case? Thanks!
the binary is now called alertmanager the systemd template is hardcoded to alert_manager
Hi!
Is there any option to change the Prometheus port to something different than 9090? I cannot see such options anywhere in the code, but maybe it is buried somewhere. I can make PR and add this possibility to specify a port, it this is not a problem.
https://forge.puppet.com/puppet/prometheus still shows release Version 1.0.0 released Mar 27th 2017
but here https://github.com/voxpupuli/puppet-prometheus/releases there is also v2.0.0 and v3.0.0
Would it be possible to refactor the code to be puppet 4 compatible sooner rather than later? From what I understand puppet3 modules are going out of support soon. I would be open to putting in a PR and the branching the module in it's current state on the puppet3 branch.
We had our Puppetfile
pointing to the puppet3
branch of this repository, but it seems that this branch no longer exists. The branch does exist, however, in https://github.com/bastelfreak/puppet-prometheus2. What is the difference between this repository and https://github.com/bastelfreak/puppet-prometheus2?
The recently merged support for the blackbox_exporter contains a typo for the config.file
parameter which is a long-form parameter in blackbox_exporter but erroneously passed as -config.file
.
Actual behaviour: blackbox_exporter is started with -config.file=/etc/blackbox-exporter.yaml
and bails out.
Expected behaviour: blackbox_exporter is started with --config.file=/etc/blackbox-exporter.yaml
and works.
Reference:
Under some circumstances (which I still couldn't pinpoint), puppet is unable to find the installed systemd file with "Could not find init script for node_exporter" (probably affects other daemons as well). An easy fix seems to be passing provider => $init_style
in daemon.pp
.
Hi,
Getting error while using puppet-prometheus
init.pp
class { '::prometheus':
version => '2.0.0',
}
Error shown below
==> prometheus: Error: Execution of '/usr/local/bin/promtool check rules /etc/prometheus/alert.rules20180110-5183-s8nrev' returned 1: Checking /etc/prometheus/alert.rules20180110-5183-s8nrev
==> prometheus: FAILED:
==> prometheus: yaml: unmarshal errors:
==> prometheus: line 1: cannot unmarshal !!seq into rulefmt.RuleGroups
==> prometheus: Error: /Stage[main]/Prometheus::Alerts/File[/etc/prometheus/alert.rules]/ensure: change from absent to file failed: Execution of '/usr/local/bin/promtool check rules /etc/prometheus/alert.rules20180110-5183-s8nrev' returned 1: Checking /etc/prometheus/alert.rules20180110-5183-s8nrev
==> prometheus: FAILED:
==> prometheus: yaml: unmarshal errors:
==> prometheus: line 1: cannot unmarshal !!seq into rulefmt.RuleGroups
probably something wrong here: https://github.com/voxpupuli/puppet-prometheus/blob/master/manifests/alerts.pp#L36
BR
Karen
Hi, you claim that it depends on >=puppetlabs-stdlib 4.6.0, but Absolutepath.pp was introduced in 4.13.0.
All version from 4.6.0 to 4.12.0 are not working with your module.
Would you be able to update the dependencies please ?
Using a puppet master/agent architecture
As soon as he prometheus module is installed in an environment he custom fact will be collected on master and agents, triggering the bug.
When attempting to collect the custom prometheus_alert_manager_running
fact an error occurs:
(Facter) error while resolving custom fact "prometheus_alert_manager_running": execution of command "puppet resource service alert_manager 2>&1" failed: command not found.
No error - fact collected as normal.
see above
I think this error is down to the fact that puppet
is not in the PATH
when puppet runs as part of the service. The default systemd
service file that comes with the puppetlabs collections does not insert /opt/puppetlabs/bin
into the PATH
of the service file.
In contrast, the same collection does add a drop-in at /etc/profile.d/
to ensure login-shells do have puppet
in the path (so running the command from the fact from a login shell works fine).
I would suggest the fix is to put in the full path of the puppet
binary into the fact - ideally there's some way of fetching that inside Facter
.
It's probably a matter of taste, but I think an explicit option for setting storage retention (as opposed to using prometheus::extra_options
) would be a nice minor addition.
Any thoughts?
We should install the promtool so that configuration can be checked before reloading the service.
./promtool -help
usage: promtool <command> [<args>]
Available commands:
check-config validate configuration files for correctness
check-rules validate rule files for correctness
help prints this help text
version print the version of this binary
It would be nice to be able to deploy the rabbitmq_exporter.
Hello,
Most of exporters pulled from github after unpacking have owner/group different than root:root. In some specific cases non-root user can replace binary with malicious code, and run it with (sometimes, maybe) more permissions as exporter user. Or just fake diagnostic data, which can lead to other issues.
Hi !
Really nice module but right now i do not want to configure the scraping options directly in manifests file. I would like to provide my own prometheus.yml
file with all the configuration already done.
As I can see right now you populate proemtheus.yml
from prometheus::config
class. What do you thing about one new parameter to prometheus
class: scrape_config_file
. If there will be something in this file just drop it in correct place and that is all. If this parameter is empty and scrape_configs
are set do what you are doing right now.
Could this module manage the rule files? I would like to be able to make sure the rule file is defined before the service starts.
Use older Puppet agent (we use Puppet master 4.8.2).
[...]
Info: Loading facts
Error: Could not parse application options: invalid option: --to_yaml
Could not retrieve fact='prometheus_alert_manager_running', resolution='<anonymous>': undefined method `[]' for false:FalseClass
Info: Caching catalog for mynode
[...]
No warning
Hi I would like to suggest that the example config also includes config for a local node_exporter :)
Also - thanks for making this module!
Based on the best practices documented in the Prometheus migration guide it may be necessary to run two versions in parallel while data history is built.
https://prometheus.io/docs/prometheus/latest/migration/
Due the puppet constraints, as I understand them, it requires some unique customization to run these two instances. It would be wonderful if this was a capability of this profile. Essentially we would want to start the 1.8.x version in a non-scraping mode and v2.1 with a remote_read reference to the old instance running on the same box during a migration period.
include prometheus::node_exporter
Notice: /Stage[main]/Prometheus::Node_exporter/Prometheus::Daemon[node_exporter]/Group[prometheus]/ensure: created
Notice: /Stage[main]/Prometheus::Node_exporter/Prometheus::Daemon[node_exporter]/User[prometheus]/ensure: created
Notice: /Stage[main]/Prometheus::Node_exporter/Prometheus::Daemon[node_exporter]/Package[prometheus-node-exporter]/ensure: created
Error: Systemd start for node_exporter failed!
journalctl log for node_exporter:
-- No entries --
Error: /Stage[main]/Prometheus::Node_exporter/Prometheus::Daemon[node_exporter]/Service[node_exporter]/ensure: change from stopped to running failed: Systemd start for node_exporter journalctl log for node_exporter:
-- No entries --
Notice: /Stage[main]/Prometheus::Node_exporter/Prometheus::Daemon[node_exporter]/Systemd::Unit_file[node_exporter.service]/File[/etc/systemd/system/node_exporter.service]/ensure: defined content as '{md5}b94d9aa7e7c105276490a5eebf37560b'
...
Note that the service was started before the unit file was created, resulting in an error.
The service is only started after the unit file has been created.
vlad@mini:~ $ /opt/puppetlabs/bin/puppet module install puppet-prometheus --version 4.0.0
vlad@mini:~ $ /opt/puppetlabs/bin/puppet apply --noop -e 'class { "prometheus::node_exporter": version => "0.15.2", }'
Error: Facter: error while resolving custom fact "prometheus_alert_manager_running": Could not find init script or upstart conf file for 'alert_manager'
No error
vlad@mini:~ $ /opt/puppetlabs/bin/puppet module install puppet-prometheus --version 4.0.0
Notice: Preparing to install into /home/vlad/.puppetlabs/etc/code/modules ...
Notice: Created target directory /home/vlad/.puppetlabs/etc/code/modules
Notice: Downloading from https://forgeapi.puppet.com ...
Notice: Installing -- do not interrupt ...
/home/vlad/.puppetlabs/etc/code/modules
└─┬ puppet-prometheus (v4.0.0)
├── camptocamp-systemd (v1.1.1)
└─┬ puppet-archive (v2.2.0)
└── puppetlabs-stdlib (v4.24.0)
vlad@mini:~ $ /opt/puppetlabs/bin/puppet apply --noop -e 'class { "prometheus::node_exporter": version => "0.15.2", }'
Error: Facter: error while resolving custom fact "prometheus_alert_manager_running": Could not find init script or upstart conf file for 'alert_manager'
Notice: Compiled catalog for mini in environment production in 1.17 seconds
Notice: Applied catalog in 0.50 seconds
I'd prefer to have the option not to manage prometheus.yml
.
As a prerequisite for #149, we probably have to support providing arbitrary environment variables to daemons, similarly to how we use prometheus::daemon::options
.
It would be nice to have the individual *_exporter
export puppet resources that could be collected on the prometheus node and automatically configure scrapers.
I already have a PR for this on the pipeline, so this ticket is just for reference, while the PR doesn't get submitted.
part of the code from the hiera (json):
"relabel_configs": [
{
"source_labels": "['__meta_consul_service']",
"regex": "'(.*)'",
"target_label": "service",
"replacement": "$1"
}
]
after converting to yaml:
- "source_labels": >-
['__meta_consul_service']
"regex": >-
'(.*)'
"target_label": >-
'service'
"replacement": >-
'$1'
relabel_configs:
- source_labels: ['__meta_consul_service']
regex: '(.*)'
target_label: 'service'
replacement: '$1'
Error: /Stage[main]/Prometheus::Config/File[prometheus.yaml]/content: change from '{md5}8822bf5e2346fa4e41e3cfb3f5e1d3d2' to '{md5}465e86f9620ff98445bc84004f7d491b' failed: Execution of '/usr/local/bin/promtool check-config /etc/prometheus/prometheus.yaml20180215-2965-nsyebg' returned 1: Checking /etc/prometheus/prometheus.yaml20180215-2965-nsyebg
FAILED: yaml: unmarshal errors:
line 26: cannot unmarshal !!str['__met...
into model.LabelNames
I would like to make sure the stdout of prometheus is logged to a file. I would like this done on the Debian version of this module.
jhooyberghs (on slack):
hey Martin, if you have some time somewhere, I've got a little question about a commit/merge of yours in puppet-prometheus:
58d0da6
We've noticed that since that commit, our prometheus wouldn't start anymore at boot time, because it now has a dependency on "multi-user.target" as well as being part of it. The reason in our setup was a problem in fstab, so the multi-user.target dependency wasn't met. Do you have any recollection of why this dependency was added? The commit message only speaks about the "restart" change and I fail to see a deeper reason 🙂
I declared in the init.pp manifest file the following code, which is pretty straightforward.
class { 'prometheus':
rule_files => ['/etc/prometheus/testing.rules'],
scrape_configs => $scrape_configs,
extra_options => $extra_options_real,
localstorage => "${data_dir}/prometheus",
storage_retention => $retention,
}
after running Puppet on the host, I couldn't see any change related to the parameter $rule_files. Furthermore, the resulting prometheus configuration has set an empty array:
I see no changes in regards of $rule_files parameters, and looks like this parameter is not in use at all. The resulting yaml configuration shows only:
...
rule_files: []
...
if you read line 194-224:
$extra_rule_files = suffix(prefix(keys($extra_alerts), "${config_dir}/rules/"), '.rules')
if ! empty($alerts) {
::prometheus::alerts { 'alert':
alerts => $alerts,
location => $config_dir,
}
$_rule_files = concat(["${config_dir}/alert.rules"], $extra_rule_files)
}
else {
$_rule_files = $extra_rule_files
}
-> class { '::prometheus::config':
global_config => $global_config,
rule_files => $_rule_files,
scrape_configs => $scrape_configs,
remote_read_configs => $remote_read_configs,
remote_write_configs => $remote_write_configs,
config_template => $config_template,
storage_retention => $storage_retention,
}
You can notice that actually the value is calculated from $_rule_files
We are currently using version 3.1.0
in production|staging|dev but we want to upgrade to 5.0.0
, however the first step is being able to set the rules parameter since at the moment is loading prometheus without any of our current rules, so I expect to have our rules in place, in a format such as:
rule_files:
- "/etc/prometheus/rules/alerts-service1/*.rules"
- "/etc/prometheus/rules/common/*.rules"
ps: there is another issue in regards of importing rules using that style, see golang/go#11862 for further information, but at the moment it works for us.
Please let me know if you need something from this side.
cheers!
Hello,
I keep prometheus config in hiera, looks like this:
prometheus::scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets:
- 'localhost:9090'
labels:
instance: 'prometheus'
- job_name: 'linux'
static_configs:
- targets:
- 'ser1:9100'
- 'ser2:9100'
- 'ser3:9100'
After switching to latest master commit, my /etc/prometheus/prometheus.yaml transformed from nice yaml into "garbage":
---
"global":
"scrape_interval": >-
15s
"evaluation_interval": >-
15s
"external_labels":
"monitor": >-
master
"rule_files":
- >-
/etc/prometheus/alert.rules
"scrape_configs":
- "job_name": >-
prometheus
"static_configs":
- "targets":
- >-
localhost:9090
After little digging I found out that it's related to change introduced in
681bc1b
Some details about environment
when trying to use version 0.13.0 of alerts manager the service configuration revert to <0.12.0
the problem is in the condition on line 211 of module:
https://github.com/voxpupuli/puppet-prometheus/blob/master/manifests/alertmanager.pp
The node exporter has updated and has decent defaults. We should change the params to simply not specify any collectors as the default.
There are packages for debian in testing and backports; it would be very nice if they were used.
Hello,
You should update the version of the module on the puppet forge, because it's not working with systemd as is and you also fixed some bugs :)
Regards
Run puppet agent with version 4.13.1 of stdlib specified in Puppetfile
Manifest failure with error:
Error: Unknown function: 'to_yaml'. at /etc/puppetlabs/code/environments/prometheus/third_party/prometheus/manifests/alerts.pp:35:30
Manifest compilation.
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Unknown function: 'to_yaml'. at /etc/puppetlabs/code/environments/prometheus/third_party/prometheus/manifests/alerts.pp:35:30 on node test
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Updated to version 4.20.0 of stdlib and the manifest compiled.
I checked the changelog of stdlib and the to_yaml
function appears to have been introduced in version 4.20.0
https://forge.puppet.com/puppetlabs/stdlib/changelog#supported-release-4200
Many thanks for this module.
In the alertmanager.pp
manifests there is a service resource that instructs puppet to stop the alert_manager
service if it is running, since the service has been renamed. This works fine on an old instance that has the alert_manager
service, but breaks down on new instances that have no such service. I am seeing the following error on a fresh instance that attempts to start the alertmanager:
/Stage[main]/Prometheus::Alertmanager/Service[alert_manager]) Could not evaluate: Could not find init script or upstart conf file for 'alert_manager'
puppet-prometheus/manifests/alertmanager.pp
Lines 205 to 207 in 56dbfab
Is the new Prometheus version 2.0.0 supported by this module?
When we use
class { 'prometheus::node_exporter':
version => '0.14.0',
...
}
for the monitored nodes, we get also a prometheus server installed on the monitored nodes.
promethes server gets installed (Version 1.5.2)
and node exporter gets installed (Version 0.14.0)
only node exporter gets installed (Version 0.14.0)
i think prometheus::node_exporter inherits prometheus
thats why prometheus server gets installed in Version 1,
see your documentation in section "On the server (for prometheus version >= 1.0.0)"
On Ubuntu14.04 we use the wrong reload command. Currently hardcoded is upstart, but that isn't present in all 14.04 releases (or in none?). It should be changed to 'service' which wraps sysv or upstart.
See templates/alert.epp
1.0 syntax
2.0 yaml syntax
Hi,
When I install the prometheus and alertmanagers fails because de download url needs a diferent path:
Example puppet output :
err: /Stage[main]/Prometheus::Alert_manager::Install/Staging::File[alert_manager-0.5.0-alpha.0.tar.gz]/Exec[/opt/staging/prometheus/alert_manager-0.5.0-alpha.0.tar.gz]/returns: change from notrun to 0 failed: curl -f -L -o /opt/staging/prometheus/alert_manager-0.5.0-alpha.0.tar.gz https://github.com/prometheus/alertmanager/releases/download/0.5.0-alpha.0/alertmanager-0.5.0-alpha.0.linux-amd64.tar.gz returned 22 instead of one of [0]
Thanks Oscar
While I can define where prometheus can find a recording rule file, this module gives me no means to specify the contents of that file.
Ideally I would like to be able to use the same module to handle both the prometheus configuration and recording rule definitions.
Changes to configuration or alert rules results in the Prometheus service restarting.
This is regularly leading to crash-recovery
mode on Prometheus. Often sending kill -HUP
to reload the service would be sufficient. Is this a change you'd be okay making?
Hi all,
The version 1.0.0 seems to be released on March 27th 2017.
Could you tag this version ?
Best regards.
While starting to use alertmanager i noticed the default inhibit_rules cause alertmanager to not start
class { '::prometheus::alertmanager':
version => '0.14.0',
install_method => 'url',
route => { 'group_by' => [ 'alertname', 'cluster', 'service' ], 'group_wait'=> '30s', 'group_interval'=> '5m', 'repeat_interval'=> '3h', 'receiver'=> 'slack' },
receivers => [ { 'name' => 'slack', 'slack_configs'=> [ { 'api_url'=> 'https://hooks.slack.com/services/secret', 'channel' => '#test', 'send_resolved' => true, 'username' => 'username'}] }]
}
(so, not setting inhibit_rules)
this causes alertmanager to be installed, but crases after starting, upon investigating i found the error when i started alertmanager by hand.
an empty running alertmanager
root@723fefb34eec:/docker/development# alertmanager --config.file /etc/alertmanager/alertmanager.yaml
level=info ts=2018-04-05T10:32:22.8074184Z caller=main.go:136 msg="Starting Alertmanager" version="(version=0.14.0, branch=HEAD, revision=30af4d051b37ce817ea7e35b56c57a0e2ec9dbb0)"
level=info ts=2018-04-05T10:32:22.8074907Z caller=main.go:137 build_context="(go=go1.9.2, user=root@37b6a49ebba9, date=20180213-08:16:42)"
level=info ts=2018-04-05T10:32:22.8112018Z caller=main.go:275 msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yaml
level=error ts=2018-04-05T10:32:22.8118108Z caller=main.go:278 msg="Loading configuration file failed" file=/etc/alertmanager/alertmanager.yaml err="unknown fields in inhibit rule: severity"
I have created a pull request to generate the same config as params.pp used to make
puppet-prometheus
downdoads packages directly from the Internet. Unfortunately my hosts are are behind a HTTP proxy thus is always fails when downloading. Since archive
moduel itself supports proxy_server
and proxy_type
, shall I add configuration in puppet-promethues
to enable HTTP proxy?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.