See Adrian's comment here #43 (comment):

I wonder if it would be better to use the puppet module face for this than shelling out. These are callable from Ruby code like this:

require 'puppet'
require 'puppet/face'

face = Puppet::Face[:module, :current]
modules = face.list(environment: 'production')

# Each key is the modulepath, and its value is an array of the modules
has_warnings = modules[:modules_by_path].any? {|k,v|
  v.any? {|mod|
    !mod.unmet_dependencies.empty?
  }
}

I tested this in a couple of scenarios and it seems to work. It's also more accurate, since it's possible for a user to have a module with the word "Warning" in it, which I know is unlikely, but still possible.

Use Case

PE manages a certificate for the console. This can be the same as the agent cert,but it doesn't has to be. I think it would be really helpful to check if the used cert expires in < 90 days.

Describe the Solution You Would Like

As done in S0015, we could read the cert:

The default path is /opt/puppetlabs/server/data/console-services/certs/console-cert.cert.pem or /opt/puppetlabs/server/data/console-services/certs/${certname}.cert.pem. But the path is configureable in puppet_enterprise::profile::console::browser_ssl_cert. I don't think we can access the parameter easily from a fact. We could read it from /etc/puppetlabs/console-services/conf.d/console.conf.

A different approach would be to make an HTTP request to the console and get the cert.

Describe Alternatives You've Considered

A customer could use their internal monitoring tool for this, but since pe_status_check already validates certificates and people might get the impression that it covers every important part, I think it makes sense to integrate this into pe_status_check.

Describe the Bug

The query used to retrieve the status for S0026 is too slow in PE environments with a large number of compilers and/or environments, which are queried as part of the full-scope check (to retrieve the file sync status information, I believe): https://github.com/puppetlabs/puppetlabs-pe_status_check/blob/main/lib/facter/pe_status_check.rb#L300

This query takes a little under 5 seconds in my environment, which significantly exceeds the hard coded 2 second request timeout in the fact.

Expected Behavior

S0026 should be able to retrieve the JVM value without timing out.

Using a more narrow scoping for the service check looks to be possible and greatly decreases the query time (from just under 5 seconds to around 0.2 seconds in my environments): /status/v1/services/status-service?level=debug

Environment

• PE 2021.7.1
• pe_status_check v2.4.1

Additional Context

The timeout exceed condition was seen in a "large" PE deployment with 21 environments, 11 compilers, and a DR node.

Use Case

It was decided that yum provider messages should only be a warning in https://tickets.puppetlabs.com/browse/PUP-5594. This can then go undetected until puppet attempts to manage packages and then causes a failed puppet run. Warnings are not visible on PE dashboards unless viewing the reports directly.

Describe the Solution You Would Like

A fact should either check yum check-update comes back clean or check the logs for "Warning: Puppet::Type::package::ProviderYum: Could not check for updates, '/bin/yum check-update' exited with 1" messages

Describe Alternatives You've Considered

This issue could be worked around by users setting repos correctly to ignore broken sources but this is rarely done and outside of our control.

Additional Context

Customers have missed configurations for months and then been impacted during key events such as upgrades.

Use Case

Sometimes my code seems to silently fail to deploy--especially in my development environment where I'm pretty ruthless with check-ins and squash-updates.

Describe the Solution You Would Like

I'd like a fact that checks the status of the last attempted code deploy--was it successful for all environments?

Describe Alternatives You've Considered

I'm not aware of any alternatives. However, maybe this fact could be rolled into something more generic like the "health" of code deployment. Not sure how reasonable it would be, but perhaps some of the checks described here https://puppet.com/docs/pe/2019.8/code_mgr_troubleshoot.html#troubleshooting_code_manager-monitor-webhook-deployment-trigger-issues could be rolled into this check...

Additional Context

Would be nice to see a warning when I do a puppet run on my primary--as I know the pe_status_check does already--if the code deployment is not healthy. For example, I have automated code deploys so didn't notice the following error until doing it manually. My puppet runs for the 'development' all seemed to work fine apart from the fact that I didn't see my new code check-ins. The issue was pretty simple: I had a typo on one of my modules...

Describe the Bug

On Linux based systems where the ss -onp state established '( dport = :8142 )' command is run the command will not finish before the timeout if the system is moderately busy and there are many thousands of processes in the process table. This is because every process has to be examined to determine which sockets are associated with it.

On a system with moderate load, thousands of users, and approximately 10,000 processes the -p option on the ss command consistently had it take longer than 2 seconds.

Expected Behavior

A large and busy system should not prevent this check from working properly. Furthermore, this check should probably attempt to be a bit more efficient so as to not run so long on busy systems.

S0006 should be confined to primary nodes only

Use Case

PE 2021.6 ships the pe_databases module by default I think. PE recommends using it for older versions as well. It configures 3 timers that run pg_repack on the pe-puppetdb. pg_repack requires free diskspace that's at least as big as the (biggest table+index)*2

from https://reorg.github.io/pg_repack/:

Performing a full-table repack requires free disk space about twice as large as the target table(s) and its indexes. For example, if the total size of the tables and indexes to be reorganized is 1GB, an additional 2GB of disk space is required.

Describe the Solution You Would Like

It would be helpful if the fact could identify the biggest table and multiply it by two and use that as lower limit

Describe Alternatives You've Considered

if another implementation is too tricky, maybe the message for S0007 should be adjusted like:

more than 20% free, but check you table size. maybe you need more free disk space.

Additional Context

Indeed pg_repack doesn't work reliable in my environment when I've less than the recommended free storage.

Describe the Bug

All pe log parsing indicator tests are incorrectly keying off "puppet config print logdir"

this resolves to

/var/log/puppetlabs/puppet
and not

/var/log/puppetlabs/

So none of the indicators are actually checking the correct log files

Use Case

Have a check to alert if a table is above a certain threshold. For example, the certnames table. It would like this value to have a sane value but also be configurable.

Describe the Solution You Would Like

Alert me if key tables are larger than a set threshold and allow me to configure the threshold where necessary.

PE2019.8.11

Plan uses environmental hiera and fails on

      - "nodes/%{trusted.certname}.yaml"

pe_status_check::infra_summary:

{
  "msg" : "Interpolations are not supported in lookups outside of an apply block: Undefined variable 'trusted' (file: /opt/puppetlabs/server/data/orchestration-services/code/environments/production/hiera.yaml)",
  "kind" : "bolt/pal-error",
  "details" : { }
}

Describe the Bug

We have this in our code:

grep -r max_queued_requests data/
data/roles/pe_xl/compiler.yaml:puppet_enterprise::master::puppetserver::jruby_puppet_max_queued_requests: 150
data/roles/pe_xl/master.yaml:puppet_enterprise::master::puppetserver::jruby_puppet_max_queued_requests: 32

Still, we’re getting “flagged” for S0036 (puppetlabs-pe_status_check v2.5.0):

Notice: S0036 is at fault. The indicator S0036 Determines if max-queued-requests is set above 150, refer to documentation for required action

Notice: /Stage[main]/Pe_status_check/Notify[pe_status_check S0036]/message: defined 'message' as 'S0036 is at fault. The indicator S0036 Determines if max-queued-requests is set above 150, refer to documentation for required action'
...
$ facter -p pe_status_check.S0036
false
...
$ grep max-queued-requests /etc/puppetlabs/puppetserver/conf.d/pe-puppet-server.conf
    max-queued-requests: 150

Could that be because https://github.com/puppetlabs/puppetlabs-pe_status_check/blob/main/lib/facter/pe_status_check.rb#L428 only checks for less than, not less than or equal?

Thanks

Expected Behavior

A clear and concise description of what you expected to happen.

Steps to Reproduce

See above

Environment

• Version 2019.8.12
• Platform RHEL7

Additional Context

Describe the Bug

When for example puppetserver-access.log does not exist or is unreadable an uncaught exception is raised when computing the custom fact.

Expected Behavior

Guard against some files not being on disk or unreadable

Steps to Reproduce

Delete puppetserver-access.log and try to compute the custom fact

Additional Context

Here is the issue we are seeing in CI (when an unrelated issue is causing puppet-access.log to not be written)

Use Case

It would be convenient to be able to store indicator_exclusions in plan hiera
so instead of having to list exclusions in the command line
bolt plan run pe_status_check::infra_summary indicator_exclusions='["S0035"]'
plan could lookup pe_status_check::indicator_exclusions from hiera, similar to the class behavior

Describe the Bug

The acceptance tests for S0036 are not reliable and cause intermittent failures. It also does not clean up the setting that was configured.

Example failure: https://github.com/puppetlabs/puppetlabs-self_service/runs/4968232026?check_suite_focus=true#step:13:25

Recommendation

Change the acceptance test from using hiera and the puppet agent to using a puppet apply that manages the setting.

https://github.com/puppetlabs/puppetlabs-self_service/blob/main/spec/acceptance/self_service_spec.rb#L153-L160

Use present to create the failure and absent to remove the failure state.

  pe_hocon_setting { 'jruby-puppet.max-queued-requests':
    ensure  => present,
    path    => '/etc/puppetlabs/puppetserver/conf.d/pe-puppet-server.conf',
    setting => 'jruby-puppet.max-queued-requests',
    value   => 151,
  }

Use Case

A status check that fails when PuppetDB table sizes in PostgreSQL exceed a threshold. E.g certnames larger than 25MB, etc

Use Case

Issues with LDAP/DS connectivity are often not noticed until functions such as API access via an AD user start to fail

Describe the Solution You Would Like

A fact and task running the DS check API

Describe Alternatives You've Considered

Metrics collector or service status but both would require much longer implementations

Additional Context

https://puppet.com/docs/pe/2019.8/rbac_api_v1_directory_get_ds_test.html test the connectivity to ldap

Use Case

Users can get revoked accidentally and its not noticed until it causes impact due to access failures.

Describe the Solution You Would Like

A status check via task and fact which listed out revoked users would be useful to allow monitoring.

Describe Alternatives You've Considered

I can't think of another way of doing this right now, maybe metrics collector should have this?

Additional Context

Check for revoked users using https://puppet.com/docs/pe/2019.8/rbac_api_v1_user_get_users.html with a filter

Acceptance test labels for 16 17 and 18 appear to setup tests for different facts that the results are for

Describe the Bug

We now have the option to use license.key or suite-license.lic as license files for PE. S0022 is looking just for license.key, if it doesn't find it, it triggers an alert.

Expected Behavior

S0022 should be updated to include checking for suite-license.lic and, maybe, triggering an alert if both license files are present.

Steps to Reproduce

Assuming you're not managing the license key as a resource:

• Remove license.key
• Install a correctly formatted suite-license.lic
• Restart pe-puppetserver
• Run puppet agent, check report.

Environment

• PE 2023.8.0

Additional Context

I'm going to try and take a look at fixing it myself, but no promises!