Giter VIP home page Giter VIP logo

lib's Introduction

Hari Sekhon - Perl Library

GitHub stars GitHub forks Lines of Code License My LinkedIn GitHub Last Commit

Codacy Badge CodeFactor SonarCloud Quality Gate Status Maintainability Rating Reliability Rating Security Rating Vulnerabilities

Linux Mac Docker Dockerfile DockerHub Pulls DockerHub Build Automated

CI Builds Overview Jenkins Concourse GoCD TeamCity

CircleCI BuildKite AppVeyor Drone Codefresh Cirrus CI Semaphore Buddy Shippable Travis CI

Azure DevOps GitLab Pipeline BitBucket Pipeline AWS CodeBuild GCP Cloud Build

Repo on GitHub Repo on GitLab Repo on Azure DevOps Repo on BitBucket

ShellCheck JSON YAML XML Validation Kics Grype Semgrep Semgrep Cloud Trivy

GitHub Actions Ubuntu Mac Mac 11 Mac 12 Ubuntu Ubuntu 20.04 Ubuntu 22.04 Debian Debian 10 Debian 11 Debian 12 CentOS CentOS 7 CentOS 8 Fedora Alpine Alpine 3

Perl versions Perl

Perl library, full of lots of validation code and utility functions.

Needed for a lot of the programs I've written over the years. In fact my current main library was actually cobbled together from lots of pieces of code I wrote over the years since I found myself reusing common things over and over. This drastically reduces the amount of code and effort required to write new robust well validated code which is why it's used extensively throughout the portions of code you'll find on my GitHub account, especially all the Advanced Nagios Plugins Collection which I've been developing for many years since 2006

Hari Sekhon

Cloud & Big Data Contractor, United Kingdom

My LinkedIn

(you're welcome to connect with me on LinkedIn)

Build + Unit Tests

make &&
make test

Continuous Integration is run on this repo to build and unit test it (Test::More, almost 800 unit tests).

Configuration

Strict validations include host/domain/FQDNs using TLDs which are populated from the official IANA list, a snapshot of which is shipped as part of this project.

To update the bundled official IANA TLD list with the latest valid TLDs do

make tld
Custom TLDs

If using bespoke internal domains such as .local, .intranet, .vm, .cloud etc. that aren't part of the official IANA TLD list then this is additionally supported via a custom configuration file resources/custom_tlds.txt containing one TLD per line, with support for # comment prefixes. Just add your bespoke internal TLD to the file and it will then pass the host/domain/fqdn validations.

IO::Socket::SSL doesn't respect ignoring self-signed certs in recent version(s) eg. 2.020

Recent version(s) of IO::Socket::SSL (2.020) seem to fail to respect options to ignore self-signed certs. The workaround is to create the hidden touch file below in the same top-level directory as the library to make it include and use Net::SSL instead of IO::Socket::SSL.

touch .use_net_ssl

Related Repositories

Python and Java ports of this library can be found below - both with higher levels of code coverage testing:

More Related Repositories

  • DevOps Bash Tools - 1000+ DevOps Bash Scripts, Advanced .bashrc, .vimrc, .screenrc, .tmux.conf, .gitconfig, CI configs & Utility Code Library - AWS, GCP, Kubernetes, Docker, Kafka, Hadoop, SQL, BigQuery, Hive, Impala, PostgreSQL, MySQL, LDAP, DockerHub, Jenkins, Spotify API & MP3 tools, Git tricks, GitHub API, GitLab API, BitBucket API, Code & build linting, package management for Linux / Mac / Python / Perl / Ruby / NodeJS / Golang, and lots more random goodies

  • SQL Scripts - 100+ SQL Scripts - PostgreSQL, MySQL, AWS Athena, Google BigQuery

  • Jenkins - Advanced Jenkinsfile & Jenkins Groovy Shared Library

  • GitHub-Actions - GitHub Actions master template & GitHub Actions Shared Workflows library

  • Templates - dozens of Code & Config templates - AWS, GCP, Docker, Jenkins, Terraform, Vagrant, Puppet, Python, Bash, Go, Perl, Java, Scala, Groovy, Maven, SBT, Gradle, Make, GitHub Actions Workflows, CircleCI, Jenkinsfile, Makefile, Dockerfile, docker-compose.yml, M4 etc.

  • Kubernetes configs - Kubernetes YAML configs - Best Practices, Tips & Tricks are baked right into the templates for future deployments

  • Terraform - Terraform templates for AWS / GCP / Azure / GitHub management

  • DevOps Python Tools - 80+ DevOps CLI tools for AWS, GCP, Hadoop, HBase, Spark, Log Anonymizer, Ambari Blueprints, AWS CloudFormation, Linux, Docker, Spark Data Converters & Validators (Avro / Parquet / JSON / CSV / INI / XML / YAML), Elasticsearch, Solr, Travis CI, Pig, IPython

  • DevOps Perl Tools - 25+ DevOps CLI tools for Hadoop, HDFS, Hive, Solr/SolrCloud CLI, Log Anonymizer, Nginx stats & HTTP(S) URL watchers for load balanced web farms, Dockerfiles & SQL ReCaser (MySQL, PostgreSQL, AWS Redshift, Snowflake, Apache Drill, Hive, Impala, Cassandra CQL, Microsoft SQL Server, Oracle, Couchbase N1QL, Dockerfiles, Pig Latin, Neo4j, InfluxDB), Ambari FreeIPA Kerberos, Datameer, Linux...

  • The Advanced Nagios Plugins Collection - 450+ programs for Nagios monitoring your Hadoop & NoSQL clusters. Covers every Hadoop vendor's management API and every major NoSQL technology (HBase, Cassandra, MongoDB, Elasticsearch, Solr, Riak, Redis etc.) as well as message queues (Kafka, RabbitMQ), continuous integration (Jenkins, Travis CI) and traditional infrastructure (SSL, Whois, DNS, Linux)

  • Nagios Plugin Kafka - Kafka API pub/sub Nagios Plugin written in Scala with Kerberos support

  • HAProxy Configs - 80+ HAProxy Configs for Hadoop, Big Data, NoSQL, Docker, Elasticsearch, SolrCloud, HBase, Cloudera, Hortonworks, MapR, MySQL, PostgreSQL, Apache Drill, Hive, Presto, Impala, ZooKeeper, OpenTSDB, InfluxDB, Prometheus, Kibana, Graphite, SSH, RabbitMQ, Redis, Riak, Rancher etc.

  • Dockerfiles - 50+ DockerHub public images for Docker & Kubernetes - Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Mesos, Consul, Riak, OpenTSDB, Jython, Advanced Nagios Plugins & DevOps Tools repos on Alpine, CentOS, Debian, Fedora, Ubuntu, Superset, H2O, Serf, Alluxio / Tachyon, FakeS3

  • HashiCorp Packer templates - Linux automated bare-metal installs and portable virtual machines OVA format appliances using HashiCorp Packer, Redhat Kickstart, Debian Preseed and Ubuntu AutoInstaller / Cloud-Init

  • Diagrams-as-Code - Cloud & Open Source architecture diagrams with Python & D2 source code provided - automatically regenerated via GitHub Actions CI/CD - AWS, GCP, Kubernetes, Jenkins, ArgoCD, Traefik, Kong API Gateway, Nginx, Redis, PostgreSQL, Kafka, Spark, web farms, event processing...

  • Knowledge-Base - IT Knowledge Base from 20 years in DevOps, Linux, Cloud, Big Data, AWS, GCP etc.

git.io/perl-lib

lib's People

Contributors

davidgibbons avatar frankgruellich avatar harisekhon avatar mfechner avatar smartattack avatar ted3 avatar themysteriousx avatar zachaller avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

lib's Issues

check_ssl_cert.pl: unable to load certificate ...:error:...:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: TRUSTED CERTIFICATE

I'm having problems with a cert. (internal cert issued by us) If I'm checking google's output is correct:

./check_ssl_cert.pl -H google.com
OK: 70 days remaining for '*.google.com'. Certificate Expires: 'Feb 23 14:17:00 2017 GMT'

However if I'm checking my internal cert is not working, is for webmail and it requieres authentication. Please find below the output.

verbose mode on

check_ssl_cert.pl version 0.9.12 => Hari Sekhon Utils version 1.18.6

host: XXXXX
port: 443
warning lower: 31
critical lower: 15

setting timeout to 10 secs

cmd: /usr/bin/openssl version -a
output:

OpenSSL 1.0.2g 1 Mar 2016
built on: reproducible build, date unspecified
platform: debian-amd64
options: bn(64,64) rc4(16x,int) des(idx,cisc,16,int) blowfish(idx)
compiler: cc -I. -I.. -I../include -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -m64 -DL_ENDIAN -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wa,--noexecstack -Wall -DMD32_REG_T=int -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
OPENSSLDIR: "/usr/lib/ssl"

exitcode: 0

Found CApath from openssl binary as: /usr/lib/ssl

CA path directory: /usr/lib/ssl

  • checking validity of cert (chain of trust)
    cmd: echo | /usr/bin/openssl s_client -connect XXXXXX:443 -CApath /usr/lib/ssl 2>&1
    output:

139761281857176:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
CONNECTED(00000003)

no peer certificate available

No client certificate CA names sent

SSL handshake has read 0 bytes and written 305 bytes

New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1481734653
Timeout : 300 (sec)
Verify return code: 0 (ok)

exitcode: 1

Verify return code: 0 (ok)

  • checking domain and expiry on cert
    cmd: echo | /usr/bin/openssl s_client -connect XXXXX:443 -CApath /usr/lib/ssl 2>&1 | /usr/bin/openssl x509 -noout -text 2>&1
    output:

unable to load certificate
140653386462872:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: TRUSTED CERTIFICATE

exitcode: 1

CRITICAL: failed to determine certificate domain name

mac_regex error

Hi,

I think there is a mistake in mac_regex.
The first block is [0-9A-F-af] and should be [0-9A-Fa-f]

our $mac_regex = '\b[0-9A-F-af]{1,2}[:-](?:[0-9A-Fa-f]{1,2}[:-]){4}[0-9A-Fa-f]{1,2}\b';

Thank you

aws secret key

The aws secret key is not necessarily alphanumeric. My suggestion is to remove the alphanum check, or go with the aws recommended regex. However, amazon says this is also subject to change.

From http://blogs.aws.amazon.com/security/blog/tag/key+rotation

"· Search for access key IDs: (?<![A-Z0-9])[A-Z0-9]{20}(?![A-Z0-9]). In English, this regular expression says: Find me 20-character, uppercase, alphanumeric strings that don’t have any uppercase, alphanumeric characters immediately before or after.

· Search for secret access keys: (?<![A-Za-z0-9/+=])[A-Za-z0-9/+=]{40}(?![A-Za-z0-9/+=]). In English, this regular expression says: Find me 40-character, base-64 strings that don’t have any base 64 characters immediately before or after."

"We can’t guarantee these regular expressions will work forever (we reserve the right to change the format of access key IDs and secret access keys) but they’re a start."

Unknown switch conditio

./check_whois.pl --help

Unknown switch condition (?(DE in regex; marked by <-- HERE in m/
(?( <-- HERE DEFINE)
(? -? (?= [1-9]|0(?!\d) ) \d+ (.\d+)? ([eE] [+-]? \d+)? )
(? true | false | null )
(? " ([^"\\]* | \ ["\bfnrt/] | \ u [0-9a-f]{4} )* " )
(? [ (?: (?&json) (?: , (?&json) )* )? \s* ] )
(? \s* (?&string) \s* : (?&json) )
(? { (?: (?&pair) (?: , (?&pair) )* )? \s* } )
at HariSekhonUtils.pm line 1224.

AttributeError: 'KafkaLoggingHandler' object has no attribute 'producer'

Hello
I use the check_kafka script and I have the following error message, and I do not see where the problem is, below the logging.conf file

[loggers]
keys=root

[handlers]
keys=consoleHandler,fileHandler,kafkaHandler

[formatters]
keys=simpleFormatter,logstashFormatter

[logger_root]
level=DEBUG
handlers=consoleHandler,fileHandler,kafkaHandler
formatter=simpleFormatter

[handler_fileHandler]
class=logging.handlers.WatchedFileHandler
level=DEBUG
formatter=simpleFormatter
args=('/var/log/check_kafka.log', 'a')

[handler_kafkaHandler]
class=python_kafka_logging.KafkaHandler.KafkaLoggingHandler
level=DEBUG
formatter=logstashFormatter
args=("192.168.0.35:2181, 192.168.0.193:2181","apps-openshift")

[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=simpleFormatter
args=(sys.stdout,)

[formatter_simpleFormatter]
format=%(asctime)s - %(levelname)s - %(name)s - %(message)s

[formatter_logstashFormatter]
class=logstash_formatter.LogstashFormatter
format={"extra": {"appName": "myPythonApp", "environment": "AWS-Test"}}

can you help me ?
thanks

Hostnames starting with numbers: 1.example.com

Hi,

check_ssl_cert.pl does not like my hostname:

% ./check_ssl_cert.pl --host 1.example.com |head -n3
invalid host '1.example.com' defined: not a valid hostname or IP address

usage: check_ssl_cert.pl [ options ]
% 

The regex hostnames are matched against expects them to start with letters ([A-Za-z]). However, numbers in hostnames are perfectly valid.

Kind regards,
Frank.

Improve your SIG{__DIE__} handler?

Hi,
I found myself here because a coworker found bug #73 in your nagios plugins repo, and nearly tried to fork NetAddr::IP::InetBase thinking it was a fault in that module. Luckily I did some quick digging to see that it was a $SIG{__DIE__} handler in our code breaking everything. It was undoubtedly the same issue affecting check_kafka.pl

I'm wondering if you haven't tried using $EXCEPTIONS_BEING_CAUGHT or $^S inside your die handler to prevent it from exiting if you don't need it to.

Better yet -- you can always stop exiting from your $SIG{__DIE__} handler which would allow exceptions to bubble up naturally. You might even be able to avoid using the signal handler entirely.. Read the bottom couple paragraphs of the signal var for more information on avoiding $SIG{__DIE__}.

allow username as all numeric

our usernames are fully numeric and that's failing (i'm guessing) your username regex, which needs to have leading alpha?

curl timeout exceeds check timeout causing unknown errors instead of critical failures

existing scripts fail with "UNKNOWN" errors because script timeouts are normally shorter than default LWP timeout. this is a problem because instead of being down/failed, service shows up as UNKNOWN which is a soft state not picked up by some monitoring platforms

this happens with or without SSL, it's simply the LWP curl timeout

[user@somehost]$ ./check_elasticsearch_cluster_disk_balance.pl -H 192.168.1.3 -S --ssl-noverify -u someuser -p somepassword -t 2
UNKNOWN: self timed out after 2 seconds

this can be fixed for each specific check type by making sure LWP timeout is set to be less than global script timeout, so that curl is allowed to time out and the error code is handled properly. this can also be set in the global curl function by also applying global timeout to LWP timeout

Expected behaviour should be:

[user@somehost]$ ./check_elasticsearch_cluster_disk_balance.pl -H 192.168.1.3 -S --ssl-noverify -u someuser -p  somepassword -t 2
CRITICAL: 500 Can't connect to 192.168.1.3:9200 (timeout)

lib will not build on centos7

git submodule update --init --recursive
fatal: reference is not a tree: c039fd2a8f50fc38e7f7bbe730b4b1013cb876d0
Unable to checkout 'c039fd2a8f50fc38e7f7bbe730b4b1013cb876d0' in submodule path 'bash-tools/templates'
Failed to recurse into submodule path 'bash-tools'

DML reserved words need whitespace

MySQL queries containing reserved words fail due to DML statement match. i.e. - 'select foo from product_updates where created_date ... ' will match both update and create. Suggestion would be to match reserved words + whitespace ? This worked for me :

$ git diff
diff --git a/HariSekhonUtils.pm b/HariSekhonUtils.pm
index 66e4fcb..c0fb60b 100644
--- a/HariSekhonUtils.pm
+++ b/HariSekhonUtils.pm
@@ -3302,7 +3302,7 @@ sub validate_database_query_select_show ($;$) {
     #debug("regex validating query: $query");
     $query =~ /^\s*((?:SHOW|SELECT)\s+.+)$/i || usage "invalid ${name}query defined: may only be a SELECT or SHOW statement";
     $query = $1;
-    $query =~ /insert|update|delete|create|drop|alter|truncate/i and usage "invalid ${name}query defined: found DML statement keywords!";
+    $query =~ / insert | update | delete | create | drop | alter | truncate /i and usage "invalid ${name}query defined: found DML statements keywords!";
     # this trips up users who put ; at the end of their query and doesn't offer that much protection anyway since DML is already checked for and it may be convenient to
     #$query =~ /;|--/i and usage "invalid ${name}query defined: suspect chars ';' or '--' detected in query!";
     $query =~ /;/ and usage "invalid ${name}query defined: you may not add semi-colons to your queries, while it works on the command line, Nagios ends up choking by pre```


I'm not a perl expert (or even a novice) so this may be wrong way of going about it.

Domain name regex

Hello,

is it possible to modify the 'domain_component' regex to accept also domains like '_spf1.example.com'.

my $domain_component = '\b[a-zA-Z0-9](?:[a-zA-Z0-9\-]{0,61}[a-zA-Z0-9])?\b';

Thank you.

Simple Instructions?

Hi,

I wanted to use your elasticsearch and couchbase plugins. Which seem to require your perl lib etc. I don't need or want any hadoop stuff etc, nor do I want to clone the entire library of plugins onto my server.

I apologize for being a noob with perl and git, but how can I simply get the prereqs and the specific plugins that I want to work on my system?

check_elasticsearch.pl required HariSekhonUtils.pm which in turn is looking for JSON.pm, which I don't see in your repo. So like the title says, do you have some simple instructions so I can get these plugins running on Nagios?

Thanks in advance for any help you can offer.

Taint Check Error

Just updated Perl (v5.18.2, not sure what it was before) and I am getting this error:
OK: Insecure dependency in``while running with -T switch at /usr/local/lib/nagios-plugins/harisekhon/lib/HariSekhonUtils.pm line 986.

Called from check_whois.pl.

Add user/password options for SolR checks

Hello,
That would be great to be able to specify the user and password used by the solr plugins.
Should be quite simple, as it is just some arguments of the curl_json function, located in lib/HariSekhon/Solr.pm, currently set to undef.
By the way, excellent plugins!
BR,
Yannick

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.