The sensu user is used by other packages and we can't assume that it's safe to remove.

Under Sensu 0.27 we're seeing an exception from EventMachine when transport ssl attributes are configured:

terminate called after throwing an instance of 'std::runtime_error'
  what():  Encryption not available on this event-machine

With systemd, smf and other service management systems we can't depend on being able to conditionally set environment variables (e.g. setting PATH and GEM_PATH based on value of EMBEDDED_RUBY).

As a result, we should investigate consolidating any such logic in refactored sensu bin stubs that can set up the environment properly before starting the process. This should give us consistent behavior across platforms without having to maintain the same behavior across multiple service management systems.

Relates to sensu/sensu-build#108

The Sensu packages currently built with the deprecated sensu-build toolchain installs sysv scripts by default. The package also offers an optional embedded runit supervisor, and provides example service scripts for upstart and systemd.

In this sense, upstart and systemd have been considered "unofficially supported" service supervisors for a while now. Unfortunately this "unofficial" designation has meant that the upstart and systemd scripts have long been subject of issues like the those described in sensu/sensu-build#108, and elsewhere.

With the upcoming 0.27 release of Sensu we will be addressing this problem by removing the embedded runit supervisor, dropping upstart as a supported init, and adding official support for systemd.

TL;DR 0.27 will include the following changes to service management:

• The embedded runit supervisor will be removed from the Sensu package
• The unofficially supported upstart scripts will be removed from the Sensu Package
• If sysv scripts are present, the package will update them to use the new sensu-service binstub
• If no sysv scripts are present and systemd is the running init, systemd unit files will be installed in /etc/systemd/system
• We will build individual packages for each supported platform and release version.
• These per-platform version packages will unconditionally install init scripts or systemd unit files, depending on the init or service management system supported by the platform release version.

This will be a breaking change for some folks, particularly those who currently rely on the embedded runit supervisor.

By now we think it is clear that systemd has become the defacto standard for the Linux distributions we aim to support, and that this change to support sysv and systemd exclusively will be an overall improvement for Sensu operators and administrators.

We're presently dependent on our own fork of the Omnibus cookbook (sensu/omnibus-cookbook) to run our builds via Travis, both on Linux and Windows. The dependency stems from the following:

• Added live_stream property to omnibus_build resources so we get enough output to allow Travis to finish a build Merged in chef-boneyard/omnibus#193)
• Added conditions to avoid errors on Windows platforms where the omnibus group cannot be created Merged in chef-boneyard/omnibus#189)

Changes that enable the above are present in the sensu branch of sensu/omnibus-cookbook.

Once the PRs linked above have been merged and released, we should switch to an upstream release of the cookbook. we should rewrite the sensu branch on upstream master.

The sensu RPM is purging sensu files during a yum upgrade. After an upgrade of the sensu RPM from sensu-0.27.0-2 to sensu-0.27.0-3 the package is installed but there are no longer /opt/sensu and /var/cache/sensu directories.

I'm seeing this happen on CentOS Linux release 7.3.1611. I'm running Puppet in opt mode on my dev and qa systems which are set to install the latest sensu package. As these systems updated to the sensu-0.27.0-3 package I noticed sensu was no longer running. Digging deeper I see that the /opt/sensu directory is gone. This is happening on dozens of VMs consistently.

I've also manually installed the sensu-0.27.0-2 package then upgraded to sensu-0.27.0-3 using yum commands and see the same behaviour. So I know it is not a problem with puppet managing the package.

I have to do some more debugging but I think it may be a problem with the postuninstall scriptlet. It looks like this part of the script is executed during an upgrade even though it should not.

# only execute this script on rhel-based distros if an rpm uninstall
# is taking place
if [ $1 -eq 0 ]; then
  purge_sensu_services
  purge_sensu_files
fi

During an upgrade the argument passed to the postuninstall scriptlet should be 1 and should not execute the code above but for some reason I think it is getting executed.

I question if the manual purging of sensu files is even necessary since the RPM itself should remove the files if the package is uninstalled.

To work around this issue right now I have to yum erase sensu and then reinstall. I'll keep digging into the root cause and potential fix of the issue.

All of our sensu-build RPM releases have epoch=1. Our new build tools do not support the epoch attribute and as a result it defaults to 0. This causes an issue where old builds are actually seen as "newer" as the epoch value takes precedence over the Sensu version.

As a temporary workaround, I think we should implement support for epoch in Omnibus.

As a permanent fix, we should ensure epoch=0 and use a new yum repository free of any epoch=1 RPMs.

We want the ability to kick off Solaris 11 builds with test-kitchen just as we do with our other builds.

We're moving towards building packages in an automated fashion by using kitchen-ec2. We need to ensure that the the following platforms are in a working state:

• Ubuntu 12.04 (i386, amd64)
• Ubuntu 14.04 (i386, amd64)
• Ubuntu 16.04 (i386, amd64)
• Debian 7 (i386, amd64)
• Debian 8 (i386, amd64)
• CentOS/RHEL 5 (i386, amd64) # no 1st party AMIs
• CentOS/RHEL 6 (i386, amd64)
• CentOS/RHEL 7 (i386, amd64)
• FreeBSD 10 (amd64)

in #58 we add some code to .kitchen.yml for copying GPG keys into the build VM when certain environment variables are set. We should try to reduce duplication in this file by defining helper methods which can be used in each applicable platform definition.

Per a friend of mine:

the solaris compiler toolchain outputs like 20x faster binaries [cf. gcc]

As a some-day nice-to-have, we should evaluate the truth of this claim and whether or not we can easily adapt our omnibus builds to use a compiler other than gcc.

The Sensu installer on Windows does not currently check for the .NET dependency needed for Sensu. I believe there is a way for the installer to refuse to install unless runtime dependencies exist. We should implement this if it is indeed possible.

Windows builds past and present ship without CA certificates in the location specified by OpenSSL::X509::DEFAULT_CERT_FILE

Proposed fix:
curl https://curl.haxx.se/ca/cacert.pem -o "$(ruby -ropenssl -e 'puts OpenSSL::X509::DEFAULT_CERT_FILE')"

We want the ability to kick off Solaris 10 builds with test-kitchen just as we do with our other builds.

The sysvinit scripts on CentOS/RedHat/Amazon Linux return an error exit code if you try to stop a stopped service or start a running service.

For all other init-script actions, the init script shall return an exit status of zero if the action was successful. Otherwise, the exit status shall be non-zero, as defined below. In addition to straightforward success, the following situations are also to be considered successful:

restarting a service (instead of reloading it) with the force-reload argument

running start on a service already running

running stop on a service already stopped or not running

running restart on a service already stopped or not running

running try-restart on a service already stopped or not running

Still the recommended behaviour -
http://refspecs.linuxfoundation.org/LSB_5.0.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html

and dates from LSB 1.0.0 http://refspecs.linuxfoundation.org/LSB_1.0.0/gLSB.html#INISCRPTACT

Please will you make this compliant, since it is causing problems with our production instances.

Migrating this issue (https://github.com/sensu/sensu-build/issues/117) from the sensu-build repository.

See http://blog.packagecloud.io/eng/2014/10/20/yum-createrepo-generate-incorrect-metadata/ for a detailed writeup.

In our case, omnibus is generating a "distro_tag" like ".el5" which is added to the package name, meaning we get artifacts like sensu-0.26.4-4.el5.x86_64.rpm. The bugs in createrepo generate repo metadata for such packages with an invalid release designation, e.g. 4.el5 instead of 4. Yum fails to download and install these newer packages with an invalid release.

In testing upgrades from 0.26.5-2 to 0.27.0.beta.1-3 I've found that failing to create the sensu-install symlink when executing the postinst script causes the package installation to be incomplete.

Via Chef debug log, i see the following

       [2016-11-24T01:35:49+00:00] DEBUG: yum_package[sensu]: yum command: "yum -d0 -e0 -y install sensu-0.27.0.beta.1-3.el5"
       Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
       You're about to install sensu!
       systemd detected, but sysv init scripts are present so we'll use those instead
       Installing sensu-api sysv init script
       Installing sensu-client sysv init script
       Installing sensu-server sysv init script
       Installing sensu-service-init sysv init script
       ln: failed to create symbolic link ‘/usr/bin/sensu-install’: File exists
       Thank you for installing Sensu!
       warning: /etc/default/sensu saved as /etc/default/sensu.rpmsave
       [2016-11-24T01:36:24+00:00] INFO: yum_package[sensu] installed sensu at 0.27.0.beta.1-3.el5

Even though the log indicates that sysv init scripts are being installed, I dont see them in /etc/init.d ?

The Windows MSI includes an XML file defining the service, and in #9 we have commits for adding the -d and -c flags to load configuration. I think it will be more consistent with our other packages if the service definition is installed by the package but not enabled/started automatically.

Let's start a checklist for any blockers preventing us from opening this repo to the public.

• Reposition AIX ISO and remove details from README
• Rename repo to sensu-omnibus for consistency w/ other projects

We should remove the config.json.example file included in packages as we're recommending using the conf.d directory for storing Sensu configuration.

Packages built this way don't appear to meet any of the requirements for inclusion in the official Debian or Ubuntu repositories.

Can you please prepare a proper package and propose it for inclusion in Debian?

Today, after upgrading sensu to v0.27, I could not find the service scripts (sensu-server, sensu-client, sensu-api) in the /etc/init.d/ directory. v0.27 seems to have removed them. Downgrading back to v0.26 brings them back.

# cat /etc/*release
PRETTY_NAME="Debian GNU/Linux 8 (jessie)"
NAME="Debian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=debian

# arch 
x86_64

Wondering if we should ship with Ruby 2.3.1 instead of 2.3.0. 2.3.1 includes many bug fixes and is considered the current stable version.

The build is green, Chef indicates omnibus publish ran successfully, but there's no artifact in the S3 bucket. When attempting to publish interactively we've run into MD5 checksum errors which we've attempted to work around thusly, but still no dice.

As described in sensu/sensu#1474, there appears to be a problem with current releases of rubygems when installing packages in some environments (see rubygems/rubygems#1626).

[SENSU-INSTALL] installing Sensu plugins ...
[SENSU-INSTALL] determining if Sensu gem 'sensu-plugins-process-checks' is already installed ...
false
[SENSU-INSTALL] Sensu plugin gems to be installed: ["sensu-plugins-process-checks"]
[SENSU-INSTALL] installing Sensu gem 'sensu-plugins-process-checks'
ERROR:  While executing gem ... (ArgumentError)
    IPv4 address expects 4 bytes but 0 bytes
[SENSU-INSTALL] failed to install Sensu gem 'sensu-plugins-process-checks'
[SENSU-INSTALL] you can run the sensu-install command again with --verbose for more info
[SENSU-INSTALL] please take note of any failure messages above
[SENSU-INSTALL] make sure you have build tools installed (e.g. gcc)
[SENSU-INSTALL] trying to determine the Sensu plugin homepage for sensu-plugins-process-checks ...
ERROR:  While executing gem ... (ArgumentError)
    IPv4 address expects 4 bytes but 0 bytes

Once this issue is fixed upstream we should make sure to update the version of rubygems used in our build.

In #34 we describe a behavior that isn't yet implemented in the postinst script, specifically:

If no sysv scripts are present and systemd is the running init, systemd unit files will be installed in /etc/systemd/system

but we also want to update the sysv scripts if they are present:

If sysv scripts are present, the package will update them to use the new sensu-service binstub

The omnibus-sensu builds currently have a timestamp in their filename and a manual rename is required before uploading them to the core repository VM.

The Omnibus-built MSI packages have a publisher name of [email protected]. This should be updated to Sensu, Inc.

In the move to using Omnibus & our new EC2 pipeline we lost the ability to build deb packages for the armhf & powerpc architectures. We should investigate using qemu to allow us to support a wide variety of architectures.

In my testing the PATH environment variable constructed by our sysv init scripts (e.g. /etc/init.d/sensu-server) is not effective when services are started by the sensu-service binstub.

As described in sensu/sensu#1606, this is causing Sensu services to run with an incorrect environment, e.g. HANDLER_DIR is being omitted in the effective PATH when sensu-service starts the sensu-server process.

#76

We currently use particular platform versions to generate "lowest common denominator" package artifacts, e.g. Building 32bit .deb packages on Debian 7.8 allows them be used on Ubuntu 12.04 and other newer 32bit distributions.

In my mind this means that cutting a new release using omnibus requires us to reference an undocumented build matrix to generate artifacts on the correct platforms, so we should document this matrix.

Add git tag ordering by date/time, no need for unique commits.

It may be possible to have upstart validate configuration before performing a restart on Sensu services. We need to look into this.

Readline support has been missing from Ruby in the omnibus packages since Sensu 0.20 as per sensu/sensu-build#151.

Installing 0.26.4-5 on Debian 7.11:

root@5b292e3a-bc2c-ccc2-b880-d0fde1408218:~# /etc/init.d/sensu-server start
Starting sensu-server:/opt/sensu/embedded/bin/ruby: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by /opt/sensu/embedded/lib/libruby.so.2.3)

Windows builds via omnibus are currently in a separate git branch and only work on a hand-configured Windows VM.

We want these changes merged to master and the ability to build artifacts by running kitchen converge as we do with other platforms.

We currently use a combination of the https://github.com/sensu/homebrew-tap and https://github.com/sensu/sensu-homebrew for MacOS builds. We should work towards adding MacOS support to this repository.

The sysvinit scripts (such as /etc/init.d/sensu-client), which are generated by https://github.com/sensu/sensu-omnibus/blob/master/config/templates/sensu-gem/sysvinit/sensu-service.erb

call start-stop-daemon thus:

start-stop-daemon --start --chuid sensu --pidfile /var/run/sensu/sensu-client.pid --exec /opt/sensu/bin/sensu-service start client fork

(/etc/init.d/sensu-client line 118 in 0.27)

The problem arises when it tries to create the PID_DIR and PID file in bin/sensu-service, which is this file: https://github.com/sensu/sensu-omnibus/blob/master/files/sensu-gem/bin/sensu-service

bin/sensu-service calls this:

ensure_dir() {
    if [ ! -d $1 ]; then
        mkdir -p $1
        chown -R $2 $1
        chmod 755 $1
    fi
}

On Ubuntu, /var/run (actually /run, to which /var/run is a symlink) is a temp dir on a ram drive, so goes away every boot. It is also owned by root. bin/sensu-service is invoked (by start-stop-daemon) as the sensu user. The sensu user cannot create anything in /var/run (nor chown anything), so it cannot create /var/run/sensu, and startup fails.

Either the init scripts need to create that directory, or sensu-service needs to be invoked as root, and drop privs before starting the sensu services.

On EL7 /var/run is a symlink to a tmpfs mounted on /run, so the contents are automatically cleared every reboot. sensu's systemd units run /opt/sensu/bin/sensu-service as the sensu user. This script attempts to ensure that /var/run/sensu and /var/log/sensu exist, but the sensu user cannot create directories under /run or /var/log. sensu services then fail to start because they cannot write a pid file.

{"timestamp":"2017-01-31T15:50:12.209662-0500","level":"fatal","message":"could not write to pid file","pid_file":"/var/run/sensu/sensu-client.pid"}

The systemd way to handle this is to install a systemd-tmpfiles configuration file and have the directories automatically created; e.g. putting d /run/sensu 0755 sensu sensu - in /etc/tmpfiles.d/sensu.conf.

FreeBSD is the only remaining platform that is still using https://github.com/sensu/sensu-build. This project successfully builds on FreeBSD 10.x but Omnibus does not support the pkgng packager. I am currently in the process of adding pkgng support.