ripienaar / puppet-module-data Goto Github PK

Hiera backend that stores data in a module

Puppet 4.30% Ruby 95.70%

puppet-module-data's Introduction

What?

NOTE: There is now native support for this feature in Puppet 4 and this repo is now officially deprecated and moving to unmaintained.

Additionally this module does not work on Puppet 4, will likely break your setup. Do not use it on Puppet 4, migrate to native Module Data support

See https://www.devco.net/archives/2016/01/08/native-puppet-4-data-in-modules.php

While hiera does a decent job of separating code and data for users it is quite difficult for module authors to use hiera to create reusable modules. This is because the puppet backend is optional and even when it is loaded the module author cannot influence the hierarchy.

With this commit we add a new module_data backend that loads data from within a module and allow the module author to set a hierarchy for this data.

The goal of this backend is to allow module authors to specify true module default data in their modules but still allow users to override the data using the standard method - especially useful with the puppet 3 hiera integration.

This backend is always loaded as the least important tier in the hierarchy - unless a user choose to put it somewhere specific, but this backend will always be enabled.

Given a module layout:

your_module
├── data
│   ├── hiera.yaml
│   └── osfamily
│       ├── Debian.yaml
│       └── RedHat.yaml
└── manifests
    └── init.pp

The hiera.yaml is optional in this example it would be:

---
:hierarchy:
- osfamily/%{::osfamily}
- common

But when no hiera.yaml exist in the module, the default would be:

---
:hierarchy:
- common

The data directory is then a standard Hiera data store.

Status?

This is but a first stab at turning my old pull request for ticket 16856 into a standalone module that any > 3.0.0 Puppet user can depend on to get this essential feature.

Some more testing is needed, sanity checking for support versions etc so consider this a early feedback-saught release

Contact?

R.I.Pienaar / [email protected] / @ripienaar / http://devco.net

puppet-module-data's People

Contributors

Stargazers

Watchers

puppet-module-data's Issues

Tag the 0.0.1 release

Hey @ripienaar would you mind tagging the 0.0.1 release on github? A simple git tag will do it, or you can use the Releases Page. If you tag as 0.0.1, this will fill the lives of librarian-puppet users with 🌈

Hiera merging defaulting to native

Configuration exists in several yaml files that I would like to merge together.

When using hiera.yaml with merge_behavior: deeper the expected behavior happens.
If I move merge_behavior: deeper out of the top level hiera.yaml and into the module's data/hiera.yaml the behavior defaults to native merging. I would like my module to be independent of global configuration and be able to function with the merge_behavior specified within the module.

I have put together a sort of hacky fix that loads the module's merged config and then replaces it before the key is returned here: elijahsgh@8e30018

I'm not sure that is the ideal way to go about changing Hiera's merge_behavior to deeper.

Module Data backend over-rides core Hiera backend...

Firstly, thanks for a great addition, I've found it very useful.

Anyhow, the issue I'm seeing is that I'm using the module data backend to ship data for some internal application modules.
This works great for prod, however we now want to use the same modules in Dev, whilst changing some of the values that are pulled in from hiera...

I'd hoped it would have been a simple case of providing the modified values in the core hiera data, however it appears that the module data backend gets looked up after the core backend, thereby preventing us from over-riding application module data...

An example of this from a debug puppet run is:

Debug: hiera(): Looking up remotepost::resources in YAML backend
Debug: hiera(): Looking for data source applications/ncr
Debug: hiera(): Looking for data source applications/remotepost
Debug: hiera(): Found remotepost::resources in applications/remotepost
Debug: hiera(): Looking for data source nodes/rps-node1
Debug: hiera(): Looking for data source os/CentOS
Debug: hiera(): Looking for data source common/common
Debug: hiera(): Looking up remotepost::resources in Module Data backend
Debug: hiera(): Reading config from /tmp/vagrant-puppet-3/modules-0/remotepost/data/hiera.yaml file
Debug: hiera(): Looking for data in source /tmp/vagrant-puppet-3/modules-0/remotepost/data/version/.yaml
Debug: hiera(): Looking for data in source /tmp/vagrant-puppet-3/modules-0/remotepost/data/common.yaml

Is there a way around this?

Cheers
Gav

Don't limit lookup to module namespace

Would it be possible to not limit the hiera lookups to the module namespace ?

PE 3 (Puppet 3.2.4) Loading Issues

When testing the module in Puppet Enterprise 3.0.1 (3.2.4) with puppet apply, the module does not appear to load data settings:

Debug: Scope(Class[main]): Calling hiera
Debug: hiera(): Hiera YAML backend starting
Debug: hiera(): Looking up profile::data::rez_server in YAML backend
Debug: hiera(): Looking for data source defaults
Debug: hiera(): Cannot find datafile /var/lib/hiera/defaults.yaml, skipping
Debug: hiera(): Looking for data source underwatch.localdomain
Debug: hiera(): Cannot find datafile /var/lib/hiera/underwatch.localdomain.yaml, skipping
Debug: hiera(): Looking for data source production
Debug: hiera(): Cannot find datafile /var/lib/hiera/production.yaml, skipping
Debug: hiera(): Looking for data source global
Debug: hiera(): Cannot find datafile /var/lib/hiera/global.yaml, skipping
Debug: Scope(Class[main]): Calling hiera_debug
Notice: Scope(Class[main]): bad

Figured out my problem with the data lookup, just not sure why it wasn't able to inject module_data_backend automatically at this point.

"Can't convert nil into hash" error

I seem to be hitting an issue when using this module in combination with hash resources.

The error I'm seeing is:

Error: can't convert nil into Hash at /home/vagrant/puppet/modules/test/manifests/test.pp:99 on node rpbo.test.local
Wrapped exception:
can't convert nil into Hash
Error: can't convert nil into Hash at /home/vagrant/puppet/modules/test/manifests/test.pp:99 on node rpbo.test.local

The code around there is:

  #hiera_resources("res")
  $res = hiera_hash('res')
  notify { 'res':
    message => "Res = ${res}"
  }

As you can see, I've tried both hiera_resources and hiera_hash, and both return the same result.

And my common.yml contains:

res:
  notify:
    'module1':
       message: 'Data in module 1'

From a bit of debugging, I've tracked it to: module_data_backend.rb#83.

If I add an additional line before that with:

answer ||= {}

then it all works as expected.

Is this a known issue? Way I'm using it? Or a genuine bug?

Cheers
Gav

Push 0.0.4 to Forge

I see 0.0.4 is tagged in Github, but it's not on the Forge yet.

[QUESTION] Does it support environment hierarchy ?

Hi,

I created two environments in my node (dev and production).

I would like to provide good defaults for each environments using different hiera files. My first approach was to combine defaults in one hiera file (common.yaml) and it works really great.

Now, I would like to split defaults for each environment.

I updated my hiera.yaml with - "%{::environment}" :


---
:version: 3
:hierarchy:
  - "%{::environment}"
  - common
:backends:
  - yaml

Then in the data folder of my module, alongside common.yaml, I created two files dev.yaml and production.yaml.

But the settings, I put in those files are not retrieved.

Am I missing something ?

Thanks,

Puppet 4 warning has broken my environment...

Updating this module to 615bc91 seems to have broken all the Data module lookups on our Puppet 3.7.5 master...

Will try and get some more debug output in due course, however wanted to get an early warning out there for others...

New tag following pull request

Can you please add a new tag (e.g. 0.0.4)? I use puppet-librarian, and it's somewhat dependent on tags in order to figure out what code to grab.

I guess I should have added a tag in my pull request? I'm not really sure of the correct workflow/etiquette for the situation.

Updating to 0.0.2 results in uninitilized constant

So I updated the module to 0.0.2 from 0.0.1 and seem to be getting the following error when running boxen

~ > boxen
Boxen is up-to-date.
Error: uninitialized constant Puppet::DataBinding::LookupError at /opt/boxen/repo/manifests/site.pp:1 on node winterfell.local
Wrapped exception:
uninitialized constant Puppet::DataBinding::LookupError
Error: uninitialized constant Puppet::DataBinding::LookupError at /opt/boxen/repo/manifests/site.pp:1 on node winterfell.local

Reverting back to 0.0.1 seems to fix the error, have I missed something when updating to 0.0.2?

[QUESTION] Is it possible to use this module to provide included class defaults ?

Hi,

I am currently writing "meta" modules that aggregates various modules by theme (security, monitoring, defaults, backup...).

I would like to store those defaults in hiera. Can I use this module to do that ?

It seems that the settings are searched directly in the hiera files attached to the node and then fall back directly to the hiera files associated with the included module. The "meta" module is skipped.

I tried several patterns, like inherits or composition but I didn't succeed.

Any hints ?

Thanks again,

Mickaël

[QUESTION] Is this module deprecated when using Puppet version > 4 or > 4.2 ?

Hi,

I want to manage data in modules using hiera. It seems that, since version 4, puppet has a way to do that.

However, I didn't find any straightforward documentation on the web on how to use it.

Can you tell me if your module is still relevant with puppet version superior to 4 ? If not, do you know some good ressources on how to implement the native way ?

Thanks,

No license

Can you add a note about how this code is licensed?

module_data incompatible with puppet 3.4.0

Your module requires 'puppet/indirector/hiera'. This is gone in puppet 3.4.0.

It has been included in puppet/indirector/code. Have tried to rename it but doesn't seem to work.