Stand with the people of Israel

HiYaPyCo - A Hierarchical Yaml Python Config

A simple python lib allowing hierarchical overlay of config files in YAML syntax, offering different merge methods and variable interpolation based on jinja2.

The goal was to have something similar to puppets hiera merge_behavior: deeper for python.

• hierarchical overlay of multiple YAML files
• multiple merge methods for hierarchical YAML files
• variable interpolation using jinja2

• PyYAML aka. python3-yaml
• Jinja2 aka. python3-jinja2

HiYaPyCo was designed to run on current major python versions without changes. Tested versions:

• 3.9
• 3.11

A simple example:

import hiyapyco
conf = hiyapyco.load('yamlfile1' [,'yamlfile2' [,'yamlfile3' [...]]] [,kwargs])
print(hiyapyco.dump(conf, default_flow_style=False))

yaml1.yaml:

---
first: first element
second: xxx
deep:
    k1:
        - 1
        - 2

yaml2.yaml:

---
second: again {{ first }}
deep:
    k1:
        - 4 
        - 6
    k2:
        - 3
        - 6

load ...

>>> import pprint
>>> import hiyapyco
>>> conf = hiyapyco.load('yaml1.yaml', 'yaml2.yaml', method=hiyapyco.METHOD_MERGE, interpolate=True, failonmissingfiles=True)
>>> pprint.PrettyPrinter(indent=4).pprint(conf)
{   'deep': {   'k1': [1, 2, 4, 6], 'k2': [3, 6]},
    'first': u'first element',
    'ma': {   'ones': u'12', 'sum': u'22'},
    'second': u'again first element'}

>>> import hiyapyco
>>> y1="""
... yaml: 1
... y:
...   y1: abc
...   y2: xyz
... """
>>> y2="""
... yaml: 2
... y:
...   y2: def
...   y3: XYZ
... """
>>> conf = hiyapyco.load([y1, y2], method=hiyapyco.METHOD_MERGE)
>>> print (conf)
OrderedDict([('yaml', 2), ('y', OrderedDict([('y1', 'abc'), ('y2', 'def'), ('y3', 'XYZ')]))])
>>> hiyapyco.dump(conf, default_flow_style=True)
'{yaml: 2, y: {y1: abc, y2: def, y3: XYZ}}\n'

All args are handled as file names or yaml documents. They may be strings or list of strings.

• method: bit (one of the listed below):
  • hiyapyco.METHOD_SIMPLE: replace values (except for lists a simple merge is performed) (default method)
  • hiyapyco.METHOD_MERGE: perform a deep merge
  • hiyapyco.METHOD_SUBSTITUTE: perform a merge w/ lists substituted (unsupported)
• mergelists: boolean try to merge lists of dict (default: True)
• interpolate: boolean : perform interpolation after the merge (default: False)
• castinterpolated: boolean : try to perform a best possible match cast for interpolated strings (default: False)
• usedefaultyamlloader: boolean : force the usage of the default PyYAML loader/dumper instead of HiYaPyCos implementation of a OrderedDict loader/dumper (see: Ordered Dict Yaml Loader / Dumper aka. ODYLDo) (default: False)
• encoding: string : encoding used to read yaml files (default: utf-8)
• failonmissingfiles: boolean : fail if a supplied YAML file can not be found (default: True)
• loglevel: int : loglevel for the hiyapyco logger; should be one of the valid levels from logging: 'WARN', 'ERROR', 'DEBUG', 'I NFO', 'WARNING', 'CRITICAL', 'NOTSET' (default: default of logging)
• loglevelmissingfiles: int : one of the valid levels from logging: 'WARN', 'ERROR', 'DEBUG', 'INFO', 'WARNING', 'CRITICAL', 'NOTSET' (default: logging.ERROR if failonmissingfiles = True, else logging.WARN)

For using interpolation, I strongly recomend not to use the default PyYAML loader, as it sorts the dict entrys alphabetically, a fact that may break interpolation in some cases (see test/odict.yaml and test/test_odict.py for an example). See Ordered Dict Yaml Loader / Dumper aka. ODYLDo

The default jinja2.Environment for the interpolation is

hiyapyco.jinja2env = Environment(undefined=Undefined)

This means that undefined vars will be ignored and replaced with a empty string.

If you like to change the jinja2 Environment used for the interpolation, set hiyapyco.jinja2env before calling hiyapyco.load!

If you like to keep the undefined var as string but raise no error, use

from jinja2 import Environment, Undefined, DebugUndefined, StrictUndefined
hiyapyco.jinja2env = Environment(undefined=DebugUndefined)

If you like to raise a error on undefined vars, use

from jinja2 import Environment, Undefined, DebugUndefined, StrictUndefined
hiyapyco.jinja2env = Environment(undefined=StrictUndefined)

This will raise a hiyapyco.HiYaPyCoImplementationException wrapped arround the jinja2.UndefinedError pointing at the string causing the error.

See: jinja2.Environment

As you must use interpolation as strings (PyYAML will weep if you try to start a value with {{), you can set castinterpolated to True in order to try to get a best match cast for the interpolated values. The ``best match`` cast is currently only a q&d implementation and may not give you the expected results!

This is a simple implementation of a PyYAML loader / dumper using OrderedDict from collections. Because chaos is fun but order matters on loading dicts from a yaml file.

https://github.com/zerwes/hiyapyco

git clone https://github.com/zerwes/hiyapyco
cd hiyapyco
sudo python setup.py install

Download the latest or desired version of the source package from https://pypi.python.org/pypi/HiYaPyCo. Unpack the archive and install by executing:

sudo python setup.py install

Install the latest wheel package using:

pip install HiYaPyCo

install the latest debian packages from http://repo.zero-sys.net/hiyapyco:

# create the sources list file:
sudo echo "deb http://repo.zero-sys.net/hiyapyco/deb ./" > /etc/apt/sources.list.d/hiyapyco.list

# import the key:
gpg --keyserver keys.gnupg.net --recv-key 77DE7FB4
# or use:
wget https://repo.zero-sys.net/77DE7FB4.asc -O - | gpg --import -

# apt tasks:
gpg --armor --export 77DE7FB4 | sudo tee /etc/apt/trusted.gpg.d/hiyapyco.asc
sudo apt-get update
sudo apt-get install python3-hiyapyco

use http://repo.zero-sys.net/hiyapyco/rpm as URL for the yum repo and https://repo.zero-sys.net/77DE7FB4.asc as the URL for the key.

An AUR package is available (provided by Pete Crighton and not always up to date).

Copyright © 2014 - 2024 Klaus Zerwes zero-sys.net

This package is free software. This software is licensed under the terms of the GNU GENERAL PUBLIC LICENSE version 3 or later, as published by the Free Software Foundation. See https://www.gnu.org/licenses/gpl.html

FIXED: #67 cosmetic changes

FIXED: #60 recursive calls to _substmerge

IMPROVED: testing and python support (3.11)

MERGED: #52 by ryanfaircloth

MERGED: #41 Jinja2 dependency increased to include Jinja2 3.x.x

REMOVED: Support for Python 2

MERGED: #37 alex-ber

MERGED: #30 lesiak:issue-30-utf

MERGED: #28 lesiak:issue-28

FIXED: issue #33

MERGED: issue #32

IMPLEMENTED: [issue #27] support multiple yaml documents in one file

FIXED: logging by Regev Golan

IMPLEMENTED: mergelists (see issue #25)

FIXED: issue #24 repo signing

FIXED: issue #23 loglevelonmissingfiles

Fixed pypi doc

Reverted: logger settings to initial state

Improved: dump

Merged:

• flatten mapping from Chris Petersen [email protected]
• arch linux package info from Peter Crighton [email protected]

MERGED: fixes from mmariani

FIXED: issues #9 and #11

deb packages:

• removed support for python 2.6
• include examples as doc

FIXED: issue #6 import of hiyapycoversion* in setup.py causes pip install failures*

Changed: moved to GPL

Improvements: missing files handling, doc

Implemented: castinterpolated

Implemented: loading yaml docs from string

Improved tests and bool args checks

Implemented a Ordered Dict Yaml Loader

Fixed unicode handling

Initial release