zerwes / hiyapyco Goto Github PK

View Code? Open in Web Editor NEW

104.0 9.0 22.0 272 KB

HiYaPyCo - A Hierarchical Yaml Python Config

License: GNU General Public License v3.0

Makefile 14.14% Python 85.86%

python yaml configuration config merge python3 open-source

hiyapyco's Introduction

Stand with the people of Israel

hiyapyco

HiYaPyCo - A Hierarchical Yaml Python Config

Description

A simple python lib allowing hierarchical overlay of config files in YAML syntax, offering different merge methods and variable interpolation based on jinja2.

The goal was to have something similar to puppets hiera merge_behavior: deeper for python.

Key Features

hierarchical overlay of multiple YAML files
multiple merge methods for hierarchical YAML files
variable interpolation using jinja2

Requirements

PyYAML aka. python3-yaml
Jinja2 aka. python3-jinja2

Python Version

HiYaPyCo was designed to run on current major python versions without changes. Tested versions:

3.9
3.11

Usage

A simple example:

import hiyapyco
conf = hiyapyco.load('yamlfile1' [,'yamlfile2' [,'yamlfile3' [...]]] [,kwargs])
print(hiyapyco.dump(conf, default_flow_style=False))

real life example:

yaml1.yaml:

---
first: first element
second: xxx
deep:
    k1:
        - 1
        - 2

yaml2.yaml:

---
second: again {{ first }}
deep:
    k1:
        - 4 
        - 6
    k2:
        - 3
        - 6

load ...

>>> import pprint
>>> import hiyapyco
>>> conf = hiyapyco.load('yaml1.yaml', 'yaml2.yaml', method=hiyapyco.METHOD_MERGE, interpolate=True, failonmissingfiles=True)
>>> pprint.PrettyPrinter(indent=4).pprint(conf)
{   'deep': {   'k1': [1, 2, 4, 6], 'k2': [3, 6]},
    'first': u'first element',
    'ma': {   'ones': u'12', 'sum': u'22'},
    'second': u'again first element'}

real life example using yaml documents as strings

>>> import hiyapyco
>>> y1="""
... yaml: 1
... y:
...   y1: abc
...   y2: xyz
... """
>>> y2="""
... yaml: 2
... y:
...   y2: def
...   y3: XYZ
... """
>>> conf = hiyapyco.load([y1, y2], method=hiyapyco.METHOD_MERGE)
>>> print (conf)
OrderedDict([('yaml', 2), ('y', OrderedDict([('y1', 'abc'), ('y2', 'def'), ('y3', 'XYZ')]))])
>>> hiyapyco.dump(conf, default_flow_style=True)
'{yaml: 2, y: {y1: abc, y2: def, y3: XYZ}}\n'

args

All args are handled as file names or yaml documents. They may be strings or list of strings.

kwargs

method: bit (one of the listed below):
- hiyapyco.METHOD_SIMPLE: replace values (except for lists a simple merge is performed) (default method)
- hiyapyco.METHOD_MERGE: perform a deep merge
- hiyapyco.METHOD_SUBSTITUTE: perform a merge w/ lists substituted (unsupported)
mergelists: boolean try to merge lists of dict (default: True)
interpolate: boolean : perform interpolation after the merge (default: False)
castinterpolated: boolean : try to perform a best possible match cast for interpolated strings (default: False)
usedefaultyamlloader: boolean : force the usage of the default PyYAML loader/dumper instead of HiYaPyCos implementation of a OrderedDict loader/dumper (see: Ordered Dict Yaml Loader / Dumper aka. ODYLDo) (default: False)
encoding: string : encoding used to read yaml files (default: utf-8)
failonmissingfiles: boolean : fail if a supplied YAML file can not be found (default: True)
loglevel: int : loglevel for the hiyapyco logger; should be one of the valid levels from logging: 'WARN', 'ERROR', 'DEBUG', 'I NFO', 'WARNING', 'CRITICAL', 'NOTSET' (default: default of logging)
loglevelmissingfiles: int : one of the valid levels from logging: 'WARN', 'ERROR', 'DEBUG', 'INFO', 'WARNING', 'CRITICAL', 'NOTSET' (default: logging.ERROR if failonmissingfiles = True, else logging.WARN)

interpolation

For using interpolation, I strongly recomend not to use the default PyYAML loader, as it sorts the dict entrys alphabetically, a fact that may break interpolation in some cases (see test/odict.yaml and test/test_odict.py for an example). See Ordered Dict Yaml Loader / Dumper aka. ODYLDo

default

The default jinja2.Environment for the interpolation is

hiyapyco.jinja2env = Environment(undefined=Undefined)

This means that undefined vars will be ignored and replaced with a empty string.

change the jinja2 Environment

If you like to change the jinja2 Environment used for the interpolation, set hiyapyco.jinja2env before calling hiyapyco.load!

use jinja2 DebugUndefined

If you like to keep the undefined var as string but raise no error, use

from jinja2 import Environment, Undefined, DebugUndefined, StrictUndefined
hiyapyco.jinja2env = Environment(undefined=DebugUndefined)

use jinja2 StrictUndefined

If you like to raise a error on undefined vars, use

from jinja2 import Environment, Undefined, DebugUndefined, StrictUndefined
hiyapyco.jinja2env = Environment(undefined=StrictUndefined)

This will raise a hiyapyco.HiYaPyCoImplementationException wrapped arround the jinja2.UndefinedError pointing at the string causing the error.

more informations

See: jinja2.Environment

cast interpolated strings

As you must use interpolation as strings (PyYAML will weep if you try to start a value with {{), you can set castinterpolated to True in order to try to get a best match cast for the interpolated values. The ``best match`` cast is currently only a q&d implementation and may not give you the expected results!

Ordered Dict Yaml Loader / Dumper aka. ODYLDo

This is a simple implementation of a PyYAML loader / dumper using OrderedDict from collections. Because chaos is fun but order matters on loading dicts from a yaml file.

Install

From Source

GitHub

https://github.com/zerwes/hiyapyco

git clone https://github.com/zerwes/hiyapyco
cd hiyapyco
sudo python setup.py install

PyPi

Download the latest or desired version of the source package from https://pypi.python.org/pypi/HiYaPyCo. Unpack the archive and install by executing:

sudo python setup.py install

pip

Install the latest wheel package using:

pip install HiYaPyCo

debian packages

install the latest debian packages from http://repo.zero-sys.net/hiyapyco:

# create the sources list file:
sudo echo "deb http://repo.zero-sys.net/hiyapyco/deb ./" > /etc/apt/sources.list.d/hiyapyco.list

# import the key:
gpg --keyserver keys.gnupg.net --recv-key 77DE7FB4
# or use:
wget https://repo.zero-sys.net/77DE7FB4.asc -O - | gpg --import -

# apt tasks:
gpg --armor --export 77DE7FB4 | sudo tee /etc/apt/trusted.gpg.d/hiyapyco.asc
sudo apt-get update
sudo apt-get install python3-hiyapyco

rpm packages

use http://repo.zero-sys.net/hiyapyco/rpm as URL for the yum repo and https://repo.zero-sys.net/77DE7FB4.asc as the URL for the key.

Arch Linux

An AUR package is available (provided by Pete Crighton and not always up to date).

License

This package is free software. This software is licensed under the terms of the GNU GENERAL PUBLIC LICENSE version 3 or later, as published by the Free Software Foundation. See https://www.gnu.org/licenses/gpl.html

Changelog

0.5.5

FIXED: #67 cosmetic changes

0.5.4

FIXED: #60 recursive calls to _substmerge

IMPROVED: testing and python support (3.11)

0.5.1

MERGED: #52 by ryanfaircloth

0.5.0

MERGED: #41 Jinja2 dependency increased to include Jinja2 3.x.x

REMOVED: Support for Python 2

0.4.16

MERGED: #37 alex-ber

0.4.15

MERGED: #30 lesiak:issue-30-utf

MERGED: #28 lesiak:issue-28

0.4.14

FIXED: issue #33

MERGED: issue #32

0.4.13

IMPLEMENTED: [issue #27] support multiple yaml documents in one file

0.4.12

FIXED: logging by Regev Golan

0.4.11

IMPLEMENTED: mergelists (see issue #25)

0.4.10

FIXED: issue #24 repo signing

0.4.9

FIXED: issue #23 loglevelonmissingfiles

0.4.8

Fixed pypi doc

0.4.7

Reverted: logger settings to initial state

Improved: dump

Merged:

flatten mapping from Chris Petersen [email protected]
arch linux package info from Peter Crighton [email protected]

0.4.6

MERGED: fixes from mmariani

0.4.5

FIXED: issues #9 and #11

0.4.4

deb packages:

removed support for python 2.6
include examples as doc

0.4.3

FIXED: issue #6 import of hiyapycoversion* in setup.py causes pip install failures*

0.4.2

Changed: moved to GPL

Improvements: missing files handling, doc

0.4.1

Implemented: castinterpolated

0.4.0

Implemented: loading yaml docs from string

0.3.2

Improved tests and bool args checks

0.3.0 / 0.3.1

Implemented a Ordered Dict Yaml Loader

0.2.0

Fixed unicode handling

0.1.0 / 0.1.1

Initial release

hiyapyco's People

Contributors

Stargazers

Watchers

hiyapyco's Issues

report which file failed to load @ event of a YAML parse error

logger should be used only if initialized

logger should be used only if initialized ...

how add !vault tag

Can't interpolate using Jinja2 template

I have following template
config.yml


---
# change profile and region to suit your needs
profile: default
region: us-east-1
env: sb-p1
domain: sb.com {{ env }}

qua_user: {{ env }} -abc (1)

qua_user2: {{ env }} (2)

With following code

conf = hiyapyco.load('config.yml', interpolate=True,usedefaultyamlloader=False)

(1) causes

---------------------------------------------------------------------------
ParserError                               Traceback (most recent call last)
<ipython-input-12-55b3992a6ce7> in <module>()
----> 1 conf = hiyapyco.load('config.yml', interpolate=True,usedefaultyamlloader=False)

/Users/iapilgrim/.virtualenvs/gordon/lib/python2.7/site-packages/hiyapyco/__init__.pyc in load(*args, **kwargs)
    409     Will mostly be a OrderedDict (dict if usedefaultyamlloader), but can be of any other type, depending on the yaml files.
    410     """
--> 411     hiyapyco = HiYaPyCo(*args, **kwargs)
    412     return hiyapyco.data()
    413

/Users/iapilgrim/.virtualenvs/gordon/lib/python2.7/site-packages/hiyapyco/__init__.pyc in __init__(self, *args, **kwargs)
    190                 ydata = yaml.safe_load(f)
    191             else:
--> 192                 ydata = odyldo.safe_load(f)
    193             if logger.isEnabledFor(logging.DEBUG):
    194                 logger.debug('yaml data: %s' % ydata)

/Users/iapilgrim/.virtualenvs/gordon/lib/python2.7/site-packages/hiyapyco/odyldo.pyc in safe_load(stream)
     68 def safe_load(stream):
     69     """implementation of safe loader using Ordered Dict Yaml Loader"""
---> 70     return yaml.load(stream, ODYL)
     71
     72 def safe_dump(data, stream=None, **kwds):

/Library/Python/2.7/site-packages/yaml/__init__.pyc in load(stream, Loader)
     69     loader = Loader(stream)
     70     try:
---> 71         return loader.get_single_data()
     72     finally:
     73         loader.dispose()

/Library/Python/2.7/site-packages/yaml/constructor.pyc in get_single_data(self)
     35     def get_single_data(self):
     36         # Ensure that the stream contains a single document and construct it.
---> 37         node = self.get_single_node()
     38         if node is not None:
     39             return self.construct_document(node)

/Library/Python/2.7/site-packages/yaml/composer.pyc in get_single_node(self)
     34         document = None
     35         if not self.check_event(StreamEndEvent):
---> 36             document = self.compose_document()
     37
     38         # Ensure that the stream contains no more documents.

/Library/Python/2.7/site-packages/yaml/composer.pyc in compose_document(self)
     53
     54         # Compose the root node.
---> 55         node = self.compose_node(None, None)
     56
     57         # Drop the DOCUMENT-END event.

/Library/Python/2.7/site-packages/yaml/composer.pyc in compose_node(self, parent, index)
     82             node = self.compose_sequence_node(anchor)
     83         elif self.check_event(MappingStartEvent):
---> 84             node = self.compose_mapping_node(anchor)
     85         self.ascend_resolver()
     86         return node

/Library/Python/2.7/site-packages/yaml/composer.pyc in compose_mapping_node(self, anchor)
    125         if anchor is not None:
    126             self.anchors[anchor] = node
--> 127         while not self.check_event(MappingEndEvent):
    128             #key_event = self.peek_event()
    129             item_key = self.compose_node(node, None)

/Library/Python/2.7/site-packages/yaml/parser.pyc in check_event(self, *choices)
     96         if self.current_event is None:
     97             if self.state:
---> 98                 self.current_event = self.state()
     99         if self.current_event is not None:
    100             if not choices:

/Library/Python/2.7/site-packages/yaml/parser.pyc in parse_block_mapping_key(self)
    437             token = self.peek_token()
    438             raise ParserError("while parsing a block mapping", self.marks[-1],
--> 439                     "expected <block end>, but found %r" % token.id, token.start_mark)
    440         token = self.get_token()
    441         event = MappingEndEvent(token.start_mark, token.end_mark)

ParserError: while parsing a block mapping
  in "sb/samples/biling/config.yml", line 3, column 1
expected <block end>, but found '<scalar>'
  in "sb/samples/biling/config.yml", line 8, column 21

and (2) causes

In [13]: conf = hiyapyco.load('config.yml', interpolate=True,usedefaultyamlloader=False)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-13-55b3992a6ce7> in <module>()
----> 1 conf = hiyapyco.load('config.yml', interpolate=True,usedefaultyamlloader=False)

/Users/iapilgrim/.virtualenvs/gordon/lib/python2.7/site-packages/hiyapyco/__init__.pyc in load(*args, **kwargs)
    409     Will mostly be a OrderedDict (dict if usedefaultyamlloader), but can be of any other type, depending on the yaml files.
    410     """
--> 411     hiyapyco = HiYaPyCo(*args, **kwargs)
    412     return hiyapyco.data()
    413

/Users/iapilgrim/.virtualenvs/gordon/lib/python2.7/site-packages/hiyapyco/__init__.pyc in __init__(self, *args, **kwargs)
    190                 ydata = yaml.safe_load(f)
    191             else:
--> 192                 ydata = odyldo.safe_load(f)
    193             if logger.isEnabledFor(logging.DEBUG):
    194                 logger.debug('yaml data: %s' % ydata)

/Users/iapilgrim/.virtualenvs/gordon/lib/python2.7/site-packages/hiyapyco/odyldo.pyc in safe_load(stream)
     68 def safe_load(stream):
     69     """implementation of safe loader using Ordered Dict Yaml Loader"""
---> 70     return yaml.load(stream, ODYL)
     71
     72 def safe_dump(data, stream=None, **kwds):

/Library/Python/2.7/site-packages/yaml/__init__.pyc in load(stream, Loader)
     69     loader = Loader(stream)
     70     try:
---> 71         return loader.get_single_data()
     72     finally:
     73         loader.dispose()

/Library/Python/2.7/site-packages/yaml/constructor.pyc in get_single_data(self)
     37         node = self.get_single_node()
     38         if node is not None:
---> 39             return self.construct_document(node)
     40         return None
     41

/Library/Python/2.7/site-packages/yaml/constructor.pyc in construct_document(self, node)
     46             self.state_generators = []
     47             for generator in state_generators:
---> 48                 for dummy in generator:
     49                     pass
     50         self.constructed_objects = {}

/Users/iapilgrim/.virtualenvs/gordon/lib/python2.7/site-packages/hiyapyco/odyldo.pyc in _odyload(self, node)
     47         data = OrderedDict()
     48         yield data
---> 49         data.update(self.construct_mapping(node))
     50
     51     # see pyyaml construct_mapping

/Users/iapilgrim/.virtualenvs/gordon/lib/python2.7/site-packages/hiyapyco/odyldo.pyc in construct_mapping(self, node, deep)
     54         m = OrderedDict()
     55         for k, v in node.value:
---> 56             m[self.construct_object(k, deep=deep)] = self.construct_object(v, deep=deep)
     57         return m
     58

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/collections.pyc in __setitem__(self, key, value, dict_setitem)
     61         # Setting a new item creates a new link at the end of the linked list,
     62         # and the inherited dictionary is updated with the new key/value pair.
---> 63         if key not in self:
     64             root = self.__root
     65             last = root[0]

TypeError: unhashable type: 'OrderedDict'

Note sure why

Push latest to PyPi

Any chance of pushing the latest code changes to pypi, current version there does not contain recent fixes.

fix pydoc

hey 2 me: pls fix the pydoc

ResourceWarning: unclosed file in init.py

Loading a file via hiyapyco results in a warning:

__init__.py:186: ResourceWarning: unclosed file <_io.TextIOWrapper name='PATH_TO_FILE.yaml' mode='r' encoding='cp1250'>
  f = open(fn, 'r')

The file is not explicitly closed.
Use python.exe -Wall if the warning is not enabled by default in your environment

Merge of empty dict and dict with entries doesn't work

# common.yaml
test:
  - entry

# node.yaml
test:

Traceback:

  File "./inventory/hiera.py", line 91, in get_inventory
    method=hiyapyco.METHOD_MERGE)
  File "build/bdist.linux-x86_64/egg/hiyapyco/__init__.py", line 410, in load
  File "build/bdist.linux-x86_64/egg/hiyapyco/__init__.py", line 200, in __init__
  File "build/bdist.linux-x86_64/egg/hiyapyco/__init__.py", line 347, in _deepmerge
  File "build/bdist.linux-x86_64/egg/hiyapyco/__init__.py", line 339, in _deepmerge
hiyapyco.HiYaPyCoImplementationException: can not merge <type 'NoneType'> to <type 'list'> (@ "['entry']"  try to merge "None")

racially loaded terms - rename default branch to main

consider using no inflammatory jargon

Versioned release

Please release a new build with the updated dependencies

release pr #42

make a new release including #42
in the worst case: drop python 2.7 support

Bracketless dump ?

Is there a way to dump() the yaml, but w/o brackets ... as if human has written it ...

Possible to change license to LGPL3?

IANAL but...

As it stands, any software that includes this library will be forced to be GPL3, which makes it incompatible with many standard python libraries (usually non-copyleft licenses like MIT or BSD), and certainly unusable by any company that might want to benefit from it.

LGPL would keep this specific code copyleft but allow the library to be linked by other code bases without compromising those licenses.

Unexpected behaviour with mapping nested in a sequence

#!/usr/bin/python

import hiyapyco

test = "\n"

for a in range(1, 3):
    number_a = a
    for b in range(1, 3):
        number_b = b
        new = hiyapyco.dump([{number_a: number_b}])
        merged = hiyapyco.load(test, new, method=hiyapyco.METHOD_MERGE)
        test = hiyapyco.dump(merged, default_flow_style=False)

print(test)

The above script gives the output:

- 1: 2
- 2: 1
- 2: 2

Whereas I would expect:

- 1: 2
- 2: 2

It seems that merging only works correctly for the first sequence (I’ve tested with more than two, all subsequent behave like the second).
Is this a bug in HiYaPyCo or in my expectation? If the latter, can my desired output be achieved in another way?

I use Python 3.5 on Arch Linux if that makes any difference.

deep merge with `failonmissingfiles=False` and missing file fails

#10 blocks deep merge with failonmissingfiles=False, because it still creates an empty dict for the missing file.

Traceback (most recent call last):
  File "./inventory/hiera.py", line 102, in <module>
    HieraInventory()
  File "./inventory/hiera.py", line 39, in __init__
    self.inventory = self.get_inventory(self.hiera_data)
  File "./inventory/hiera.py", line 83, in get_inventory
    method=hiyapyco.METHOD_MERGE)
  File "build/bdist.linux-x86_64/egg/hiyapyco/__init__.py", line 410, in load
  File "build/bdist.linux-x86_64/egg/hiyapyco/__init__.py", line 200, in __init__
  File "build/bdist.linux-x86_64/egg/hiyapyco/__init__.py", line 364, in _deepmerge
hiyapyco.HiYaPyCoImplementationException: can not merge <type 'NoneType'> to <class 'collections.OrderedDict'>....

Cannot import name 'soft_unicode' from 'markupsafe'

requirements.txt:
HiYaPyCo==0.4.16

The latest version of markupsafe has removed soft_unicode resulting in the following error:

ImportError while importing test module '/home/runner/work/secureCodeBox/secureCodeBox/scanners/zap-advanced/scanner/tests/test_zap_configuration.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/hostedtoolcache/Python/3.9.10/x[64](https://github.com/secureCodeBox/secureCodeBox/runs/5255972111?check_suite_focus=true#step:5:64)/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_zap_configuration.py:14: in <module>
    from zapclient.configuration.zap_configuration import ZapConfiguration
zapclient/__init__.py:12: in <module>
    from .zap_abstract_client import ZapClient
zapclient/zap_abstract_client.py:15: in <module>
    from .configuration import ZapConfiguration
zapclient/configuration/__init__.py:12: in <module>
    from .zap_configuration import ZapConfiguration
zapclient/configuration/zap_configuration.py:12: in <module>
    import hiyapyco
/opt/hostedtoolcache/Python/3.9.10/x64/lib/python3.9/site-packages/hiyapyco/__init__.py:30: in <module>
    from jinja2 import Environment, Undefined, DebugUndefined, StrictUndefined, TemplateError
/opt/hostedtoolcache/Python/3.9.10/x64/lib/python3.9/site-packages/jinja2/__init__.py:12: in <module>
    from .environment import Environment
/opt/hostedtoolcache/Python/3.9.10/x64/lib/python3.9/site-packages/jinja2/environment.py:25: in <module>
    from .defaults import BLOCK_END_STRING
/opt/hostedtoolcache/Python/3.9.10/x64/lib/python3.9/site-packages/jinja2/defaults.py:3: in <module>
    from .filters import FILTERS as DEFAULT_FILTERS  # noqa: F401
/opt/hostedtoolcache/Python/3.9.10/x64/lib/python3.9/site-packages/jinja2/filters.py:13: in <module>
    from markupsafe import soft_unicode
E   ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/opt/hostedtoolcache/Python/3.9.10/x64/lib/python3.9/site-packages/markupsafe/__init__.py)

pypi release fails: The description failed to render in the default format of reStructuredText.

400 The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information.

extend tests to debian 12

deb repo sign / auth issue ond debian 9 aka. stretch

W: GPG error: http://repo.zero-sys.net/hiyapyco/deb ./ Release: The following signatures were invalid: 9AE468B54DF2FB603A82528EC55A7EBEED7D414C
W: The repository 'http://repo.zero-sys.net/hiyapyco/deb ./ Release' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.

Distributor ID: Debian
Description: Debian GNU/Linux 9.0 (stretch)
Release: 9.0
Codename: stretch

bring python build process up to date

avoid calling setup.py directly
https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html

New Merge Method proposal

I've change your deep merge method to make another merge variant.
I don't want to list to be merge, I want them to be replaced.

My use case is the following: I have config.yml and config-dev.yml, etc (dev is profile). I want default configuration to be taken from config.yml and if I use dev profile appropriate configuration I want to be overridden by config-dev.yml. Spring Boot (this is Framework in Java) don't merge lists, it replaces them, so I want to mimic this behavior.

It will be nice, if you will incorporate this change in your code base. Thanks.

This is my variant of hiyapyco.HiYaPyCo.__deepmerge:

def __deepmerge(self, a, b):
    logger.debug('>' * 30)
    logger.debug('deepmerge %s and %s' % (a, b,))
    # FIXME: make None usage configurable
    if b is None:
        logger.debug('pass as b is None')
        pass

    #MYRPOPOSAL:, treat listTypes as primitiveTypes in merge
    #subsititues list, don't merge them

    if a is None or isinstance(b, primitiveTypes) or isinstance(b, listTypes):
        logger.debug('deepmerge: replace a "%s"  w/ b "%s"' % (a, b,))
        a = b

    elif isinstance(a, dict):
        if isinstance(b, dict):
            logger.debug('deepmerge: dict ... "%s" and "%s"' % (a, b,))
            for k in b:
                if k in a:
                    logger.debug('deepmerge dict: loop for key "%s": "%s" and "%s"' % (k, a[k], b[k],))
                    a[k] = self._deepmerge(a[k], b[k])
                else:
                    logger.debug('deepmerge dict: set key %s' % k)
                    a[k] = b[k]
        elif isinstance(b, listTypes):
            logger.debug('deepmerge: dict <- list ... "%s" <- "%s"' % (a, b,))
            for bd in b:
                if isinstance(bd, dict):
                    a = self._deepmerge(a, bd)
                else:
                    raise HiYaPyCoImplementationException(
                        'can not merge element from list of type %s to dict (@ "%s" try to merge "%s")' %
                        (type(b), a, b,)
                    )
        else:
            raise HiYaPyCoImplementationException(
                'can not merge %s to %s (@ "%s" try to merge "%s")' %
                (type(b), type(a), a, b,)
            )
    logger.debug('end deepmerge part: return: "%s"' % a)
    logger.debug('<' * 30)
    return a

import of hiyapyco version in setup.py causes pip install failures

Importing hiyapyco in setup.py causes pip install failures when PyYAML (and Jinja2) have not previously been installed in a virtualenv.

Collecting PyYAML==3.11 (from ...) Using cached PyYAML-3.11.tar.gz Collecting HiYaPyCo==0.4.1 (from ...) Using cached HiYaPyCo-0.4.1.tar.gz Complete output from command python setup.py egg_info: Traceback (most recent call last): File "", line 20, in File "/private/var/folders/f7/js66bsc95mq6qp9b178zvngr0000gq/T/pip-build-oyKfvh/HiYaPyCo/setup.py", line 10, in from hiyapyco import **version** as hiyapycoversion File "hiyapyco/**init**.py", line 24, in import yaml ImportError: No module named yaml

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/f7/js66bsc95mq6qp9b178zvngr0000gq/T/pip-build-oyKfvh/HiYaPyCo

Jinja2 dependency increased to include Jinja2 3.x.x

installrequires = ['PyYAML', 'Jinja2>1,<4',]

All tests pass. Updating the dependency allows HiYaPico to work with the latest release of Flask and Jinja.

what are the requirements for this package?

I get the error No module named 'testsetup'

Merge with anchors in a different file impossible

This example will not work:

import hiyapyco

a = '''

---
foo: &foo
    a: 1
    b: 2
'''

b = '''

---
bar: *foo
'''

hiyapyco.load(a,b, method=hiyapyco.METHOD_MERGE)

loglevelmissingfiles ignore keyword arg

loglevelmissingfiles ignore keyword args an always ERROR or WARNING dependense value for failonmissingfiles

optional: support python3-ruamel

remove deprecated distutils

hiyapyco/__init__.py:25:0: W4901: Deprecated module 'distutils.util' (deprecated-module)

caused by strtobool

https://peps.python.org/pep-0632/#migration-advice

so we can use a own implementation or use:

Incorrect utf-8 handling under Python3

When reading from a file in Python 3, default file encoding is platform dependent (whatever locale.getpreferredencoding() returns)
https://docs.python.org/3.7/library/functions.html#open

There is a test case provided test_interpolationunicode, but I believe it is too weak.
It only tests German characters, which may pass silently on machines with German locale.

On my machine with cp1250 locale, on Python 3, loaded yaml:

has German characters incorrectly interpreted
parsing fails with UnicodeDecodeError on East-Asian characters

Same setup works fine on Python2

Per instance loader and enviroment option

Currently can use the default loader or override the envinroment for everything using hiyapico. Allow providing different ones per instance.

class HiYaPyCo:
    def __init__(self, *args, loader_cls:type[yaml.Loader] = None, environment:jinja2.Environment=None, **kwargs):

hiyapyco.load(*paths, , loader_cls:type[yaml.Loader] = CustomLoader, environment=myenv)

If None is provided use the default ones if not use the ones provided.

How do you specify jinja variables in yaml files for interpolation?

Could you please provide an example how this works? I searched for one but couldn't find it.

What if I just want to use interpolation and don't have any yaml stuff to merge, just to place jinja placeholders in yaml values?
Wouldn't it be better to de-couple this? As it is now I would have to create the object perhaps set some attributes, and then call the internal _interpolate method.

Merging arrays of dicts?

Hi, I had some yaml files with an array of dicts in it, and I'm confused how hiyapyco is supposed to behave in this scenario. Not sure if it's a bug or not.

base.yml

key_one: 1
key_two: 2
array_of_dicts:
- dict_key_one: a
  dict_key_two: b
  dict_key_three: c
- dict_key_one: a1
  dict_key_two: b1
  dict_key_three: c1

layer.yml

key_two: 2222
array_of_dicts:
- dict_key_one: a2
  dict_key_two: b2
  dict_key_three: c2

merge.py

import hiyapyco

CONF = hiyapyco.load("base.yml", "layer.yml", method=hiyapyco.METHOD_MERGE)
print hiyapyco.dump(CONF)

result:

key_one: 1
key_two: 2222
array_of_dicts:
- {dict_key_one: a2, dict_key_two: b2, dict_key_three: c2}
- {dict_key_one: a1, dict_key_two: b1, dict_key_three: c1}

I thought it would have produced:

key_one: 1
key_two: 2222
array_of_dicts:
- {dict_key_one: a, dict_key_two: b, dict_key_three: c}
- {dict_key_one: a1, dict_key_two: b1, dict_key_three: c1}
- {dict_key_one: a2, dict_key_two: b2, dict_key_three: c2}

Is this a bug?

ansible !vault variable not load

if parse ansible-vault variable

my_encrypted_var: !vault |
          $ANSIBLE_VAULT;1.2;AES256;dev
          30613233633461343837653833666333643061636561303338373661313838333565653635353162
          3263363434623733343538653462613064333634333464660a663633623939393439316636633863
          61636237636537333938306331383339353265363239643939666639386530626330633337633833
          6664656334373166630a363736393262666465663432613932613036303963343263623137386239
          6330

yaml.constructor.ConstructoError: Could not determine a constructor for the tag "!vault"

missing yaml file in the middle of the loaded file list throws an error

I looked into the issue I had a bit more.
What (seems to) happen is:
0. Try to load a list of files, the middle one is missing (settings - interpolate false, failonmissingfiles = false)

First file is loaded - f variable (Line 170) is a file, and is also read - at end
Second file is missing, but in the except IOError as e, the f variable is not set, and it remains at the previous file value
yaml.safe_load / odyldo.safe_load just reads the file, which is at end, so the result is None
self._data is not None, so the code continues, but merging throws an error

Remove logging configurations code (or opt-in)

Thank you for this super useful project.

In general, libraries should not configure logging themselves[1][2], but leave that to the driving code using them; otherwise they may overwrite pre-existing configuration.
That way issues like #9 would not even exist.

Personally I prefer not even to add a NullHandler suggested by the docs, and let the warning about "uninitialized loggers" notify client-developers that the lib is using logging which needs their attention.

In addition, the logger-name should not be configurable - if needed, client-devs can always monkeypatch module's logger variable.
The log-levels for various actions (loglevelmissingfiles) may also be left to be configured in the same way, i.e. by using 2 separate loggers, but it is not that important (no harm to the clients done).

Kind regards,
Kostis

[1] https://docs.python.org/3/howto/logging.html#configuring-logging-for-a-library
[2] http://docs.python-guide.org/en/latest/writing/logging/#logging-in-a-library

Wrong Recursive call on method _substmerge

Inside method _substmerge (line 343, hiyapyco/init.py), the recursive call is wrongly written as self._deepmerge (line 374, hiyapyco/init.py). It's a quick fix plaguing this superb library.

Please fix. It's a quick one.

is there a way to use ruamel YAML isntead of pyyaml?

when loading files that have aliases and anchors, because of PyYaml the aliases get renamed to ie id001

conf = hiyapyco.load('/tmp/yaml1.yaml', '/tmp/yaml2.yaml','/tmp/yaml3.yml','/tmp/yaml4.yml', '/tmp/yaml5.y
     ...: ml','/tmp/yaml6.yml', method=hiyapyco.METHOD_MERGE, interpolate=True, failonmissingfiles=True)

here's the yaml6 file - the one with the anchors/aliases

cat yaml6.yml 
definitions: 
  steps:
    - step: &build-test
        name: Build and test
        script:
          - mvn package
        artifacts:
          - target/**

pipelines:
  branches:
    develop:
      - step: *build-test
    master:
      - step: *build-test

here's a partial of the output

  nr2: 6
key:
- value line 1
- value line 2
- value line 3
definitions:
  steps:
  - step: &id001
      name: Build and test
      script:
      - mvn package
      artifacts:
      - target/**
pipelines:
  branches:
    develop:
    - step: *id001
    master:
    - step: *id001

ruamel yamle has addressed and fixed this issue already, is there a overwrite for this? I don't see a flag of sorts in the ini.py

List Items order matters?

Hello there,
first of all, thank you for your availability. I've recently played with hiyapyco and I discovered a strange behavior using nested lists inside YAML files. The goal is to overwrite the list values with the latest ones.

Issue
I ran the snippet below and hiyapyco detects different values and merges all of them. Here below the snippet to replicate the issue:

import hiyapyco
base = """deploy:
  strategy: RollingUpdate
  maxSurge: 2
  maxUnavailable: 1
  env:
    - CORECLR_ENABLE_PROFILING: 0
    - DD_ENV: development
"""
shared = """deploy:
  strategy: RollingUpdate
  maxSurge: 5
  maxUnavailable: 1
  env:
    - CORECLR_ENABLE_PROFILING: 0
    - DD_ENV: stage
"""
develop = """deploy:
  strategy: RollingUpdate
  maxSurge: 2
  maxUnavailable: 1
  env:
    - DD_ENV: production
    - CORECLR_ENABLE_PROFILING: 1
"""
merged_yaml = hiyapyco.load([base, shared, develop], method=hiyapyco.METHOD_MERGE, interpolate=False, failonmissingfiles=True)
print(hiyapyco.dump(merged_yaml))

Test 1 Result

>>> python3 test-merge2.py
deploy:
  strategy: RollingUpdate
  maxSurge: 2
  maxUnavailable: 1
  env:
  - CORECLR_ENABLE_PROFILING: 0
  - DD_ENV: stage
  - DD_ENV: production
  - CORECLR_ENABLE_PROFILING: 1

Workaround
I changed the order of the list items and used the same order in the list in each file (f.i. in develop, 1st CORECLR_ENABLE_PROFILING: 1, 2nd DD_ENV: production), below is the result:

Test 2 Result

>>> python3 test-merge2.py
deploy:
  strategy: RollingUpdate
  maxSurge: 2
  maxUnavailable: 1
  env:
  - CORECLR_ENABLE_PROFILING: 1
  - DD_ENV: production

Is it should work like that? Do the list items have to be in the same position in each file?

Many Thanks,
Fabrizio

update outdated dependencies

update somehow outdated dependencies:

Jinja2 <3,>1
MarkupSafe <2

github actions

after removing the travis ci testing, a new github action should be implemented as a replacement ...

RFE: Support multiple documents in the same file

Currently, only one document per file is supported.

Consider this simple example:
In c1.yaml:

v: 1
---
v: 2

Actual behavior

Running python -c "import hiyapyco; conf = hiyapyco.load('c1.yaml')" will result in:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/thofmann/.local/lib/python2.7/site-packages/hiyapyco/__init__.py", line 426, in load
    hiyapyco = HiYaPyCo(*args, **kwargs)
  File "/home/thofmann/.local/lib/python2.7/site-packages/hiyapyco/__init__.py", line 205, in __init__
    ydata = odyldo.safe_load(f)
  File "/home/thofmann/.local/lib/python2.7/site-packages/hiyapyco/odyldo.py", line 70, in safe_load
    return yaml.load(stream, ODYL)
  File "/usr/lib64/python2.7/site-packages/yaml/__init__.py", line 71, in load
    return loader.get_single_data()
  File "/usr/lib64/python2.7/site-packages/yaml/constructor.py", line 37, in get_single_data
    node = self.get_single_node()
  File "/usr/lib64/python2.7/site-packages/yaml/composer.py", line 43, in get_single_node
    event.start_mark)
yaml.composer.ComposerError: expected a single document in the stream
  in "/home/thofmann/test/hiyapyco/c1.yaml", line 1, column 1
but found another document
  in "/home/thofmann/test/hiyapyco/c1.yaml", line 2, column 1

Expected behavior

I expected the same behavior as if those documents were in separate files.

Interpolation without converting type to string

First, thanks for the cool library. This is exactly what I was looking for for one of my projects.

So one of the things I want to do in my file is to have a configuration that looks like this:


---
one: 1
two: 2
three: {{ one + two }}  # Doesn't work
four: "{{ two + two }}" # Works but forced into string

But the line containing "three" throws an exception that reads "OrderedDict not hashable".

In Python:

print(conf['one'] + conf['two']) # Returns 3 as expected
print(conf['one'] + conf['four']) # TypeError: unsupported operand type(s) for +: 'int' and 'str'

In your tests folder, the values are always coerced into strings. Is this a limitation of Jinja2/PyYaml? It would be great if this could be fixed, so that I get an integer/float as an interpolation of integer/floats.

Looking at the code, it seems like in HiYaPyCo._interpolate() it might make sense to assume that the value is a string if it is not any of the other types? (What are the cases where this could break?)
If you agree with this approach, I could send a pull request with the change.

It's time to update the year!

It's about time to repeat this commit, but one more year.

How to include/reference complex data structures/objects?

This is a great tool but it seems to lack (at least documentation about) a way to include/interpolate nested structures, e.g. I have some default values (in my case, AWS Cloudformation Tag definitions) that I need insert a bunch of times at different nesting levels. A more simple example might look something like this:

defaultprefs.yaml

pref1: 1
pref2: 2

stream.yaml

user1:
    prefs: {% include 'defaultprefs.yaml' %}
user2:
    prefs: {% include 'defaultprefs.yaml' %}

---
user1:
    prefs:
        pref2: override2
user2:
    prefs:
        pref3: 3

With a deep-merge output of:

user1:
    prefs:
        pref1: 1
        pref2: override2
user2:
    prefs:
        pref1: 1
        pref2: 2
        pref3: 3

Obviously, the order that hiyapyco interpolates the jinja2 doesn't let this work (I'd have to wrap the { character in quotes to at least make the yaml parseable) but it gives an idea of the functionality. A more realistic solution would be similar to the !include call described here.

I assume it's possible to override/extend the ODYLDo but it would be nice if it could do this on its own.

Internal YAML loader does not support merge `<<` tag. (usedefaultyamlloader = False)

This example shows the issue:

Yaml

%YAML 1.2
%TAG !file! merge.yml

---
foo: &foo
    a: 1
    b: 2
bar:
    <<: *foo
    b: 3

Use-case

>>> hiyapyco.load('merge.yml', usedefaultyamlloader=True) # Does work
{'bar': {'a': 1, 'b': 3}, 'foo': {'a': 1, 'b': 2}}

>>> hiyapyco.load('merge.yml', usedefaultyamlloader=False) # Does not work
---------------------------------------------------------------------------
ConstructorError                          Traceback (most recent call last)
...

Trace

/usr/lib/python2.7/site-packages/HiYaPyCo-0.4.4-py2.7.egg/hiyapyco/odyldo.pyc in _odyload(self, node)
     47         data = OrderedDict()
     48         yield data
---> 49         data.update(self.construct_mapping(node))
     50
     51     # see pyyaml construct_mapping

/usr/lib/python2.7/site-packages/HiYaPyCo-0.4.4-py2.7.egg/hiyapyco/odyldo.pyc in construct_mapping(self, node, deep)
     53         m = OrderedDict()
     54         for k, v in node.value:
---> 55             m[self.construct_object(k, deep=deep)] = self.construct_object(v, deep=deep)
     56         return m
     57

/usr/lib/python2.7/site-packages/yaml/constructor.pyc in construct_object(self, node, deep)
     86                     constructor = self.__class__.construct_mapping
     87         if tag_suffix is None:
---> 88             data = constructor(self, node)
     89         else:
     90             data = constructor(self, tag_suffix, node)

/usr/lib/python2.7/site-packages/yaml/constructor.pyc in construct_undefined(self, node)
    412         raise ConstructorError(None, None,
    413                 "could not determine a constructor for the tag %r" % node.tag.encode('utf-8'),
--> 414                 node.start_mark)
    415
    416 SafeConstructor.add_constructor(

ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:merge'

Weird merge behavior with anchors

Hi,

I am facing a weird behavior when merging while using anchors.
Without anchors, this isn't happening, but I can't understand why this is happening and how to fix it.

hiyapyco.load('''
app-specs: &app-specs
  resources:
    requests:
      memory: "1"
      cpu: "1"

applications:
  app1:
    <<: *app-specs
    imageTag: azerty
  app2:
    <<: *app-specs
    imageTag: azerty
    replicas: 1
''', 
'''
applications:
  app1:
    imageTag: azerty
    replicas: 1
    resources:
      requests:
        memory: "2"
        cpu: "2"
  app2:
    imageTag: azerty
    replicas: 1
''', method=hiyapyco.METHOD_MERGE, mergelists=True, interpolate=True, failonmissingfiles=True)

Here is the merged dict, I expected to only have app1 with overrided resources, but both are :

{
  "app-specs": {
    "resources": {
      "requests": {
        "memory": "2",
        "cpu": "2"
      }
    }
  },
  "applications": {
    "app1": {
      "resources": {
        "requests": {
          "memory": "2",
          "cpu": "2"
        }
      },
      "imageTag": "azerty",
      "replicas": 1
    },
    "app2": {
      "resources": {
        "requests": {
          "memory": "2",
          "cpu": "2"
        }
      },
      "imageTag": "azerty",
      "replicas": 1
    }
  }
}

Here is the yaml dump :

app-specs:
  resources: &id001
    requests:
      memory: '2'
      cpu: '2'
applications:
  app1:
    resources: *id001
    imageTag: azerty
    replicas: 1
  app2:
    resources: *id001
    imageTag: azerty
    replicas: 1