grokzen / pykwalify Goto Github PK

View Code? Open in Web Editor NEW

289.0 289.0 86.0 901 KB

Python YAML/JSON schema validation library

License: MIT License

Makefile 0.74% Python 99.26%

pykwalify's Introduction

Hi there 👋

Github Statistics

pykwalify's People

Contributors

Stargazers

Watchers

pykwalify's Issues

length constraint passes for empty string

Let's say I have a schema defined as follows:

type: map
mapping:
    streams:
        type: seq
        required: True
        sequence:
            -   type: map
                mapping:
                    name:
                        type: str
                        length:
                            min: 1
                        required: True
                    sampleRateMultiple:
                        type: int
                        required: True

Given the following data:

streams:
  - name:
    sampleRateMultiple: 1

  - name: media
    sampleRateMultiple: 2

I would expect the constraint on the 'name' field would fail since it does not have an associated value. With pykwalify this validates successfully. If I fill in a name (with at least one character) the 'length' constraint will work correctly. So it seems a 'null' name field causes some issue.

Value 'None' is not of type 'str'

With this YAML:


---
foo: # This field is empty and despite YAML parser sees it as a None value, I am expecting an empty string

And this schema:

type: map
mapping: 
  foo:
    type: str

I get the error "- Value 'None' is not of type 'str'". However, a string can be empty.

A key allownone should be supported for this particular case.

Un-fix the versions of docopt, PyYAML, python-dateutil

Currently the versions are fixed, which is giving me errors on Mac OS X when I have python-dateutil==2.5.2.

I don't believe there's actually any problems with PyYAML and python-dateutil==2.5.2, however.

issues with float, None and scientific notation

Hi!

I'm trying validate some json I have. The json contains a mapping with float values. pykwalify probably correctly pykwalify doesn't accept float values like 1e-06 and None:

pykwalify.errors.SchemaError: <SchemaError: error code 2: Schema validation failed:
 - Value '1e-06' is not of type 'float'. Path: '/moresane_accuracy'.
 - Value 'None' is not of type 'float'. Path: '/sefd'.

So 2 questions (or maybe issues):

isn't 1e06 just valid syntax? Python eats it as float:

>>> x = float(1e-06)
>>> type(x)
<type 'float'>

Is it possible to give multiple types for a specified key in a mapping? I just can't figure it out. My none problem would be solved if i was able to set the type to 'float' or 'none'.

awesome, thanks!

pykwalify under Python 2.7.X?

I know your README stated that Python 2.7.x is not supported (and never will be) but it's not necessary clear this is indeed true. I removed the Python version check in runtest.py and ran your unit tests. Surprisingly, under Python 2.7.3 all of these tests passed! Is this expected or are you aware of scenarios where you explicitly used Python 3.X.Y features that have no hope of working under Python 2.7.X?

Obviously, I'm interested in getting pykwalify to work under Python 2.7.x. It would appear that it already works (at least the unit-tests pass) or only requires minor adjustments. I'm just trying to scope out the effort necessary to fork your project and create a variant that is Python 2.7.x compatible. Looks like you've actually done the work but arbritrarily decided to only claim 3.X.Y support moving forward (maybe to keep the door open for future Python 3.X.Y-specific enhancements).

Any feedback would be appreciated.

Glenn

Failing with Unicode chars.

If i have the schema:

  type: map
  mapping:
    msg:
      type: str

and the data file:

msg:
    "Alô do Brasil!!"

i get this

Traceback (most recent call last):
  File "/home/user/workspace/.virtualenvs/project/bin/pykwalify", line 9, in <module>
    load_entry_point('pykwalify==1.3.0', 'console_scripts', 'pykwalify')()
  File "/home/user/workspace/.virtualenvs/project/local/lib/python2.7/site-packages/pykwalify/cli.py", line 84, in cli_entrypoint
    run(parse_cli())
  File "/home/user/workspace/.virtualenvs/project/local/lib/python2.7/site-packages/pykwalify/cli.py", line 71, in run
    c.validate()
  File "/home/user/workspace/.virtualenvs/project/local/lib/python2.7/site-packages/pykwalify/core.py", line 144, in validate
    errors = self._start_validate(self.source)
  File "/home/user/workspace/.virtualenvs/project/local/lib/python2.7/site-packages/pykwalify/core.py", line 189, in _start_validate
    self._validate(value, root_rule, path, errors, done)
  File "/home/user/workspace/.virtualenvs/project/local/lib/python2.7/site-packages/pykwalify/core.py", line 209, in _validate
    self._validate_mapping(value, rule, path, errors, done=None)
  File "/home/user/workspace/.virtualenvs/project/local/lib/python2.7/site-packages/pykwalify/core.py", line 522, in _validate_mapping
    self._validate(v, r, "{}/{}".format(path, k), errors, done)
  File "/home/user/workspace/.virtualenvs/project/local/lib/python2.7/site-packages/pykwalify/core.py", line 207, in _validate
    self._validate_sequence(value, rule, path, errors, done=None)
  File "/home/user/workspace/.virtualenvs/project/local/lib/python2.7/site-packages/pykwalify/core.py", line 302, in _validate_sequence
    self._validate(item, r, "{}/{}".format(path, i), tmp_errors, done)
  File "/home/user/workspace/.virtualenvs/project/local/lib/python2.7/site-packages/pykwalify/core.py", line 205, in _validate
    self._validate_include(value, rule, path, errors, done=None)
  File "/home/user/workspace/.virtualenvs/project/local/lib/python2.7/site-packages/pykwalify/core.py", line 265, in _validate_include
    self._validate(value, partial_schema_rule, path, errors, done)
  File "/home/user/workspace/.virtualenvs/project/local/lib/python2.7/site-packages/pykwalify/core.py", line 209, in _validate
    self._validate_mapping(value, rule, path, errors, done=None)
  File "/home/user/workspace/.virtualenvs/project/local/lib/python2.7/site-packages/pykwalify/core.py", line 469, in _validate_mapping
    log.debug(" + rr: {} {}".format(k, v))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 13-14: ordinal not in range(128)

Any thoughts?

[]'s

Allowempty permits non-map values

Observed:

Providing 'allowempty' with type: 'map' validates non-map values.

Expected:

Based on documentation:

If True, the map can have keys which are not present in the schema, and these can map to anything.
Any keys which are specified in the schema must have values which conform to their corresponding constraints, if
they are present.

I would expect 'allowempty' to only permit:

an empty map
map with keys not in the schema

Otherwise, I would expect a NotMappingError

Example:

In [1]: pykwalify.core.Core(
    source_data=['not', 'a', 'map'],
    schema_data={'type': 'map', 'allowempty': True}
).validate(raise_exception=True)
Out[1]: ['not', 'a', 'map']

In [2]: pykwalify.core.Core(
    source_data=['not', 'maps'],
    schema_data={
        'type': 'seq',
        'sequence': [{
            'type': 'map',
            'mapping': {'a': {'type': 'str'}},
            'allowempty': True
        }]
    }
).validate(raise_exception=True)
Out[2]: ['not', 'maps']

Allow key to be specified or left empty

Is this feature present in the current release?

I am using a single schema file to validate against different scenarios. Some of the scenarios have a specific key and some don't. I would like to be able to specify in the schema to ignore if the key isn't present.

Implement partial schemas

I like the idea and here is how i think it should be implemented

schema: can be the new keyword for this but because i wont use any native YAML abilities i have to add a some new keyword inside schema: like schema-name: to enable name tagging.
schema: can only be specified at top level of a file for easy pre parsing of all partials before any other rule is parsed.
I dunno yet how to fix partial inside partial because that can lead to infinite recursion problems if done wrong. I have to work abit on this to get it right.
schema can easily be parsed as a Rule() so not much new has to be done there.
A new keyword have to be created for when to include a partial. An easy one would be "include-schema" and that can only be used at the same place/level where the keyword "type" can be used today. Will see how that works but it should work out i think.
It should be possible to specify more then 1 schema file via cli via multiple "-s schema.yaml -s schema2.yaml ..." to give the option to have a file with predefined partials allready.
Overriding schema partials should not be possible, if defined in 2 files then error will be thrown.
I do not think that currently it should be possible to mix a partial with some other defined rule, for example a partial cannot be a map and that should be merged with another map just to extend the ammount of keys that is possible to validate. This would be a future improvment but not in the first itteration because there is some other things that i have to change in the validation process for that to work good.

I do not think that i will make an option to directly include another file from inside a schema like this

type: map
required: True
mapping:
  polygon:
    type: seq
    sequence:
      - include-schema:
          file: /foo/bar.yaml

Maybe in a second itteration of the feature but not in the first one.

Yes both ways of including another file will be supported, via cli -s and as a argument to Core() as a list of multiple source_files that is parsed in sequence.

One problem that i thought of tho is that schema: as a keyword will not work because then it will only be possible to have one schema: key per file. The two solutions is to first have a list that then contains all partials but that will make it harder to integrate them into the same file as the main schema that wants to include the partial. The second solution that i am thinking of is to use the same feature as regex; and do "schema;schema-id:" because then it is possible to have any number of schema partials in one file. Then you could do

schema;seq:
  type: seq
  sequence:
    - type: int

schema;map
  type: map
  mapping:
    x:
      type: int

type: map
mapping:
  polygon:
    include: map
  square:
    include: seq

And with this setup then it could be possible to have all defenitions in the same file or you could split-up all partials into one file and the main schema in a separate file.

Port docs to readthedocs site

All the docs should be ported and moved to a readthedocs site.

Recursive schema definition fails

I want to define a tree-like schema:

type: map
mapping: &nodeid
  name:
    type: str
  edges:
    type: seq
    sequence:
      - type: map
        mapping: *nodeid

for something like this:

name: 'Top'
edges:
  -
    name: '2nd'
    edges:
      -
        name: '3rd'

Then i get this:

  File "/home/user/workspace/.virtualenvs/pykwalify/pykwalify/pykwalify/rule.py", line 130, in init
    func_mapping[k](v, rule, path)
  File "/home/user/workspace/.virtualenvs/pykwalify/pykwalify/pykwalify/rule.py", line 405, in init_mapping_value
    rule.init(v, "{}/mapping/{}".format(path, k))
  File "/home/user/workspace/.virtualenvs/pykwalify/pykwalify/pykwalify/rule.py", line 102, in init
    self.init_type_value(t, rule, path)
  File "/home/user/workspace/.virtualenvs/pykwalify/pykwalify/pykwalify/rule.py", line 177, in init_type_value
    log.debug("Type: {} {}".format(v, rule))
  File "/home/user/workspace/.virtualenvs/pykwalify/pykwalify/pykwalify/rule.py", line 65, in __str__
    return "Rule: {}".format(str(self._schema_str))
RuntimeError: maximum recursion depth exceeded while getting the repr of a list

Constraint on the length of a sequence?

Would it be possible to allow the 'range' or 'length' directive be applied to a sequence type? This would allow the schema to enforce a min/max sequence length. Some of my schema's contain a sequence with a minimum (or maximum) number of elements. Allowing either range/length to be extended to sequences would be helpful enforcing this constraint instead of relying on a secondary check after the input source has been validated by pykwalify. Is this do-able?

Idea: Show line number in source on error

Is there a way to show line numbers (as kwalify does) and suppress traceback when validating using the CLI?

unique not enforced on mappings

I may be missing the semantics of unique when used within a mapping.

For example...

Schema

type: map
mapping:
  type:
    type: str
    required: True
    unique: True
  user:
      type: str
      required: True
      unique: True
  password:
      type: str
      unique: True
      required: True

data file:

type: password
user: user1
password: pass1
user: user2
password: pass2
type: whatever

py code:

import pykwalify.core as core

c = core.Core(source_file='./data.yaml',
              schema_files=['./schema.yaml'])
c.validate(raise_exception=True)

No validation errors... How can I constrain the map keys to only being used once?

Quotes are required to validate datetime type

Given the schema:

type: map
mapping:
  d1:
    type: timestamp

and data:

d1: 2015-03-29

The following error is returned:

Value '2015-03-29' is not of type 'timestamp'. Path: '/d1', Not a valid timestamp

However, enclosing the date in quotes works:

d1: "2015-03-29"

Expectation: Quotes should not be required for the value to be recognized as a valid datetime type.

Unknown scalar type checking

Data:

- foo
- bar

with scheme:

type: seq
sequence:
  - type: scalar

returns:

pykwalify.errors.CoreError: <CoreError: error code 3: Unknown type check: /0 : foo : scalar: Path: '/'>

Pattern matching in a sequence?

Is it not possible to check a pattern against a sequence?

data:
  deploy_args:
    allowed_address_pairs:
      - 10.10.10.10
schema:
  type: map
  mapping:
    deploy_args:
      required: true
      type: map
      mapping:
        allowed_address_pairs:
          type: seq
          required: true
          pattern: ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$
          sequence:
            - type: str

error in usage

the pykwalify usage is in conflict with the documentation on readthedocs
usage:
-d FILE, --data-file FILE the file to be tested
-s FILE, --schema-file FILE schema definition file

readthedocs:
pykwalify -d data.yaml -s schema.yaml

Documentation not available on readthedocs

The hyperlink on https://github.com/grokzen/pykwalify:

All documentation can be found at http://pykwalify.readthedocs.org/en/latest/
This readme contains a reduced version of the full documentation.

Does not work. It shows me an error message:

This is an autogenerated index file.

Please create a /home/docs/checkouts/readthedocs.org/user_builds/pykwalify/checkouts/latest/docs/index.rst or /home/docs/checkouts/readthedocs.org/user_builds/pykwalify/checkouts/latest/docs/README.rst file with your own content.

A correction to this issue is pending here #74

AttributeError when evaluating timestamp

Given the following source:

my_date: 2013-05-23

... and schema:

mapping:
  "my_date":
    type: timestamp
    required: yes

I get an AttributeError on datetime.date (is it expecting a string?):

Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/pykwalify", line 9, in <module>
    load_entry_point('pykwalify==1.5.0', 'console_scripts', 'pykwalify')()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pykwalify/cli.py", line 84, in cli_entrypoint
    run(parse_cli())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pykwalify/cli.py", line 71, in run
    c.validate()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pykwalify/core.py", line 145, in validate
    errors = self._start_validate(self.source)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pykwalify/core.py", line 190, in _start_validate
    self._validate(value, root_rule, path, errors, done)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pykwalify/core.py", line 210, in _validate
    self._validate_mapping(value, rule, path, errors, done=None)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pykwalify/core.py", line 524, in _validate_mapping
    self._validate(v, r, u"{}/{}".format(path, k), errors, done)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pykwalify/core.py", line 212, in _validate
    self._validate_scalar(value, rule, path, errors, done=None)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pykwalify/core.py", line 589, in _validate_scalar
    v = value.strip()
AttributeError: 'datetime.date' object has no attribute 'strip'

How to associate a schema to a YAML data?

In XML you can specify your XSD with a XML xsi:schemaLocation attribute. YAML specs lack with kind of information.

So in my opinion two options are possible:

Using a global tag:

%YAML 1.2
%SCHEMA foo.schema.yml

---
some: data
...

Having a special key in the YAML description

%YAML 1.2

---
some: data
<schema>:
    location: 'foo.schema.yml'
...

The second method allows to embed the schema inside the YAML description, this could be nice. Also, it allows to partially apply a schema to a particular node:

%YAML 1.2

---
foo:
   bar:
      - list
<schema>:
    foo:
      bar: 
         <location>: 'foo.schema.yml'   
...

I don't know which option would be the best...

Any idea?

Regex Matching Issues

Hi,

I have the following example YAML file:

foo:

and the following schema file:

type:           map
matching-rule:  "any"
mapping:
    regex;(^[a-zA-Z]TEST[a-zA-Z\d]*$):
        type:   map
        mapping:
            "bar":
                type:       str

Somehow the regex matches the key although it shouldn't. Am I missing something?
I am using pykwalify (14.08) from PyPi.

Based on your test file 17s.yaml I also tried the following. Data file:

notmic:
- input
  foo
- output
  bar

Schema file:

type: map
matching-rule: "any"
mapping:
  regex;(^mi.+$):
    type: seq
    sequence:
      - type: str

Again the file matches the schema, despite the "not" prefix to "mic".

Kind regards,

Replace PyYAML by ruamel.yaml

Wouldn't it be a good idea to drop PyYAML and replace it by the actively developed ruamel.yaml? It support YAML 1.2 and is a fork of the original PyYAML. It looks like it is a drop-in replacement (at least it was in my personal project 😄 ).

I'll be happy to submit a pull request.

The "any" type does not accept non-implemented types like timestamp

The documentation specifies that any accepts any implemented type.

Is there a reason behind throwing a validation error when one uses other non-implementd types like timestamps?

I understand that a possible fix in my case is to send a PR to add timestamps support - and I'll do that if I get around to it :-)

Thanks,

Can't mix normal keys and regex keys?

I'd like to validate this data, which is part of the OpenControl project:

name:
  FRIST-800-53
AU-1:
  family: AU
  name: Audit and Accountability Policy and Procedures
AU-2:
  family: AU
  name: Audit Events
AU-2 (3):
  family: AU
  name: Audit Events | Reviews and Updates

If I try and use this schema:

type: map
mapping:
  name:
    type: str

  regex;/[A-Z]-/:
    type: map
    mapping:
      name:
        type: str
      family:
        type: str

Then it won't match, giving the error - Key 'name' does not match any regex '[A-Z]-'. Path: ''.: Path: '/'>. You can work around this by matching with regex;/^name$/:, but it's not clear in which order the regexes would be checked if several of them matched.

Create different implementations/generations of kwalify syntax

Currently there is some problems with that the syntax definition just grows as more and more features is added on top of the original kwalify syntax.

The long term suggestion is to make 2 different syntax schemas where one is a strict version that just implements the syntax that the original kwalify supported. Then there should be a newer and more modern version of the syntax that has all the backwards compatible kwalify syntax but also includes the new rules and syntax.

Suggested terms could be strict for the original kwalify and modern or standard for the second iteration.

The suggestion is to make the modern variant the default so that no changes will have to be imposed on people who currently uses all the new features. Then there will be added an option to use the strict version for people that requires it.

This is a long term plan and will probably make/bump the version to 2.0.0 when implemented.

Conditional validation?

Let's say we have this contrived example YAML:

foods:
  first_food:
    classification: fruit
    properties:
      name: apple
      color: red
      other: properties_go_here
  second_food:
    classification: vegetable
    properties:
      name: carrot
      color: orange
      other: properties_go_here

Presume that we can't change the format. In this case, each food gets an arbitrary (key) name, and the classification of each food must be kept separate from its properties.

I'm trying to find a way to conditionally validate the name property of a food based on its classification. In this example, if the classification is "fruit" then I want to validate that food's name property based on a list of fruits. If the classification is "vegetable" then I want to validate that food's name property based on a list of vegetables.

I started to imagine a schema to help with this:

mapping:
  "foods":
    mapping:
      regex;(.+):
        allowempty: True
        mapping:
          "classification": { type: str, pattern: fruit }
          "properties":
            allowempty: True
            mapping:
              "name": { type: str, pattern: (apple|orange|grape) }
      regex;(.+):
        allowempty: True
        mapping: 
          "classification": { type: str, pattern: vegetable }
          "properties":
            allowempty: True
            mapping:
              "name": { type: str, pattern: (carrot|broccoli|cucumber) }

Only this doesn't quite work:

 ERROR - validation.invalid
 ERROR -  --- All found errors ---
 ERROR - [u"Value 'apple' does not match pattern '(carrot|broccoli|cucumber)'. Path: '/foods/first_food/properties/name'", u"Value 'fruit' does not match pattern 'vegetable'. Path: '/foods/first_food/classification'"]

I think I'm trying to say this: "Each food mapping must pass any (not all) of these regex-based validations." I'm not sure if I can do that though. I thought perhaps with YAML anchors, but perhaps not. The next best thing I think I can do is make these separate schemas and run each one in turn.

Thoughts?

(Meanwhile, I'm really enjoying exploring pykwalify!)

regex;(blah[1-2]) - AttributeError: 'dict' object has no attribute 'encode'

Using the below schema:

map:
  regex;(blah[1-2]):
    type: int

against this data:

blah1: 1
blah2: 1
blah3: 1

I am getting the below stacktrace:

    self._validate_mapping(value, rule, path, errors, done=None)
  File "/Users/me/.pyenvs/demo_dc/lib/python2.7/site-packages/pykwalify/core.py", line 496, in _validate_mapping
    value=value.encode('unicode_escape'),
AttributeError: 'dict' object has no attribute 'encode'

I would have expected to see a more descriptive warning which indicates a bad keyword at 'blah3'. If I update the schema to the below, I get clean results:

map:
  regex;(blah[1-3]):
    type: int

Are strict regex matches supported for mapping keyword checks?

Add min/max to regex keyword

Consider:

map:
  regex;(^db\.type):
    min: 1
...

Would be nice if we could specify the min/max constraints under a regex. These would specify the minimum and maximum number of matches for that regex. Before I take a stab at it, any initial thoughts?

Include tests

Downstream projects like Linux distributions like to test the packages during installation. Please include the tests needed for the test suite in you releases at pypi.

Readthedocs doesn't have any docs.

The README says "All documentation can be found at http://pykwalify.readthedocs.org/en/latest/", which exists but is empty. Should this link be changed to http://pykwalify.readthedocs.org (which redirects to http://pykwalify.readthedocs.org/en/unstable/) ?

A sequence of maps in a schema does not check the type in the elements of the sequence

#Schema
type: seq
required: True
matching: all
seq:
    - type: map
      required: True
      map:
          foo:
              type: str

#yaml
- foo: whatever
- foo: whatever
- "sgdf"
- 2
-

INFO - validation.valid

pattern fail if data to validate is encoded in utf-8 with python2.7

in python 2.7 if a yaml contain uf8 characters and if we try to validate field with pattern.
an error is raised

pykwalify -d utf8_data.yml  -s utf8_schema.yml 
Traceback (most recent call last):
  File "/usr/lib/python-exec/python2.7/pykwalify", line 9, in <module>
    load_entry_point('pykwalify==1.5.1', 'console_scripts', 'pykwalify')()
  File "/usr/lib64/python2.7/site-packages/pykwalify/cli.py", line 84, in cli_entrypoint
    run(parse_cli())
  File "/usr/lib64/python2.7/site-packages/pykwalify/cli.py", line 71, in run
    c.validate()
  File "/usr/lib64/python2.7/site-packages/pykwalify/core.py", line 147, in validate
    self._start_validate(self.source)
  File "/usr/lib64/python2.7/site-packages/pykwalify/core.py", line 192, in _start_validate
    self._validate(value, root_rule, path, done)
  File "/usr/lib64/python2.7/site-packages/pykwalify/core.py", line 211, in _validate
    self._validate_mapping(value, rule, path, done=None)
  File "/usr/lib64/python2.7/site-packages/pykwalify/core.py", line 529, in _validate_mapping
    self._validate(v, r, u"{}/{}".format(path, k), done)
  File "/usr/lib64/python2.7/site-packages/pykwalify/core.py", line 213, in _validate
    self._validate_scalar(value, rule, path, done=None)
  File "/usr/lib64/python2.7/site-packages/pykwalify/core.py", line 561, in _validate_scalar
    res = re.match(rule.pattern, str(value))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 1: ordinal not in range(128)

whereas it works in python3

data.zip

When will commit 9da45c7 be released on PyPI?

Commit 9da45c7 fixes an issue with displaying information about a scalar value not matching a pattern, which I've hit and will probably confuse the users of my software.

Validate if list contains dictionaries without validating keys in dictionary

If scenario file has a list of dictionaries and the keys could be different each time, and user doesn't want to test for key names but just wants to check if the the sequence is a list of dictionaries.

Data
rally:
  plugins:
    - netcreate-boot: rally/rally-plugins/netcreate-boot 
#Schema
type: map
mapping:
  rally:
    type: map
    allowempty: True
    mapping:
      plugins:
        type: seq
        sequence:
        - type: map
          allowempty: True

In the above case, user just wants to check if plugins is a list of key value pairs.
The above schema validation throws an exception. It would be nice to have this feature as discussed in #53

Sequences of mappings are not validated

# data
- "hello"
- "world"

# schema
type: seq
sequence:
    - type: map
      mapping:
          A:
              type: str
          B:
              type: str

$ pykwalify -s schema.yml -d data.yml
 INFO - validation.valid

Question: recursive types?

Hi!

I've a question about pykwalify's handling of nested types. Is it possible to define new recursive types? For example, if I want to represent a filesystem whose depth is not known at schema definition. How can I do that?

Here is a fake example of such a YAML file:

root:
 - file: file_1
 - file: file_2
 - directory: 
   - name: dir_a
   - content: 
     - file: file_a_1
     - file: file_a_2
     - directory:
       - name: dir_a_a
       - content: 
          ...

Is this kind of nested structures already supported?

Align timestamp format with the yaml standard library

I have a suggestion about what formats to support for the timestamp format: why don't we use yaml.dump(the_date_value) to support exactly what the actual yaml library supports? We can check the the call returns a datetime.

That way there is no question of whether we accept invalid formats (or whether we throw an error on a yaml-compliant format).

Using the yaml library to parse the date delegates this kind of decision to the people maintaining the yaml lib and ensures the formats are aligned.

Returning False in custom validation raising core error

Hello,

In custom validation function, when I return False I get the following error

File "/usr/local/lib/python2.7/dist-packages/pykwalify/core.py", line 238, in _handle_func
raise CoreError(u"Error when running extension function : {}".format(func))
pykwalify.errors.CoreError: <CoreError: error code 3: Error when running extension function : is_valid_ipv4: Path: '/'>

But it is working fine when I raise the assertion error. Is this expected behavior ?

File "/usr/local/lib/python2.7/dist-packages/pykwalify/core.py", line 234, in _handle_func
ret = method(value, rule, path)
File "/home/xxx/yaml/val.py", line 31, in is_valid_ipv4
raise AssertionError(ipError)
AssertionError: Invalid IP

Thanks
~Hari

Support for default rule on mapping?

I may be using the syntax incorrectly, but is there support for the =: syntax from kwalify such as shown in section 2-3 from here?

I'm trying to use it where I may have many different map keys, but each has the same sequence associated with it, e.g.:

type: map
mapping:
  =:
    type: seq
    required: true
    sequence:
      - type: map
        mapping:
          'code':
            type: int
            required: true
            unique: true
          'key':
            type: str
            required: true
          'alias':
            type: str
            required: true

When running this through pykwalify, I get the following errror messages:

- Cannot find required key '='. Path: ''.
- Key 'OWNER' was not defined. Path: ''.
- Key 'ALLOCATION' was not defined. Path: ''.
etc ...

Sequence of Map - `AttributeError: 'str' object has no attribute 'items'`` when data contains Map in Map

Following schema describes a sequence map:

schema.yml

type: map
required: yes
mapping:
  "employees":
    type: seq
    required: yes
    sequence:
      - type: map
        required: yes
        mapping:
          "name": { type: str, required: yes }

for data containing map in map:

data.yml

employees:
    name: John Doe

when I try to check this data I get following exceptions:

$ pykwalify -d data.yml -s schema.yml
Traceback (most recent call last):
  File "/home/vagrant/tycho/bin/pykwalify", line 9, in <module>
    load_entry_point('pykwalify==1.1.0', 'console_scripts', 'pykwalify')()
  File "/home/lib/python2.7/site-packages/pykwalify/cli.py", line 79, in cli_entrypoint
    run(parse_cli())
  File "/home/lib/python2.7/site-packages/pykwalify/cli.py", line 66, in run
    c.validate()
  File "/home/lib/python2.7/site-packages/pykwalify/core.py", line 108, in validate
    errors = self._start_validate(self.source)
  File "/home/lib/python2.7/site-packages/pykwalify/core.py", line 151, in _start_validate
    self._validate(value, root_rule, path, errors, done)
  File "/home/lib/python2.7/site-packages/pykwalify/core.py", line 172, in _validate
    self._validate_mapping(value, rule, path, errors, done=None)
  File "/home/lib/python2.7/site-packages/pykwalify/core.py", line 352, in _validate_mapping
    self._validate(v, r, "{}/{}".format(path, k), errors, done)
  File "/home/lib/python2.7/site-packages/pykwalify/core.py", line 170, in _validate
    self._validate_sequence(value, rule, path, errors, done=None)
  File "/home/lib/python2.7/site-packages/pykwalify/core.py", line 216, in _validate_sequence
    self._validate(item, r, "{}/{}".format(path, i), errors, done)
  File "/home/lib/python2.7/site-packages/pykwalify/core.py", line 172, in _validate
    self._validate_mapping(value, rule, path, errors, done=None)
  File "/home/lib/python2.7/site-packages/pykwalify/core.py", line 310, in _validate_mapping
    for k, v in value.items():
AttributeError: 'str' object has no attribute 'items'

Multi regex and matching-rule any reporting failure even though there should be a match.

Data file:

foobar1: 1
foobar1: 2
bar2: 3

Schema file:

type: map
matching-rule: 'any'
mapping:
  regex;([1-2]$):
    type: int
  regex;(^foobar): 
    type: int

I would assume that 'bar2' would pass using the matching-rule any. However, I am getting the below failure message:

ERROR - validation.invalid
 ERROR -  --- All found errors ---
 ERROR - [u"Key 'bar2' does not match any regex '^foobar' or '[1-2]$'. Path: ''"]
Traceback (most recent call last):
  File "/Users/me/.venv/pykwalify/bin/pykwalify", line 9, in <module>
    load_entry_point('pykwalify==1.4.1', 'console_scripts', 'pykwalify')()
  File "build/bdist.macosx-10.10-intel/egg/pykwalify/cli.py", line 84, in cli_entrypoint
  File "build/bdist.macosx-10.10-intel/egg/pykwalify/cli.py", line 71, in run
  File "build/bdist.macosx-10.10-intel/egg/pykwalify/core.py", line 157, in validate
pykwalify.errors.SchemaError: <SchemaError: error code 2: Schema validation failed:
 - Key 'bar2' does not match any regex '^foobar' or '[1-2]$'. Path: ''.: Path: '/'>

Output is from branch 'regex_fix_and_test' which is in pull request #24.

Unique for mapping

it seems that kwalify in ruby supports unique for sequence AND mapping, the pykwalify only supports it for sequence.

this seems to me pretty important.

see the following example:

foo:
  some_feature: prod
  some_feature: dev

based on the fact that pyaml will just load the last key this can have some pretty scary consequences

Remove keyword pattern in favor of regex;<regex-pattern>

The keyword "pattern" is no longer useful at the top level of map validation because of the new rule "regex;" that works alot better.

AttributeError: 'int' object has no attribute 'encode' (In _validate_sequence)

When trying to validate a YAML file with integer instead of a sequence, the following exception occurs:

    def _validate_sequence(self, value, rule, path, errors, done=None):
    log.debug(u"Core Validate sequence")
    log.debug(u" * Data: {}".format(value))
    log.debug(u" * Rule: {}".format(rule))
    log.debug(u" * RuleType: {}".format(rule._type))
    log.debug(u" * Path: {}".format(path))
    log.debug(u" * Seq: {}".format(rule._sequence))
    log.debug(u" * Map: {}".format(rule._mapping))

    if len(rule._sequence) <= 0:
        raise CoreError(u"Sequence must contains atleast one item : {}".format(path))

    if value is None:
        log.debug(u" * Core seq: sequence data is None")
        return

    if not isinstance(value, list):
>           raise NotSequenceError(u"Value: {} is not of a sequence type".format(value.encode('unicode_escape')))
E           AttributeError: 'int' object has no attribute 'encode'

Here is part of my schema:

my_list:
    type: seq
    matching: "all"
    sequence:
        -   type: str
            required: True

I tried to verify a YAML file that contained:

my_list: 3

Expected behaviour:

Raising SchemaError exception.

Observed behaviour:

Raising AttributeError exception

Thank you for your help!

setup.py fail

the setup.py look for a file ReleaseNotes.rst which obviously does not exists

$python setup.py install
Traceback (most recent call last):
  File "setup.py", line 8, in <module>
    with open('ReleaseNotes.rst') as f:
IOError: [Errno 2] No such file or directory: 'ReleaseNotes.rst'

Bug in raising NotSequenceError

There seems to be a bug in sequence validation.
Here's the stack trace from validating a yaml file where a field was supposed to be a sequence of objects but was in fact an object:

Traceback (most recent call last):
  File "c:\python27\lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "c:\python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Python27\Scripts\pykwalify.exe\__main__.py", line 9, in <module>
  File "c:\python27\lib\site-packages\pykwalify\cli.py", line 84, in cli_entrypoint
    run(parse_cli())
  File "c:\python27\lib\site-packages\pykwalify\cli.py", line 71, in run
    c.validate()
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 145, in validate
    errors = self._start_validate(self.source)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 190, in _start_validate
    self._validate(value, root_rule, path, errors, done)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 210, in _validate
    self._validate_mapping(value, rule, path, errors, done=None)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 524, in _validate_mapping
    self._validate(v, r, u"{}/{}".format(path, k), errors, done)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 208, in _validate
    self._validate_sequence(value, rule, path, errors, done=None)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 303, in _validate_sequence
    self._validate(item, r, "{}/{}".format(path, i), tmp_errors, done)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 210, in _validate
    self._validate_mapping(value, rule, path, errors, done=None)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 524, in _validate_mapping
    self._validate(v, r, u"{}/{}".format(path, k), errors, done)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 208, in _validate
    self._validate_sequence(value, rule, path, errors, done=None)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 303, in _validate_sequence
    self._validate(item, r, "{}/{}".format(path, i), tmp_errors, done)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 210, in _validate
    self._validate_mapping(value, rule, path, errors, done=None)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 524, in _validate_mapping
    self._validate(v, r, u"{}/{}".format(path, k), errors, done)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 208, in _validate
    self._validate_sequence(value, rule, path, errors, done=None)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 285, in _validate_sequence
    raise NotSequenceError(u"Value: {} is not of a sequence type".format(value.encode('unicode_escape')))
AttributeError: 'dict' object has no attribute 'encode'

which points to:
[https://github.com/Grokzen/pykwalify/blob/unstable/pykwalify/core.py#L285]

Partial schema issue when used with mapping

Thanks for maintaining this project

While trying to use partial schema with map, I notice a small problem with use of partial schema with mapping.

The snippet posted below is failing whereas I think it should not.

version used if 8c3327e from Sep 3 2016
To reproduce:

schema_ok = """
type: map
mapping:
  foo:
    type: seq
    sequence:
      - type: str
"""

schema_fail = """
schema;list_str:
  type: seq
  sequence:
  - type: str

type: map
mapping:
  foo:
    - include: list_str
"""

data = """
foo:
  - bar
"""

import os

for file, content in [
    ('schema_ok.yml',schema_ok),
    ('schema_fail.yml',schema_fail),
    ('data.yml',data)]:
    with open(file, 'w') as f:
        f.write(content)
    assert os.path.exists(file)

import pykwalify.core

pykwalify.core.Core(
    source_file="data.yml",
    schema_files=["schema_ok.yml"]
).validate(raise_exception=True)

pykwalify.core.Core(
    source_file="data.yml",
    schema_files=["schema_fail.yml"]
).validate(raise_exception=True)

error reported:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-2-6d13cd20baba> in <module>()
     45     source_file="data.yml",
     46     schema_files=["schema_fail.yml"]
---> 47 ).validate(raise_exception=True)

/home/jovyan/work/pykwalify/pykwalify/core.py in validate(self, raise_exception)
    145         log.debug(u"starting core")
    146 
--> 147         self._start_validate(self.source)
    148         self.validation_errors = [unicode(error) for error in self.errors]
    149         self.validation_errors_exceptions = self.errors

/home/jovyan/work/pykwalify/pykwalify/core.py in _start_validate(self, value)
    185 
    186         log.debug(u"Building root rule object")
--> 187         root_rule = Rule(schema=self.schema)
    188         self.root_rule = root_rule
    189         log.debug(u"Done building root rule")

/home/jovyan/work/pykwalify/pykwalify/rule.py in __init__(self, schema, parent)
     60 
     61         if isinstance(schema, dict):
---> 62             self.init(schema, "")
     63 
     64     @property

/home/jovyan/work/pykwalify/pykwalify/rule.py in init(self, schema, path)
    344         for k, v in schema.items():
    345             if k in func_mapping:
--> 346                 func_mapping[k](v, rule, path)
    347             elif k.startswith("schema;"):
    348                 log.debug(u"Found schema tag...")

/home/jovyan/work/pykwalify/pykwalify/rule.py in init_mapping_value(self, v, rule, path)
    831             else:
    832                 rule = Rule(None, self)
--> 833                 rule.init(v, u"{}/mapping/{}".format(path, k))
    834                 self.mapping[k] = rule
    835 

/home/jovyan/work/pykwalify/pykwalify/rule.py in init(self, schema, path)
    276         log.debug(u"Init schema: %s", schema)
    277 
--> 278         include = schema.get("include", None)
    279 
    280         # Check if this item is a include, overwrite schema with include schema and continue to parse

AttributeError: 'list' object has no attribute 'get'

required attribute ignored when including partial schemas.

I expected the following two schemas to behave the same. Both should require A be defined in the schema. In actuality, the partial schema seems to ignore the required attribute.

Data

# A: "required" but omitted
B: hello

Full Schema

mapping:
   A:
      required: True
      type: map
      mapping:
         D:
            required: True
            type: str
         E:
            type: str
   B:
      type: str
      required: True

Output (Error Expected)

$ pykwalify -d data.yml -s schema.yml
 ERROR - validation.invalid
 ERROR -  --- All found errors ---
 ERROR - [u"Cannot find required key 'A'. Path: ''"]
Traceback (most recent call last):
  File "/usr/local/bin/pykwalify", line 11, in <module>
    sys.exit(cli_entrypoint())
  File "/usr/local/lib/python2.7/dist-packages/pykwalify/cli.py", line 84, in cli_entrypoint
    run(parse_cli())
  File "/usr/local/lib/python2.7/dist-packages/pykwalify/cli.py", line 71, in run
    c.validate()
  File "/usr/local/lib/python2.7/dist-packages/pykwalify/core.py", line 159, in validate
    error_msg=u'.\n - '.join(self.validation_errors)))
pykwalify.errors.SchemaError: <SchemaError: error code 2: Schema validation failed:
 - Cannot find required key 'A'. Path: ''.: Path: '/'>

Partial Schema Included

mapping:
   A:
      required: True
      include: C
   B:
      type: str
      required: True

schema;C:
   type: map
   mapping:
      D:
         required: True
         type: str
      E:
         type: str

Output (Error Expected)

$ pykwalify -d data.yml -s partial.yml
INFO - validation.valid

Investigate support for multiple validations for same key in mapping

As talked about in #39 there is a request that you want a easier way have multiple validation paths for one key in a mapping. This would however invent a new syntax.

grokzen / pykwalify Goto Github PK

pykwalify's Introduction

Hi there 👋

Github Statistics

pykwalify's People

Contributors

Stargazers

Watchers

Forkers

pykwalify's Issues

Observed:

Expected:

Example:

Expected behaviour:

Observed behaviour:

Data

Full Schema

Output (Error Expected)

Partial Schema Included

Output (Error Expected)

Recommend Projects

Recommend Topics

Recommend Org