Implement partial schemas about pykwalify HOT 15 CLOSED

@gschmottlach-xse I have done a first test implementation of the partial schemas and include directive. You can look at testfiles 32, 33, 34 and readme for how it is implemented.

Currently only limitation is that "include:" cannot be used inside a "schema;" but that is because of some limitations in my implementation that i will fix in later commits but the basic idea and usage will still be the same

Play around with it some and see how you like it and give comments/suggestions how to improve this feature :]

The brand this is implemented in can be found here : https://github.com/Grokzen/pykwalify/tree/partial_schemas

from pykwalify.

Sorry for not getting back with you sooner but I've been away focused on other tasks. I have resumed playing around with the partial schema implementation. It seems to be working for me although it would be very nice if you could address the limitation whereby a schema definition cannot "include" another schema definition. I tend to like to build my schemas based on smaller sub-units. Right now the YAML linking hack is my only work around. Is this something easy for you to fix?

from pykwalify.

I looked at your code that provides the partial schema support and I think I know why test cases 34a, 34b, and 34c fail. Part of the problem is that the first partial schema refers to a schema that is defined later in the file. Unfortunately, yaml.load() does not preserve the order of the definitions since they are stored in a Python dictionary (rather than array). The schemas are (effectively) recursively specified. Before the first partial schema has been completely resolved it references (or includes) the second partial schema and since it hasn't been parsed/processed yet it is never found.

Now if there was a way to force the parser to validate the schemas in the order defined in the file (and the second partial schema 'footwo' were defined before 'fooone') then I think things would work nicely. Otherwise, I think you almost need a two-stage parse. Any "include" statements are not resolved until after the first pass through all the files. Likely, there would have to be a "fix-up" step done to resolve all the partial schemas with their fully expressed rules. I'm not sure how hard that would be but suspect it may be non-trivial.

from pykwalify.

@gschmottlach-xse The code was almost working allready :] The solution was that when parsing all rules no includes should be parsed recursively so that the problem with partial schemas that is not parsed yet. And when the rules is used when validating the data, then the include directives is used and by that time all partial schemas is loaded and everything works.

By commit 1032660 the include inside partial schema works with test 34 & 35. If you can break it or find some corner case that do not currently work that would be awesome :]

from pykwalify.

Some issues was found during testing and in this commit 8f43567 it now works fine again.

from pykwalify.

I've tried your latest changes for partial schema support (with nested "includes" in the schema itself) and things seem to be working fine for me. That's a real nice addition - thanks for the quick turn-around on that feature. It simplified my schema definitions greatly.

I have another question related to the "required" directive. Does this only apply to the "scalar" types (rather than the container types: map, sequences)? I was wondering if it was possible to specify in a schema whether a map or sequence (as a whole) is required (or not). Not sure if this can be achieved with the existing grammar. The parser seems to accept "required: True/False" after/before "type: seq" or "type: map" so I erroneously that it might be applicable.

from pykwalify.

Hmm, could you give some more detailed examples and cases where it would be good to have required in container types because i cant figure out how that would be good/bad. Please do that in a new issue because i am about to close this one and merge the code related to this issue.

One of the only places where i can see that it would be possible in the future is when you can have multiple items in the 'sequence' block so that a sequence can be validated in multiple different ways. Then you could enforce/require that some of the blocks validate while other are optional. But that is not implemented yet. And when you are in a map there is no way test a map against multiple rule defenitions so required on the map structure would not make sense.

from pykwalify.

Branch merged into master.

from pykwalify.

I guess I was thinking of the scenario where you have a map/sequence attached as the "value" of a map's key. In that situation I guess I can just make the parent key required (or not) to indicate whether or not there needs to be an associated child map/sequence. You're right, at the top level of a map/sequence it probably doesn't make sense.

from pykwalify.

Hmm i think that is already possible. Don't you mean this?

type: map
mapping:
  streams:
    type: map
    required: True
    mapping:
      name:
        type: str

Here required is bound to streams key and it will fail if if it is not defined in the data structure.

from pykwalify.

Yes, that's what I meant. It's already handled (as you indicated above) by making the map key required (or not).

from pykwalify.

Is it possible to load partials from another file? I would like to have partials defined in separate files, however, it doesn't appear that is possible via the CLI. Am I missing something?

from pykwalify.

@retr0h Yes it should be possible to specify more then 1 schema file in the cli.

The cli has the following defenition

 pykwalify -d DATAFILE -s SCHEMAFILE ... [-q] [-v ...]

and the ... following -s SCHEMAFILE means that you can specify -s more then 1 time and when it is processed it will be interpreted as a list of items.

You can read up more on how docopt works here if you like http://docopt.org/

If you however found a bug or a case where the multiple -s do not work as expected then you can file a issue here and i will fix it. I might go through the docs and make it more obvious that you can specify more then one -s option from cli.

from pykwalify.

@Grokzen Ah yes. I had tried that but stupidly was parsing 2 files which included the same schema name, and didn't throughly read the error. My apologies.

from pykwalify.

@retr0h No problems :]

from pykwalify.