tefra / xsdata-pydantic Goto Github PK

View Code? Open in Web Editor NEW

42.0 3.0 8.0 52 KB

Naive XML & JSON Bindings for python pydantic classes!

Home Page: https://xsdata-pydantic.readthedocs.io/

License: MIT License

Python 100.00%

python json parser xml bindings code-generator pydantic schema wsdl xsd

xsdata-pydantic's Introduction

xsdata powered by pydantic!

xsData is a complete data binding library for python allowing developers to access and use XML and JSON documents as simple objects rather than using DOM.

Now powered by pydantic!

$ xsdata http://rss.cnn.com/rss/edition.rss --output pydantic
Parsing document edition.rss
Analyzer input: 9 main and 0 inner classes
Analyzer output: 9 main and 0 inner classes
Generating package: init
Generating package: generated.rss

class Rss(BaseModel):
    class Meta:
        name = "rss"

    model_config = ConfigDict(defer_build=True)
    version: float = field(
        metadata={
            "type": "Attribute",
            "required": True,
        }
    )
    channel: Channel = field(
        metadata={
            "type": "Element",
            "required": True,
        }
    )

>>> from xsdata_pydantic.bindings import XmlParser
>>> from urllib.request import urlopen
>>> from generated.rss import Rss
>>>
>>> parser = XmlParser()
>>> with urlopen("http://rss.cnn.com/rss/edition.rss") as rq:
...     result = parser.parse(rq, Rss)
...
>>> result.channel.item[2].title
'Vatican indicts 10 people, including a Cardinal, over an international financial scandal'
>>> result.channel.item[2].pub_date
'Sat, 03 Jul 2021 16:37:14 GMT'
>>> result.channel.item[2].link
'https://www.cnn.com/2021/07/03/europe/vatican-financial-scandal-intl/index.html'

Changelog: 24.5 (2024-05-08)

Support pydantic v2 models
Add missing parser/serializer shortcuts
General project maintenance

xsdata-pydantic's People

Contributors

Stargazers

Watchers

Forkers

mauriziocasciano tlambert03 bryan-hunt compton-us aj-stein-nist xee5ch fleimgruber shpigunov

xsdata-pydantic's Issues

Hints for usage with fastapi

I used this code to generate pydantic schemas intended for use with fastapi. Background is, I receive data from Exlibris Alma Webhooks. The schema is available as XSD only (no JSON schemas). With this great tool here I directly generate pydantic schemas for fastapi Thanks so much!
Anyway, they do not work out-of-the-box, some manual steps needed:

nested has to be set to false
the metadata, specifically the required=true in there, broke fastapis get_model_definitions. I manually removed the whole metadata, because Elements and Attributes are not needed in JSON.
I replaced XMLDate and XMLDateTime by datetime.date and datetime.datetime.

Now it at least loads the model and it seems incoming JSON data is validated as expected.

Autosar failure

There is an open issue here:

pydantic/pydantic#3695

ADD pydantic restrictions

It would be great to be able to use the restrictions that pydantic Field offers, eg: min_items, max_items, regex; etc.

XmlPeriod don't have validator

I generated a model for KML file but I got an no validator found exception.

To reproduce

xsdata generate https://schemas.opengis.net/kml/2.2.0/ogckml22.xsd --output pydantic --package kml

Run

from kml import ogckml22


def main():
    point = ogckml22.Placemark(
        name="Sample", point=ogckml.Point(coordinates="-122.0822035425683,37.42228990140251,0)"
    )

if __name__ == "__main__":
    main()

Output

Exception has occurred: RuntimeError       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
no validator found for <class 'xsdata.models.datatype.XmlPeriod'>, see `arbitrary_types_allowed` in Config
  File "C:\Users\User\AppData\Local\pypoetry\Cache\virtualenvs\pos2xml-nzgsLrQL-py3.10\Lib\site-packages\pydantic\validators.py", line 765, in find_validators
    raise RuntimeError(f'no validator found for {type_}, see `arbitrary_types_allowed` in Config')
  File "C:\Users\User\AppData\Local\pypoetry\Cache\virtualenvs\pos2xml-nzgsLrQL-py3.10\Lib\site-packages\pydantic\fields.py", line 831, in pydantic.fields.ModelField.populate_validators
    *(get_validators() if get_validators else list(find_validators(self.type_, self.model_config))),
  File "C:\Users\User\AppData\Local\pypoetry\Cache\virtualenvs\pos2xml-nzgsLrQL-py3.10\Lib\site-packages\pydantic\fields.py", line 557, in pydantic.fields.ModelField.prepare
    self.populate_validators()
  File "C:\Users\User\AppData\Local\pypoetry\Cache\virtualenvs\pos2xml-nzgsLrQL-py3.10\Lib\site-packages\pydantic\fields.py", line 436, in pydantic.fields.ModelField.__init__
    self.prepare()
  File "C:\Users\User\AppData\Local\pypoetry\Cache\virtualenvs\pos2xml-nzgsLrQL-py3.10\Lib\site-packages\pydantic\fields.py", line 808, in pydantic.fields.ModelField._create_sub_type
    return self.__class__(
  File "C:\Users\User\AppData\Local\pypoetry\Cache\virtualenvs\pos2xml-nzgsLrQL-py3.10\Lib\site-packages\pydantic\fields.py", line 663, in pydantic.fields.ModelField._type_analysis
    self.sub_fields = [self._create_sub_type(t, f'{self.name}_{display_as_type(t)}') for t in types_]
  File "C:\Users\User\AppData\Local\pypoetry\Cache\virtualenvs\pos2xml-nzgsLrQL-py3.10\Lib\site-packages\pydantic\fields.py", line 552, in pydantic.fields.ModelField.prepare
    self._type_analysis()
  File "C:\Users\User\AppData\Local\pypoetry\Cache\virtualenvs\pos2xml-nzgsLrQL-py3.10\Lib\site-packages\pydantic\fields.py", line 436, in pydantic.fields.ModelField.__init__
    self.prepare()
  File "C:\Users\User\AppData\Local\pypoetry\Cache\virtualenvs\pos2xml-nzgsLrQL-py3.10\Lib\site-packages\pydantic\fields.py", line 506, in pydantic.fields.ModelField.infer
Config    return cls(
  File "C:\Users\User\AppData\Local\pypoetry\Cache\virtualenvs\pos2xml-nzgsLrQL-py3.10\Lib\site-packages\pydantic\main.py", line 197, in pydantic.main.ModelMetaclass.__new__
    fields[ann_name] = ModelField.infer(
  File "C:\Users\User\AppData\Local\pypoetry\Cache\virtualenvs\pos2xml-nzgsLrQL-py3.10\Lib\site-packages\pydantic\main.py", line 1026, in pydantic.main.create_model
    return meta(__model_name, resolved_bases, namespace, **kwds)
  File "C:\Users\User\AppData\Local\pypoetry\Cache\virtualenvs\pos2xml-nzgsLrQL-py3.10\Lib\site-packages\pydantic\dataclasses.py", line 400, in pydantic.dataclasses.create_pydantic_model_from_dataclass
    model: Type['BaseModel'] = create_model(
  File "C:\Users\User\AppData\Local\pypoetry\Cache\virtualenvs\pos2xml-nzgsLrQL-py3.10\Lib\site-packages\pydantic\dataclasses.py", line 347, in pydantic.dataclasses._add_pydantic_validation_attributes
    setattr(dc_cls, '__pydantic_model__', create_pydantic_model_from_dataclass(dc_cls, config, dc_cls_doc))
  File "C:\Users\User\AppData\Local\pypoetry\Cache\virtualenvs\pos2xml-nzgsLrQL-py3.10\Lib\site-packages\pydantic\dataclasses.py", line 224, in pydantic.dataclasses.dataclass.wrap
    _add_pydantic_validation_attributes(cls, the_config, should_validate_on_init, dc_cls_doc)
  File "C:\Users\User\AppData\Local\pypoetry\Cache\virtualenvs\pos2xml-nzgsLrQL-py3.10\Lib\site-packages\pydantic\dataclasses.py", line 231, in pydantic.dataclasses.dataclass
    return wrap(_cls)
  File "C:\Users\User\Documents\Projects\position-extract\kml\ogckml22.py", line 147, in <module>
    class Begin:
  File "C:\Users\User\Documents\Projects\position-extract\kml\__init__.py", line 9, in <module>
    from kml.ogckml22 import (
  File "C:\Users\User\Documents\Projects\position-extract\pos2kml\main.py", line 1, in <module>
    from kml import ogckml22
  File "C:\Users\User\scoop\apps\pyenv\current\pyenv-win\versions\3.10.2\Lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\User\scoop\apps\pyenv\current\pyenv-win\versions\3.10.2\Lib\runpy.py", line 196, in _run_module_as_main (Current frame)
    return _run_code(code, main_globals, None,
RuntimeError: no validator found for <class 'xsdata.models.datatype.XmlPeriod'>, see `arbitrary_types_allowed` in Config

However, I do saw a make_validators() for XmlPeriod was called in xsdata_pydantic/compact.py. I tried set a breakpoint there but it wasn't triggered.

Help?

@tefra, please excuse the use of "Issues" for this communication but I didn't see an email. Anything you are looking for help with on the xsdata or xsdata-pydantic projects?

unable to install

hi i am not able to install this extension

$pip install xsdata-pydantic[cli]
ERROR: Could not find a version that satisfies the requirement xsdata-pydantic[cli] (from versions: none)
ERROR: No matching distribution found for xsdata-pydantic[cli]

$ python -V
Python 3.10.4

Issue installing [cli] extras

No problem install with pip3 install xsdata-pydantic. However, unable to install with pip3 install xsdata-pydantic[cli]

Running Python 3.11.3

Error while using XmlDuration models

  File "pydantic/main.py", line 1026, in pydantic.main.create_model
  File "pydantic/main.py", line 198, in pydantic.main.ModelMetaclass.__new__
  File "pydantic/fields.py", line 506, in pydantic.fields.ModelField.infer
  File "pydantic/fields.py", line 436, in pydantic.fields.ModelField.__init__
  File "pydantic/fields.py", line 557, in pydantic.fields.ModelField.prepare
  File "pydantic/fields.py", line 831, in pydantic.fields.ModelField.populate_validators
  File "pydantic/validators.py", line 765, in find_validators
RuntimeError: no validator found for <class 'xsdata.models.datatype.XmlDuration'>, see `arbitrary_types_allowed` in Config

WSDL elements that rely on XMlDuration fail to validate because this is not a natively supported type.

Changing following: @dataclass to the following works: @dataclass(config=dict(arbitrary_types_allowed=True))

This is the same for some lxml elements:

RuntimeError: no validator found for <class 'xml.etree.ElementTree.QName'>, see `arbitrary_types_allowed` in Config

Field ordering issue in generated output

Attempting to dump the JSONschema from the pydantic class generated by xsdata-pydantic results in a type error due to field ordering: "TypeError: non-default argument 'name_identifier' follows default argument"

To reproduce:

Source file: https://github.com/datacite/schema/blob/master/source/meta/kernel-4.4/metadata.xsd (plus associated files)

xsdata kernel-4.4/metadata.xsd

then run the following python code:

>>> import pydantic
>>> import generated.metadata as md
>>> print(pydantic.schema_json_of(md.Resource, title="Resource", indent=2))

Output:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pydantic/tools.py", line 92, in pydantic.tools.schema_json_of
  File "pydantic/tools.py", line 30, in pydantic.tools._get_parsing_type
  File "pydantic/main.py", line 1026, in pydantic.main.create_model
  File "pydantic/main.py", line 198, in pydantic.main.ModelMetaclass.__new__
  File "pydantic/fields.py", line 506, in pydantic.fields.ModelField.infer
  File "pydantic/fields.py", line 436, in pydantic.fields.ModelField.__init__
  File "pydantic/fields.py", line 557, in pydantic.fields.ModelField.prepare
  File "pydantic/fields.py", line 831, in pydantic.fields.ModelField.populate_validators
  File "pydantic/validators.py", line 725, in find_validators
  File "pydantic/dataclasses.py", line 479, in make_dataclass_validator
    value = f'{default_name}()'
  File "pydantic/dataclasses.py", line 231, in pydantic.dataclasses.dataclass
    def __repr__(self):
  File "pydantic/dataclasses.py", line 224, in pydantic.dataclasses.dataclass.wrap
    
  File "pydantic/dataclasses.py", line 336, in pydantic.dataclasses._add_pydantic_validation_attributes
    f'eq={self.eq!r},'
  File "pydantic/dataclasses.py", line 391, in pydantic.dataclasses.create_pydantic_model_from_dataclass
    # This function's logic is copied from "recursive_repr" function in
  File "pydantic/main.py", line 1026, in pydantic.main.create_model
  File "pydantic/main.py", line 198, in pydantic.main.ModelMetaclass.__new__
  File "pydantic/fields.py", line 506, in pydantic.fields.ModelField.infer
  File "pydantic/fields.py", line 436, in pydantic.fields.ModelField.__init__
  File "pydantic/fields.py", line 557, in pydantic.fields.ModelField.prepare
  File "pydantic/fields.py", line 831, in pydantic.fields.ModelField.populate_validators
  File "pydantic/validators.py", line 725, in find_validators
  File "pydantic/dataclasses.py", line 479, in make_dataclass_validator
    value = f'{default_name}()'
  File "pydantic/dataclasses.py", line 231, in pydantic.dataclasses.dataclass
    def __repr__(self):
  File "pydantic/dataclasses.py", line 224, in pydantic.dataclasses.dataclass.wrap
    
  File "pydantic/dataclasses.py", line 336, in pydantic.dataclasses._add_pydantic_validation_attributes
    f'eq={self.eq!r},'
  File "pydantic/dataclasses.py", line 391, in pydantic.dataclasses.create_pydantic_model_from_dataclass
    # This function's logic is copied from "recursive_repr" function in
  File "pydantic/main.py", line 1026, in pydantic.main.create_model
  File "pydantic/main.py", line 198, in pydantic.main.ModelMetaclass.__new__
  File "pydantic/fields.py", line 506, in pydantic.fields.ModelField.infer
  File "pydantic/fields.py", line 436, in pydantic.fields.ModelField.__init__
  File "pydantic/fields.py", line 552, in pydantic.fields.ModelField.prepare
  File "pydantic/fields.py", line 758, in pydantic.fields.ModelField._type_analysis
  File "pydantic/fields.py", line 808, in pydantic.fields.ModelField._create_sub_type
  File "pydantic/fields.py", line 436, in pydantic.fields.ModelField.__init__
  File "pydantic/fields.py", line 557, in pydantic.fields.ModelField.prepare
  File "pydantic/fields.py", line 831, in pydantic.fields.ModelField.populate_validators
  File "pydantic/validators.py", line 725, in find_validators
  File "pydantic/dataclasses.py", line 479, in make_dataclass_validator
    value = f'{default_name}()'
  File "pydantic/dataclasses.py", line 231, in pydantic.dataclasses.dataclass
    def __repr__(self):
  File "pydantic/dataclasses.py", line 207, in pydantic.dataclasses.dataclass.wrap
    
  File "/usr/local/Cellar/[email protected]/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/dataclasses.py", line 1185, in dataclass
    return wrap(cls)
  File "/usr/local/Cellar/[email protected]/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/dataclasses.py", line 1176, in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash,
  File "/usr/local/Cellar/[email protected]/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/dataclasses.py", line 1025, in _process_class
    _init_fn(all_init_fields,
  File "/usr/local/Cellar/[email protected]/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/dataclasses.py", line 546, in _init_fn
    raise TypeError(f'non-default argument {f.name!r} '
TypeError: non-default argument 'name_identifier' follows default argument

DTD parsing, CLI, and BaseModel support

Excellent library. I was able to successfully generate a a tool for parsing fairly complex XML. There were a few bumps in the road, so I thought I'd give some feedback and see if there are some things you might find useful.

For reference, the library I generated:
https://github.com/nicholas-schaub/pubmed-types

You can see the entire generation process and run it yourself if you're interested, since it uses a nox workflow. You can find it in noxfile.py, and can run it by installing with pip install nox and running from the commandline using nox.

First, dtd support wasn't documented, but I tried running it on a dtd file and to my surprised it worked. However, it wasn't able to properly link mml:math, and it's unclear to me where the issue with this is. I played around with the .mod files and the entity definitions, but I just could not get it to link properly. Fortunately, the other schema I was trying to convert had the same class, so I was able to go in and link to the Math class generated by the other schema. But I had to manually modify the file, as you will see in the noxfile workflow. So I guess my question/comment would be: Is this an issue with the dtd, or is this a quirk of the current implementation because dtd isn't officially supported?

Second, a minor comment: Could you add an input argument to the CLI to ignore a top level folder? A common practice in making Python libraries is to put all source code into a src folder. The reason for this is that this additional layer helps to prevent a local import while testing. However, when I try to generate pydantic classes I get an __init__.py in the src directory that I have to manually remove.

Third, could you support pydantic.BaseModel instead of just pydantic.dataclass? I ask because BaseModel has some useful serialization/deserialization functionality that the dataclass doesn't have. It's obviously still possible to get the same functionality, but it's more boilerplate work.

I realize this is a lot. I'm happy to close this issue and submit issues for each topic to make them more granular.

Issue with optional fields with flag --ouput pydantic vs dataclass

Hi @tefra,

Your package looks promising.
I encountered an issue while generating models from a xsd.
The dataclass models work fine while the pydantic models don't.

It seems to me that there is a problem in how optional fields are generated (only for --output pydantic)
You'll see the difference in the following image (for the same initial DPEv2.2.xsd file), where fields where tagged "optional" in the dataclass version while they were not in the pydantic. Maybe this is due to the fact that these fields are grouped in a <xsd:all> tag ?!

I generated the models from the same xsd like this :

xsdata DPEv2.2.xsd --package models_dataclass
xsdata DPEv2.2.xsd --output pydantic --package models_pydantic

And run the following code on a sample xml file

# Uncomment to work on models_pydantic version
# from models_pydantic import Dpe
# from xsdata_pydantic.bindings import XmlParser

# Uncomment to work on models_dataclass version
from models_dataclass import Dpe
from xsdata.formats.dataclass.parsers import XmlParser

parser = XmlParser()
result = parser.parse("2344E0308327N.xml", Dpe)

murs = result.logement.enveloppe.mur_collection.mur
for mur in murs:
    print(mur.donnee_entree.surface_paroi_totale)

You'll find the example xsd and xml files here : mwe.zip

Add custom Qname implementation

add custom qname implementation with pydantic validators

command line only outputs: Install cli requirements "pip install xsdata[cli]"

Tried to install xsdata-pydantic as indicated on the docs (pip install xsdata-pydantic[cli]) when running a xsdata command (xsdata --help) it only outpurs: Install cli requirements "pip install xsdata[cli]"

Even doing first a pip install xsdata[cli] and pip install xsdata-pydantic[cli] it outputs the same error message. This can be tested using this dockerfile:

FROM python:3.8.18-slim-bookworm

RUN pip install xsdata[cli]
RUN pip install xsdata-pydantic[cli]
WORKDIR /data
ENTRYPOINT ["xsdata"]

Running:

docker build --progress=plain --no-cache -t xsdata-pydantic .
docker run  xsdata-pydantic --version
Install cli requirements "pip install xsdata[cli]"