Not necessarily up to date, but that the .txt is a valid result from the .in.

Given a definition of a binary tree type hint BTREE = Union[LEAF, Tuple['BTREE', 'BTREE']], this cause desert schema to end up in an indefinite loop in lib\typing.py. Given that it is stuck in a loop in stdlib, I am not sure if this is a desert issue vs a typing issue or if typing wasn't really designed for recursive type hints like this in the first place.

from typing import Optional, Union, Tuple
import desert
import attr

# Can be any type
LEAF = str

# Binary tree definition
BTREE = Union[LEAF, Tuple['BTREE', 'BTREE']]


def test_btree():

    @attr.s(slots=True)
    class A:
        x: BTREE = attr.ib(default='leaf')

    a = A()
    schema = desert.schema(A)
    d = schema.dump(a)

Results in

Traceback (most recent call last):
  File "C:\sveinse\tests\test_tree.py", line 19, in test_btree
    schema = desert.schema(A)
  File "c:\sveinse\venv\lib\site-packages\desert\__init__.py", line 24, in schema
    return desert._make.class_schema(cls, meta=meta)(many=many)
  File "c:\sveinse\venv\lib\site-packages\desert\_make.py", line 124, in class_schema
    field.metadata,
  File "c:\sveinse\venv\lib\site-packages\desert\_make.py", line 225, in field_for_schema
    if not field and typ in _native_to_marshmallow:
  File "C:\Users\sveinse\AppData\Local\Programs\Python\Python37-32\lib\typing.py", line 666, in __hash__
    return hash((Union, frozenset(self.__args__)))
  File "C:\Users\sveinse\AppData\Local\Programs\Python\Python37-32\lib\typing.py", line 667, in __hash__
    return hash((self.__origin__, self.__args__))
  File "C:\Users\sveinse\AppData\Local\Programs\Python\Python37-32\lib\typing.py", line 666, in __hash__
    return hash((Union, frozenset(self.__args__)))
.... and forever.

"type" is a very overloaded term. We can use "class" to refer to Python classes to reduce some confusion.

Hi, thanks for the nice library.

The following use case triggers an error in desert:

from dataclasses import dataclass

import desert

@dataclass
class WebhooksInfoItem:
    url: str = None

WebhooksInfoItemSchema = desert.schema(WebhooksInfoItem)

WebhooksInfoItemSchema.loads('{}')
# Ok: WebhooksInfoItem(url=None)

WebhooksInfoItemSchema.loads('{"url": null}')
# marshmallow.exceptions.ValidationError: {'url': ['Field may not be null.']}

marshmallow_dataclass seems to support optional declaration through default None in both cases.
Is it forbidden in desert by design? It does not look like it according to the first usecase.

I feel that = None to be an implicit equivalent of Optional[] to be a good shortcut.
Maybe I do not see a potential drawback?

A really common use-case I've run into in my codebase is that our raw data schemas use hyphens - thus not mapping directly to a python identifier. In these cases we need to specify data_key and when we do that, we have to break out and define the entire marshmallow field ourselves, losing all the great stuff desert does for us. I'd love to see desert.field, desert.ib, and desert.metadata extended to support data_key= to forward along to the underlying type, as well as potentially other standard field options if applicable. The other two that stand out to me are validate and error_messages. I acknowledge that this is a slippery slope but maybe it'd make sense for those APIs to just forward **kwargs to the underlying field, and potentially to even error on kwargs if a field is passed in. Were there other use cases of kwargs on those APIs that this would conflict with? If so, we could define a marshmallow_field_args arg instead to disambiguate, alongside the existing marshmallow_field positional?

@dataclass
class Parent:
    name: str = desert.field(mm.fields.Str(data_key='userid', required=True))

vs

@dataclass
class Parent:
    name: str = desert.field(data_key='userid')

I'm happy to contribute a PR if I can get some confidence that this will be accepted design-wise.

The following test generates an Validation error when the type hint typing.Any is used for a field and the de-serializer is given the value of None. From reading the py docs, I think the typing.Any hint intends to also cover None. The current workaround is to use t.Optional[t.Any].

Is this perhaps a Marshmallow issue?

def test_typing_any(module):
    """Test typing.Any

    """

    @module.dataclass
    class A:
        x: t.Any

    schema = desert.schema_class(A)()
    dumped = {"x": None}
    loaded = A(x=None)

    assert schema.load(dumped) == loaded

I haven't thought through whether this even ought to be a feature but there's code related to it and it doesn't seem to work. The following fails.

def test_default_via_metadata_loads(module):
    """"""

    default = 5

    @module.dataclass
    class A:
        x: int = module.field(metadata={"default": default})

    schema = desert.schema_class(A)()

    assert schema.load({}) == A()

tests/test_make.py:313 (test_default_via_metadata_loads[attrs])
tests/test_make.py:325: in test_default_via_metadata_loads
    assert schema.load({}) == A()
venv/lib/python3.8/site-packages/marshmallow/schema.py:713: in load
    return self._do_load(
venv/lib/python3.8/site-packages/marshmallow/schema.py:892: in _do_load
    raise exc
E   marshmallow.exceptions.ValidationError: {'x': ['Missing data for required field.']}

It seems at least possible that someone would want to require the value be present when taking it from json but not from Python. Is that what we think this means?:

attr.ib(default=1, metadata={desert._make._DESERT_SENTINEL:{'required': True}})

I'm getting the following error when I try to create a schema from the following:

from dataclasses import dataclass, field
from typing import *
import desert


@dataclass
class Reference:
    reference: str = ""

@dataclass
class Code:
    code: str = ""
    system: str = ""
    display: str = ""

@dataclass
class Coding:
    coding: List[Code]


@dataclass
class Type:
    coding: Coding


@dataclass
class Input:
    type: Type
    valueString: str

@dataclass
class ExecutionPeriod:
    start: str

@dataclass
class BusinessStatus:
    coding: List[Coding]

@dataclass
class TaskResponse:
    basedOn: List[Reference]
    businessStatus: BusinessStatus
    code: Coding
    description: str
    executionPeriod: ExecutionPeriod
    id: str
    input: List[Input]
    intent: str
    output: list
    requester: dict
    resourceType: str
    status: str

 schema = desert.schema(TaskResponse)

Traceback (most recent call last):
  File "tests/xxx/test.py", line 50, in test_task_response
    schema = desert.schema(TaskResponse)
  File "/xxx/python_env/py3/py3env/lib/python3.7/site-packages/desert/__init__.py", line 24, in schema
    return desert._make.class_schema(cls, meta=meta)(many=many)
  File "/xxx/python_env/py3/py3env/lib/python3.7/site-packages/desert/_make.py", line 122, in class_schema
    field.metadata,
  File "/xxx/python_env/py3/py3env/lib/python3.7/site-packages/desert/_make.py", line 260, in field_for_schema
    nested = forward_reference or class_schema(typ)
  File "/xxx/python_env/py3/py3env/lib/python3.7/site-packages/desert/_make.py", line 110, in class_schema
    raise desert.exceptions.NotAnAttrsClassOrDataclass(clazz)
desert.exceptions.NotAnAttrsClassOrDataclass: <class 'list'>

Not sure why it's saying class: 'list' when it's defined as a dataclass as you can see above?

Below is my attempt to minimally recreate the issue and show some not-working options and output.

https://mypy-play.net/?mypy=latest&python=3.8&flags=strict%2Ccheck-untyped-defs&gist=7956a372fa34375077012351364a3aec

import dataclasses
import typing


T = typing.TypeVar('T')


def make_a_field_t() -> T:
    return dataclasses.field()


def make_a_field_field() -> dataclasses.Field:
    return dataclasses.field()


def make_a_field_field_any() -> dataclasses.Field[typing.Any]:
    return dataclasses.field()


def make_a_field_nothing():
    return dataclasses.field()


@dataclasses.dataclass
class C:
    a: int = dataclasses.field()
    b: int = dataclasses.Field()
    c: int = make_a_field_t()
    d: int = make_a_field_field()
    e: int = make_a_field_field_any()
    f: int = make_a_field_nothing()


reveal_type(dataclasses.field)
reveal_type(dataclasses.Field)
reveal_type(make_a_field_t)
reveal_type(make_a_field_field)
reveal_type(make_a_field_field_any)
reveal_type(make_a_field_nothing)

main.py:9: error: Returning Any from function declared to return "T"
main.py:12: error: Missing type parameters for generic type "Field"
main.py:13: error: Returning Any from function declared to return "Field[Any]"
main.py:17: error: Returning Any from function declared to return "Field[Any]"
main.py:20: error: Function is missing a return type annotation
main.py:27: error: Incompatible types in assignment (expression has type "Field[<nothing>]", variable has type "int")
main.py:29: error: Incompatible types in assignment (expression has type "Field[Any]", variable has type "int")
main.py:30: error: Incompatible types in assignment (expression has type "Field[Any]", variable has type "int")
main.py:31: error: Call to untyped function "make_a_field_nothing" in typed context
main.py:34: note: Revealed type is 'Overload(def [_T] (*, default: _T`-1, init: builtins.bool =, repr: builtins.bool =, hash: Union[builtins.bool, None] =, compare: builtins.bool =, metadata: Union[typing.Mapping[builtins.str, Any], None] =) -> _T`-1, def [_T] (*, default_factory: def () -> _T`-1, init: builtins.bool =, repr: builtins.bool =, hash: Union[builtins.bool, None] =, compare: builtins.bool =, metadata: Union[typing.Mapping[builtins.str, Any], None] =) -> _T`-1, def (*, init: builtins.bool =, repr: builtins.bool =, hash: Union[builtins.bool, None] =, compare: builtins.bool =, metadata: Union[typing.Mapping[builtins.str, Any], None] =) -> Any)'
main.py:35: note: Revealed type is 'def [_T] () -> dataclasses.Field[_T`1]'
main.py:36: note: Revealed type is 'def [T] () -> T`-1'
main.py:37: note: Revealed type is 'def () -> dataclasses.Field[Any]'
main.py:38: note: Revealed type is 'def () -> dataclasses.Field[Any]'
main.py:39: note: Revealed type is 'def () -> Any'
Found 9 errors in 1 file (checked 1 source file)

Sometimes it would be good to store a type identifier in union fields. This would be especially true for sparsely populated data and unions containing str (described below).

Consider:

import typing

import attr
import desert


@attr.dataclass
class A:
    color: str


@attr.dataclass
class B:
    color: str


@attr.dataclass
class C:
    things: typing.List[typing.Union[A, B]]

We then create some instances:

instances = C(
    things=[
        A(color='red'),
        A(color='green'),
        B(color='blue'),
    ],
)

Theoretically that would serialize to (it doesn't seem to so there will presumably be another issue report):

{"things": [{"color": "red"}, {"color": "green"}, {"color": "blue"}]}

Which leaves us wondering what any of the things are.

On the flip side, if we take an example in the tests where there is a field t.Union[int, str] then adjust it to include a custom class after str such as t.Union[int, str, MyClass] then str will 'always' work and MyClass instances will be serialized to a str such as 'MyClass()' (well, for an attrs class with such a repr() anyways).

tl;dr, marshmallow-union has some pretty significant limitations. I got myself confused and started creating a solution over in marshmallow-polyfield Bachmann1234/marshmallow-polyfield#34 which would introduce an extra layer to the serialization and include a string to designate the type of that element. No more guessing.

{
    "things": [
        {
            "type": "A",
            "value": {
                "color": "red",
            }
        },
        {
            "type": "A",
            "value": {
                "color": "green",
            }
        },
        {
            "type": "B",
            "value": {
                "color": "blue",
            }
        }
    ]
}

Perhaps desert is interested in using marshmallow-polyfield for that feature? Perhaps desert would rather the feature be added to marshmallow-union?

In the mean time I'll try to find time to work my way through reporting and possibly fixing some other issues.

I haven't looked into this yet but it made me think of python-attrs/attrs#278.

Did we have any changes recently that didn't make it into the change log? Let's document them. Add them to changelog.d/.breaking.rst or .change.rst etc. See the CONTRIBUTING doc.

Marshmallow supports serializing arbitrary sequences and mappings, so long as they provide the correct interface. Unfortunately, desert doesn't seem to allow generic types out of the box:

The following crashes:

import dataclasses

@dataclasses.dataclass
class D:
    foo: Sequence[str]

schema = desert.schema(D)

Traceback:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dargueta/.pyenv/versions/3.8.6/lib/python3.8/site-packages/desert/__init__.py", line 24, in schema
    return desert._make.class_schema(cls, meta=meta)(many=many)
  File "/Users/dargueta/.pyenv/versions/3.8.6/lib/python3.8/site-packages/desert/_make.py", line 130, in class_schema
    attributes[field.name] = field_for_schema(
  File "/Users/dargueta/.pyenv/versions/3.8.6/lib/python3.8/site-packages/desert/_make.py", line 291, in field_for_schema
    nested = forward_reference or class_schema(typ)
  File "/Users/dargueta/.pyenv/versions/3.8.6/lib/python3.8/site-packages/desert/_make.py", line 105, in class_schema
    raise desert.exceptions.UnknownType(
desert.exceptions.UnknownType: Desert failed to infer the field type for typing.Sequence[str].
Explicitly pass a Marshmallow field type.

Can we add support for these? It seems like the changes would need to be made here and not too much work.

Environment

Python 3.8.6

• desert: 2020.11.18
• marshmallow: 3.13.0
• marshmallow-enum: 1.5.1
• marshmallow-union: 0.1.15.post1

Using an optional field with any custom marshmallow field class cause desert to crash on de-serialization. E.g. a field with b: Optional[B] = desert.ib(BField(), default=None) does not work.

from typing import Optional
import desert
import marshmallow
import attr

class B:
    pass

class BField(marshmallow.fields.Field):
    pass

@attr.s
class A:
    a: int = attr.ib()
    b: Optional[B] = desert.ib(BField(), default=None)

a = A(42)
print(a)

s = desert.schema(A)
d = s.dump(a)
print(d)

a2 = s.load(d)
print(a2)

This results in:

A(a=42, b=None)
{'a': 42, 'b': None}
Traceback (most recent call last):
  File "C:\Svein\Prosjekt\elns\__work\desertbug.py", line 24, in <module>
    a2 = s.load(d)
  File "C:\Svein\Prosjekt\elns\venv\lib\site-packages\marshmallow\schema.py", line 717, in load
    return self._do_load(
  File "C:\Svein\Prosjekt\elns\venv\lib\site-packages\marshmallow\schema.py", line 900, in _do_load
    raise exc
marshmallow.exceptions.ValidationError: {'b': ['Field may not be null.']}

PEP 604, which was implemented in Python 3.10, allows for unions to be specified through using the pipe (union) operator. As the Optional[<type>] type already is essentially a shortcut for Union[<type>,None], the new implementation also means that Union[<type>, None] is able to be written as <type> | None. This confuses desert right now and it fails to be able to make a schema.

The current work-around is to still use the Optional[<type>] syntax, which involves importing Optional from typing. It would be nice if desert was more flexible for this.

Python has a bunch of validation libraries. They're mostly the same, but each one has a different ecosystem.

• Marshmallow
• typesystem
• schematics
• voluptuous
• pydantic
• colander
• cerberus
• jsonschema
• schema

There are even lists of them.

A comment: Competition is good for pushing each one to get better, but it comes at a cost.

• There's a lot of duplicated effort by the maintainers of those libraries.
• Partitioning the users means each user base loses out on some features. For example, users of Marshmallow can use OpenAPI schemas with the apispec library from the Marshmallow ecosystem, but users of Typesystem and Colander can generate HTML forms. As a user, picking one library means losing out on some set of features from the ecosystem, even if the libraries themselves were feature-equivalent.

How could this problem be solved?

• Make one library that's way better than all the others, and then the ancillary ecosystem libraries will move to support it. Requests is like this, maybe? Try to think of better examples of a library emerging as a clear winner due to quality and ecosystem converging on it.

• Provide a built-in library with this functionality, and then the ancillary ecosystem libraries will move to support it. The trouble is that after a few years better alternatives will emerge but the standard library will be frozen. asyncio (and many other Python standard library modules) are like this.

• Define a language-level standard API, and then all the libraries can support it interoperably. async/await and the DB-API PEP are like this.

• Make a library that can convert schemas from each library to each other library. Then whatever feature you need, you can get it by just converting your schema to the other kind. odo is like this for data formats.

Are there more solutions?

Hi,

I am quite new to the marshmallow ecosystem.
I am considering the use of desert as a way to write python API clients by explicitly describing their data structure to ease the deserialisation.

In the process of writing an API Client for a specific purpose one could be confronted to a lot of unnecessary data in API responses that he does not intend to use.
Exploring this data and writing up the schemas while continuously testing, a way to build such code could be to default to marshmallow EXCLUDE for unknown fields.
That way, continuous TDD with "deserialisation of real life JSONS" tests would be possible without ValidationError while not having to describe unused chunks of the data.

Furthermore, continuing to use EXCLUDE after this initial stage of development could be acceptable, if the policy of the API itself is to authorize adding of values for minor changes.
One could want to make its API client future-proof by not breaking when such a change happens on the other side that he does not necessarily controls.

But this behavior is broken when deserialising unknown fields in a List:

from dataclasses import dataclass
from typing import List

import desert
import marshmallow


@dataclass
class Ambiguous:
    firstname: str
    lastname: str


@dataclass
class AmbiguousList:
    id: str
    not_yet_known: List[Ambiguous]


AmbiguousListSchema = desert.schema_class(AmbiguousList, meta={"unknown": marshmallow.EXCLUDE})()

AmbiguousListSchema.loads("""
{
    "id":"789456132",
    "not_yet_known": [
        {
            "firstname": "John",
            "lastname": "Smith"
        },
        {
            "firstname": "Henry",
            "lastname": "Smith",
        }
    ]
}
""")
# Ok

AmbiguousListSchema.loads("""
{
    "id":"789456132",
    "shiny_new_arg": "uninteresting info",
    "not_yet_known": [
        {
            "firstname": "John",
            "lastname": "Smith"
        },
        {
            "firstname": "Henry",
            "lastname": "Smith"
        }
    ]
}
""")
# Still Ok

AmbiguousListSchema.loads("""
{
    "id":"789456132",
    "shiny_new_arg": "uninteresting info",
    "not_yet_known": [
        {
            "firstname": "John",
            "lastname": "Smith"
        },
        {
            "firstname": "Henry",
            "lastname": "Smith",
            "shiny_new_arg": "uninteresting info"
        }
    ]
}
""")
# marshmallow.exceptions.ValidationError: {'not_yet_known': {1: {'shiny_new_arg': ['Unknown field.']}}}

I am not sure what is the library that is responsible. It seems that marshmallow_dataclass shares the same error.

Suppose I have

@attr.dataclass
class A:
    color: str


@attr.dataclass
class C:
    things: typing.List[A]

and I dump some data to disk.

Later I change the code to

@attr.dataclass
class A:
    color: str


@attr.dataclass
class B:
    color: str


@attr.dataclass
class C:
    things: typing.List[typing.Union[A, B]]

and I try to load the data. There's no indication of whether each thing should be an A or a B. The new dataclasses are a superset of the old ones, but the schema loader can't load the old data unambiguously. So then how do I load the data to then dump it again with the tagged union?

• Use A because it's first
• Use B because it's last
• Specify with a keyword argument which to prefer

Or, we could tag even non-union fields at dump-time so this situation never comes up. I guess if we want to be compatible with Serde, we can't make this the only option, but it does seem like a situation to keep in mind.

Originally posted by @python-desert in #36 (comment)

Currently, I think private (i.e. underscore-prefixed) dataclass fields are not added to the schema. We could instead add them or make that decision configurable.

The doctests in the rst files aren't being executed in CI.

marshmallow-code/marshmallow#1702 deprecates **metadata in favor of metadata=.

Assuming in some form GitHub Actions comes into user, #70 or otherwise, we'll probably want to figure out Codecov. It seems that their auto mechanisms don't cover PR's from branches at least and neither does their action (which I think I would rather not use anyways).

codecov/codecov-action#29

I think trio just makes the upload token totally public in general, but let me double check.

The report and codecov envs are called for each Travis job. In the pypy3 job I added it ends up failing while trying to use python3 to create an env. So I removed them and now it fails because basepython is nothing.

What I see this assume is the ability to override with TOXPYTHON. The cost is that you must explicitly set basepython for all cases. I think...

If we want to keep TOXPYTHON around then maybe we swap the report env for {py36,py37,py38,pypy3}-report? Then I think the py.* reference would trigger the existing lines to pick the basepython.

Opinions? Facts?

Mypy fails on desert currently.

Running 32-bit Py 3.8.0 on Win10. desert from master, commit 073a40e

desert does not support types with ellipsis, which is used to indicate a variable length list or tuple with homogeneous types. The following test case:

def test_tuple_ellipsis(module):
    """Test tuple with ellipsis"""

    @module.dataclass
    class A:
        x: t.Tuple[int, ...]

    schema = desert.schema_class(A)()
    dumped = {"x": (1, 2, 3)}
    loaded = A(x=(1, 2, 3))

    assert schema.load(dumped) == loaded
    assert schema.dump(loaded) == dumped
    assert schema.loads(schema.dumps(loaded)) == loaded

Fails on making the schema with:

src\desert\_make.py:107: in class_schema
    raise desert.exceptions.UnknownType(
E   desert.exceptions.UnknownType: Desert failed to infer the field type for Ellipsis.
E   Explicitly pass a Marshmallow field type.

Maybe just some of the permanents? Maybe not? I get I'm doing a bunch of non-feature/bugfix stuff, just trying to get the environment tidied up and red for the permanent redirects muddies reading the logs. I'm happy to do this if it's wanted.

(line    8) redirect  https://github.com/python-desert/desert/issues/28 - with Found to https://github.com/python-desert/desert/pull/28
(line   38) redirect  https://github.com/python-desert/desert/issues/18 - with Found to https://github.com/python-desert/desert/pull/18
(line   22) redirect  https://github.com/python-desert/desert/issues/27 - with Found to https://github.com/python-desert/desert/pull/27
(line   40) redirect  https://github.com/python-desert/desert/issues/19 - with Found to https://github.com/python-desert/desert/pull/19
(line   42) redirect  https://github.com/python-desert/desert/issues/20 - with Found to https://github.com/python-desert/desert/pull/20
(line   72) redirect  https://github.com/python-desert/desert/issues/8 - with Found to https://github.com/python-desert/desert/pull/8
(line   93) redirect  https://github.com/python-desert/desert/issues/2 - with Found to https://github.com/python-desert/desert/pull/2
(line  112) redirect  https://github.com/python-desert/desert/issues/1 - with Found to https://github.com/python-desert/desert/pull/1
(line   52) redirect  http://tox.readthedocs.io/en/latest/install.html - permanently to https://tox.readthedocs.io/en/latest/install.html
(line    3) redirect  https://pypi.org/pypi/desert - permanently to https://pypi.org/project/desert/
(line    3) redirect  https://pypi.org/pypi/desert - permanently to https://pypi.org/project/desert/
(line    3) redirect  https://pypi.org/pypi/desert - permanently to https://pypi.org/project/desert/
(line    3) redirect  https://pypi.org/pypi/desert - permanently to https://pypi.org/project/desert/
(line  134) redirect  https://desert.readthedocs.io/ - with Found to https://desert.readthedocs.io/en/latest/
(line  149) redirect  http://www.attrs.org/ - with Found to http://www.attrs.org/en/stable/
(line    3) redirect  https://codecov.io/github/python-desert/desert/coverage.svg?branch=master - permanently to https://codecov.io/gh/python-desert/desert/branch/master/graphs/badge.svg?branch=master
(line   78) redirect  https://poetry.eustace.io - permanently to https://python-poetry.org/
(line    9) redirect  https://poetry.eustace.io - permanently to https://python-poetry.org/
(line    3) redirect  https://pypi.org/pypi/desert - permanently to https://pypi.org/project/desert/
(line    3) redirect  https://pypi.org/pypi/desert - permanently to https://pypi.org/project/desert/
(line    3) redirect  https://pypi.org/pypi/desert - permanently to https://pypi.org/project/desert/
(line    3) redirect  https://pypi.org/pypi/desert - permanently to https://pypi.org/project/desert/
(line   78) redirect  https://poetry.eustace.io - permanently to https://python-poetry.org/
(line  134) redirect  https://desert.readthedocs.io/ - with Found to https://desert.readthedocs.io/en/latest/
(line  149) redirect  http://www.attrs.org/ - with Found to http://www.attrs.org/en/stable/
(line    3) redirect  https://codecov.io/github/python-desert/desert/coverage.svg?branch=master - permanently to https://codecov.io/gh/python-desert/desert/branch/master/graphs/badge.svg?branch=master

I would think https://desert.readthedocs.io/ would be ok, if an exception to fixing permanent redirects is wanted. Attrs too though it should be https probably. I'm not sure about the codecov ones.

present: https://codecov.io/github/python-desert/desert/coverage.svg?branch=master
redirect: https://codecov.io/gh/python-desert/desert/branch/master/graphs/badge.svg?branch=master
from codecov: https://codecov.io/gh/python-desert/desert/branch/master/graph/badge.svg

All different... and even the redirect and from codecov have graphs vs. graph and also the extra ?branch=master. I would probably try the URL from codecov and see if it redirects... (no URL rewrite in the browser at least)

Maybe every PR should have a changelog entry.

The difference between a commit message and a changelog entry is usually said to be that changelog entries are for users and commit messages are for maintainers. Project-management PRs don't necessarily have any user-facing information so maybe they don't need a changelog message. But they could have one anyway.

Should it be possible to omit typings on fields which are excluded from the schema? Consider the following test. This code currently fails with desert.exceptions.UnknownType: Desert failed to infer the field type for None.

def test_typing_missing():
    """Test exluded field

    """

    @attr.s
    class A:
        x: int = attr.ib()
        y = attr.ib()

    schema = desert.schema_class(A, meta={'exclude': ('y')})()
    loaded = A(x=1, y=None)
    actually_dumped = {"x": 1}

    assert schema.dump(loaded) == actually_dumped

pycli isn't getting blacked because it's not a .py.

1. Add pycli to the black command in tox.ini

• Doesn't help when running manually from the command line

2. Make it .pycli.py.

• ./pycli can't work on Windows as the shebang only works if you have the .py extension and it is associated with the py launcher.

3. Get this into the pyproject.toml

Number 1 only helps when calling tox so it seems inferior. Number 3 solves this issue but leaves another unsolved. Number 2 solves this and the directly-runnable-bootstrapper-on-windows issue (that has no ticket).

Capnproto has a Python
implementation.

Capnproto defines a language for defining a schema. For example,

@0xdbb9ad1f14bf0b36;  # unique file ID, generated by `capnp id`

struct Person {
  name @0 :Text;
  birthdate @3 :Date;

  email @1 :Text;
  phones @2 :List(PhoneNumber);

  struct PhoneNumber {
    number @0 :Text;
    type @1 :Type;

    enum Type {
      mobile @0;
      home @1;
      work @2;
    }
  }
}

struct Date {
  year @0 :Int16;
  month @1 :UInt8;
  day @2 :UInt8;
}

We might consider adding generation
of Capnproto schemas as an alternative to json.

It's possible this would be better suited for a separate library that takes Marshmallow
schemas into Capnproto schemas.

Documentation for

• Our workflow
• Each stage/job/task

Comments in the workflow files would be good.

Came up around from #58/#59.

There are a few problems. You can see them with mypy .. We could also enable some of the stricter mypy flags.

Just got into this library and really like it for deserializing data into attrs objects. Is there a plan for a coming release? It appears there's a lot of new changes since 2020-11-18!

Python3.7+ comes with the package dataclasses built in. Desert fails to install from our internal pypi mirror, as that does not mirror the dataclasses packet anymore.

It seems the entry in this file should be made conditional somehow:

Using type hints that is only pulled in via if TYPE_CHECKING constructs will fail serialization, even if the field is excluded.

from typing import TYPE_CHECKING, Optional
import desert
import attr

if TYPE_CHECKING:
    from twisted.internet.defer import Deferred

@attr.s
class A:
    a: int = attr.ib()
    b: Optional['Deferred'] = attr.ib(default=None, repr=False)

a = A(1)
s = desert.schema(A, meta={'exclude': ('b', )})
d = s.dump(a)

In this example, Deferred is pulled in under the TYPE_CHECKING paradigm. There might be good reasons to pull a type hint this way, e.g. to avoid circular references. This cause desert._make to crash. The expected outcome would be that since the field is excluded in the schema specification, desert wouldn't need its type hint.

Traceback (most recent call last):
  File "C:\sveinse\desert_bug.py", line 15, in <module>
    s = desert.schema(A, meta={'exclude': ('b', )})
  File "C:\sveinse\venv\lib\site-packages\desert\__init__.py", line 24, in schema
    return desert._make.class_schema(cls, meta=meta)(many=many)
  File "C:\sveinse\venv\lib\site-packages\desert\_make.py", line 127, in class_schema
    hints = t.get_type_hints(clazz)
  File "C:\Users\sveinse\AppData\Local\Programs\Python\Python310\lib\typing.py", line 1808, in get_type_hints
    value = _eval_type(value, base_globals, base_locals)
  File "C:\Users\sveinse\AppData\Local\Programs\Python\Python310\lib\typing.py", line 328, in _eval_type
    ev_args = tuple(_eval_type(a, globalns, localns, recursive_guard) for a in t.__args__)
  File "C:\Users\sveinse\AppData\Local\Programs\Python\Python310\lib\typing.py", line 328, in <genexpr>
    ev_args = tuple(_eval_type(a, globalns, localns, recursive_guard) for a in t.__args__)
  File "C:\Users\sveinse\AppData\Local\Programs\Python\Python310\lib\typing.py", line 326, in _eval_type
    return t._evaluate(globalns, localns, recursive_guard)
  File "C:\Users\sveinse\AppData\Local\Programs\Python\Python310\lib\typing.py", line 691, in _evaluate
    eval(self.__forward_code__, globalns, localns),
  File "<string>", line 1, in <module>

The ominously numbered PR #666 has some changes that shouldn't break us but may be worth reviewing. It introduces a new attrs namespace, while retaining backward compatibility on the old attr namespace.

One notable difference in the new one is that slots=True by default.

The requirements files would be better off in a single directory to make them easier to see.

There are at least two issues going on with our handling of recursive types.

One is handling of forward references in type hints. Python's representation inside the typing module seems to be evolving, and the tools for dealing with them (typing_inspect and pytypes) may be still catching up. I think that's what's catching us in #88.

There is also a general issue about the robustness of our union fields.

• Theoretically, a greedy recursive descent should serialize fine in simple cases like the one, but the implementation we're using seems to be having problems. There may be some fixes for marshmallow-union that could address this.
• There are also more complicated situations where greedy strategies will necessarily yield ambiguous parses. We could look into fancier parser generators that could detect and report ambiguities.
• A more explicit union representation would eliminate the issue of ambiguity by simply declaring what types are being represented.

Running on python 3.8.0 on Windows 10, 32-bit version in a venv. desert retrieved from master commit a9b41e1.

desert does not support reading input data containing None from a schema with Optional type. The following test shows the issue:

def test_optional_present(module):
    """Including optional with None set the field."""

    @module.dataclass
    class A:
        x: t.Optional[int]

    data = desert.schema_class(A)().load({"x": None})
    assert data == A(None)

This will fail with the error: marshmallow.exceptions.ValidationError: {'x': ['Field may not be null.']}. The documentation for typing.Optional states that it is equivalent of Union[X, None] indicates that None is valid input to a Optional field.

@sveinse brought up in #python yesterday their use case of a full desert class where the serialization is only used to pass a subset of the attributes to a remote GUI. I think this is presently supported by manually creating a marshmallow field.

do_not_serialize: str = desert.ib(marshmallow.Fields.String(dump_only=False, load_only=False))

But that's pretty unwieldy and WET.

I proposed the following interface that seems pretty straightforward to implement and use.

do_not_serialize: str = desert.ib(dump=False, load=False)

For completeness, I also proposed an alternative that is unlikely to end up a good choice but it relates to my frustration that 'annotations' have been co-opted to exclusively hold 'type hints' instead of retaining a more general role of annotating things. This could include serialization data, documentation strings, probably lots more. But, that's not what anyone does...

do_not_serialize: desert.Ignore[str]

or... something. It's unlikely to be a thing so I didn't put much thought into how it would be made to not inhibit regular type hints checking or any alternatives etc.

As to implementation of the desert.ib() option, I got so far as considering that the marshmallow_field parameter ought to be optional and if not specified the regular inspection processes should be used to create the field.

I'm thinking that the following two attributes should be treated the same way. After that, extra info can be passed to desert.ib() to adjust the otherwise automatic marshmallow field creation.

x: str
y: str = desert.ib()

Or, maybe my limited exposure to desert has left me overlooking existing related features...

What needs to happen for a release

• bump the version in files
• git tag
• generate the changelog from .changelog.d
• merge into master
• publish to PyPI

#100 (comment) shows one workaround right now but the problem with it is that it requires defining the metadata at the location where the class is used, not where the class is defined.

It would be preferable many times to define the metadata on the class itself.

@desert.meta(unknown=marshmallow.EXCLUDE)
@dataclass
class Child:
    name: str

@dataclass
class Parent:
    name: str
    dummy_list: List[str]
    children: List[Child]

This is a lot less error prone in situations where I know that the Child schema might evolve and I want to define directly on it the intent that I don't want my deserialization to break when new fields are added.

Arguably the proposed desert.meta is ambiguous with the existing desert.metadata API so open to suggestions on naming as well...

I'm happy to contribute a PR if I can get some confidence that this will be accepted design-wise.

We should do this whenever our GHA setup is ready.

desert throws an error when a dataclass has more than one subtype other than None

import desert
from dataclasses import dataclass
from typing import Union

from marshmallow import Schema

@dataclass
class MyOptionalKlass:
    a: Union[float, str, None] = 3

schema = desert.schema(MyOptionalKlass)
#> ValueError: too many values to unpack (expected 1)

I think this is to do with the check of is_optional_type() before is_union_type(). I think within the is_optional_type() branch we should create subfields for all remaining subtypes other than NoneType.

Even without the None in the type, the default value of the field is still <marshmallow.missing> rather than 3

Running py 3.8 on win10, 32-bit. Using desert 7ca1e23 from git.

When making a schema from data class A, calling schema.dump(a) to serialize an object a, will fail the serialization if class A contains a method named dump.

def test_calling_dump():
    @attr.s
    class A:
        x: int = attr.ib(default=0)
        def dump(self, *args, **kw):
            raise Exception(f"A.dump({args},{kw}) should not be called")

    schema = desert.schema(A)
    a = A()
    schema.dump(a)

If it happens that the defintion of A.dump() is incompatible with schema.dump(), a very strange error is returned that is confusing at best:

  File "c:\sveinse\myfile\settings.py", line 153, in dumps
    load = schema.dump(self)
TypeError: dump() takes 1 positional argument but 2 were given

The calling of methods in the data class from the schema object can be extended to any arbitrary function names, e.g. foof as shown below. Note that if schema.foof(a) is called, then A.foof will be called with arguments foof(schema_object, self), not adhering to the defined prototype def foof(self,...). This can lead to very unexpected results.

def test_calling_foof():
    @attr.s
    class A:
        x: int = attr.ib(default=0)
        def foof(self, *args, **kw):
            raise Exception(f"A.foof({type(self)},{args},{kw}) should not be called")

    schema = desert.schema(A)
    a = A()
    schema.foof(a)

I don't think both are needed?