Giter VIP home page Giter VIP logo

audobject's People

Contributors

frankenjoe avatar hagenw avatar psibre avatar

Stargazers

 avatar

Watchers

 avatar  avatar  avatar  avatar

audobject's Issues

Allow arguments to be stored as properties not only attributes?

Currently, arguments have to be stored as attributes with the same name in order to be seralized.
It might be the case that for whatever reason we use a property instead of the attribute, but might still want to store that inside the YAML file.
The big question is: does this makes sense or should arguments that are handled by properties never be stored as they are calculated on the fly anyway?
I don't thing it is urgent to answer this soon, but to not forget this discussion (see also audbenchmark!108 (comment 83805)), I created this issue.

Fix handling of kwargs in constructor

Currently, all attributes of an object are serialized if the constructor accepts **kwargs, e.g.:

class Object(audobject.Object):

    def __init__(
            self,
            **kwargs,
    ):
        self.foo = kwargs['foo']
        self.bar = 'not a kwarg'


o = Object(foo='a kwarg')
o.to_yaml_s()
$__main__.Object==1.0.0:
  foo: a kwarg
  bar: not a kwarg

But the expected behavior is:

$__main__.Object==1.0.0:
  foo: a kwarg

Decoding non-keyword arguments fails in derived class

class Base(audobject.Object):
    @audobject.init_decorator(
        resolvers={
            't': audobject.resolver.Tuple,
        }
    )
    def __init__(
            self,
            t: typing.Tuple,
    ):
        self.t = t

s = Base(t).to_yaml_s()
obj = audobject.from_yaml_s(s)
type(obj.t)
<class 'tuple'>

but:

class Child(Base):
    def __init__(
            self,
            t: typing.Tuple,
    ):
        super().__init__(t)

s = Child(t).to_yaml_s()
obj = audobject.from_yaml_s(s)
type(obj.t)
<class 'list'>

In the second case, t is not properly decoded to a tuple. This is because we only decode keyword arguments. Hence, it works if we call the constructor of the base class using keyword arguments, but it would be nicer if this was not necessary:

class Child(Base):
    def __init__(
            self,
            t: typing.Tuple,
    ):
        super().__init__(t=t)

s = Child(t).to_yaml_s()
obj = audobject.from_yaml_s(s)
type(obj.t)
<class 'tuple'>

Serialization is too slow

Serializing an object to YAML and loading it from there adds a big overhead (factor 100), which seems not reasonable to me:

import audobject
import auglib
import numpy as np
import time

def measure_wo_serialization():
    start = time.time()
    for _ in range(100):
        signal = np.zeros((1, 16000))
        transform = auglib.transform.Tone(1000)
        transform(signal)
    end = time.time()
    print(end - start)

def measure_w_serialization():
    start = time.time()
    for _ in range(100):
        signal = np.zeros((1, 16000))
        transform = auglib.transform.Tone(1000)
        transform = audobject.from_yaml_s(transform.to_yaml_s(include_version=False))
        transform(signal)
    end = time.time()
    print(end - start)

Then we get:

>>> measure_wo_serialization()
0.17473506927490234
>>> measure_w_serialization()
18.60582685470581

Deprecated warning

Line : decorator.py:57, throws a warning

UserWarning: TupleResolver is deprecated and will be removed with version 1.0.0. Use resolver.Tuple instead.
resolver_obj = resolve()

OSError: could not get source code

When we want to serialize a function we use inspect.getsource(func) to get the source code of the function. This, however, fails for dynamically defined functions. Consequently, we get an error if we try to serialize the function of an object that has already been loaded from YAML:

class MyObjectWithFunction(audobject.Object):

    @audobject.init_decorator(
        resolvers={
            'func': audobject.resolver.Function,
        }
    )
    def __init__(
            self,
            func: typing.Callable,
    ):
        self.func = func

    def __call__(self, *args, **kwargs):
        return self.func(*args, **kwargs)


def add(a, b):
    return a + b


o = MyObjectWithFunction(add)
o_yaml = o.to_yaml_s()
o2 = audobject.from_yaml_s(o_yaml)

# cannot inspect code of a dynamically defined function
o2.to_yaml_s(s)
OSError: could not get source code

Enhance DX & Type Safety with Typed Loading Methods

Currently the loading methods (e.g. audobject.from_yaml_s) return an instance of audobject.Object, causing the loss of specific type information for LSPs and tools like mypy.

It might be a good idea to add an optional type parameter (e.g. instance_of) to improve static analysis and type verification.
In addition, this could ensure that the loaded object is actually an instance of the expected type, providing an additional layer of safety and reducing potential runtime errors.

This could default to instance_of=audobject.Object such that full backwards compatibility should be provided (assuming no parameter with the same name was passed).

Example with the loss of typing information:

import audobject


__version__ = "1.0.0"  # pretend we have a package version


class TestObject(audobject.Object):
    def some_method(self, some_arg: str) -> None:
        print(some_arg)


test_object = TestObject()  # test_object is an instance of TestObject
test_object.some_method("hello")  # LSP / mypy information avaliable here

test_yaml = test_object.to_yaml_s()

test_object2 = audobject.from_yaml_s(
    test_yaml
)  # test_object2 is an instance of TestObject but only known as audobject.Object

test_object2.some_method(123)  # no LSP / mypy information available here

Example with added typing information:

from typing import Type, TypeVar


T = TypeVar("T", bound=audobject.Object)


def from_yaml_s_typed(
    yaml_s: str,
    instance_of: Type[T] = audobject.Object,
    **kwargs,
) -> T:
    instance = audobject.from_yaml_s(yaml_s, **kwargs)
    if not isinstance(instance, instance_of):
        raise ValueError(...)
    return instance


test_object3 = from_yaml_s_typed(
    test_yaml,
    instance_of=TestObject,
)  # test_object3 is an instance of TestObject and known as such

test_object3.some_method(123)  # LSP / mypy information available here
# mypy error: Argument 1 to "some_method" of "TestObject" has incompatible type "int"; expected "str"  [arg-type]

`root` not supported as name of hidden attribute

Due to these lines which were recently added and I don't quite understand it is now not possible to do the following:

class Foo(audobject.Object):
@audobject.init_decorator(hide=['root'])
def __init__(self, root: str = None):
     self.root = root

bar = Foo('./test')
print(bar.root)  # shows './test'
bar = audobject.from_yaml_s(bar.to_yaml_s(), root='./test')
print(bar.root)  # shows None

This means that root can no longer be a hidden argument that the user would not store as YAML but define during loading. As it's kind of a common name, maybe it's not ideal to use it like this in audobject? Anyways, I guess you guys are aware of it but it gave me some trouble recently..Maybe you want to add a warning in the docs?

Error with latest importlib-metadata on MacOS

When I run audb.load(...) I get an error on MacOS that seems to be related to importlib-metadata from this package here:

Error message
Traceback (most recent call last):
  File "/Library/Python/3.9/site-packages/conans/errors.py", line 34, in conanfile_exception_formatter
    yield
  File "/Library/Python/3.9/site-packages/conans/client/conanfile/build.py", line 16, in run_build_method
    conanfile.build()
  File "/Users/vagrant/builds/Zq9KZGs5/0/project/devaice/components/cnn10-scene-model/test_package/conanfile.py", line 32, in build
    self._build_utils.get_reference_data(self.conan_data["model"]["uid"], self.build_folder, get_values_func, None, self.output)
  File "/tmp/ci-jobs/399674/.conan/data/devaice-build-utils/latest/audeering/develop/export/conanfile.py", line 303, in get_reference_data
    download_audb_database(entry, build_folder, output=output)
  File "/tmp/ci-jobs/399674/.conan/data/devaice-build-utils/latest/audeering/develop/export/conanfile.py", line 245, in download_audb_database
    db = audb.load(entry["database"], version=entry["version"], media=entry["media"], full_path=False, verbose=True)
  File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/audb/core/load.py", line 1047, in load
    db_root = database_cache_root(name, version, cache_root, flavor)
  File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/audb/core/cache.py", line 44, in database_cache_root
    flavor.path(name, version),
  File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/audb/core/flavor.py", line 136, in path
    return os.path.join(name, version, self.short_id)
  File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/audb/core/flavor.py", line 148, in short_id
    return self.id[-8:]
  File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/audobject/core/object.py", line 171, in id
    string = self.to_yaml_s(include_version=False)
  File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/audobject/core/object.py", line 337, in to_yaml_s
    return yaml.dump(self.to_dict(include_version=include_version))
  File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/audobject/core/object.py", line 271, in to_dict
    name = utils.create_class_key(self.__class__, include_version)
  File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/audobject/core/utils.py", line 34, in create_class_key
    package_names = packages_distributions()
  File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/importlib_metadata/__init__.py", line 967, in packages_distributions
    for pkg in _top_level_declared(dist) or _top_level_inferred(dist):
  File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/importlib_metadata/__init__.py", line 1010, in _top_level_inferred
    opt_names = set(map(_get_toplevel_name, always_iterable(dist.files)))
  File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/importlib_metadata/__init__.py", line 511, in files
    return skip_missing_files(
  File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/importlib_metadata/_functools.py", line 102, in wrapper
    return func(param, *args, **kwargs)
  File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/importlib_metadata/__init__.py", line 509, in skip_missing_files
    return list(filter(lambda path: path.locate().exists(), package_paths))
  File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/importlib_metadata/__init__.py", line 509, in <lambda>
    return list(filter(lambda path: path.locate().exists(), package_paths))
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/pathlib.py", line 1414, in exists
    self.stat()
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/pathlib.py", line 1222, in stat
    return self._accessor.stat(self)
PermissionError: [Errno 13] Permission denied: '/Library/Python/3.9/site-packages/../../../../var/root/Library/Caches/com.apple.python/Library/Python/3.9/site-packages/virtualenv/__init__.cpython-39.pyc'

Python version is 3.9. I cannot reproduce it on Linux unfortunately..

Steps to reproduce:

  • Install audb: pip3 install audb

  • Check latest importlib-metadata is installed (as from a fresh install of audb): pip3 install -U importlib-metadata

  • Run in Python:

    >>> from audb.core.flavor import Flavor
    >>> flavor = Flavor(sampling_rate=16000)
    >>> flavor.path("test", "1.0.0")
    
  • Result is the error above.

When installing older importlib-metadata it works fine:

  • pip3 install importlib-metadata==5.0.0

  • Run in Python:

    >>> from audb.core.flavor import Flavor
    >>> flavor = Flavor(sampling_rate=16000)
    >>> flavor.path("test", "1.0.0")
    
  • Result is:

    'test/1.0.0/9be30deb'

What to do if a callable object is passed instead of a function

Since a callable function looks like a function, a user may try the following:

class MyCallableObject:
    def __call__(self, x):
        return x * x


func = MyCallableObject()


class MyObjectWithFunction(audobject.Object):

    @audobject.init_decorator(
        resolvers={
            'func': audobject.resolver.Function,
        }
    )
    def __init__(
            self,
            func: typing.Callable,
    ):
        self.func = func

    def __call__(self, x):
        return self.func(x)


o = MyObjectWithFunction(func)
o(3)
9

However, trying to serialize o fails:

audobject.from_yaml_s(o.to_yaml_s())
AttributeError: 'MyCallableObject' object has no attribute '__name__'

Generally, there is no solution for this and we should raise a proper error saying it's not possible to serialize a callable object.

However, there is an exception where it should be possible and that is when it derives from Object, i.e.:

class MyCallableObject(audobject.Object):
    def __call__(self, x):
        return x * x

In that case we can simply dump the YAML representation of the object, which will allow us to recreate the object later on.

BUG: Cannot inherit borrow, hide and resolvers

When deriving from a class that already sets borrow, hide or resolvers, we lose this information in the child class.

Here's an example for borrow:

class Parent(audobject.Object):
    @audobject.init_decorator(
        hide=[            
            'parent',
        ],
    )
    def __init__(
            self,
            parent: bool = False,
    ):
        self.parent = parent


class Child(Parent):
    @audobject.init_decorator(
        hide=[
            'child',            
        ],
    )
    def __init__(
            self,
            child: bool = False,
            parent: bool = False,
    ):        
        super().__init__(parent=parent)
        self.child = child

c = Child()
print(c.to_yaml_s())
$__main__.Child==1.0.0:
  child: false

We can see that child is not hidden as we would expect.

Default arguments are not preserved with serialized functions

The following is working as expected:

class MyObjectWithFunction(audobject.Object):

    @audobject.init_decorator(
        resolvers={
            'func': audobject.resolver.Function,
        }
    )
    def __init__(
            self,
            func: typing.Callable,
    ):
        self.func = func

    def __call__(self, *args, **kwargs):
        return self.func(*args, **kwargs)


def add(a, b=1):
    return a + b


o = MyObjectWithFunction(add)
o(1)  # uses b=1
2

But after serialization the default argument for b is lost:

o_yaml = o.to_yaml_s()
o2 = audobject.from_yaml_s(o_yaml)
o2(1)
TypeError: add() missing 1 required positional argument: 'b'

Move static Object functions to module level?

If a user does something like:

transform = audsp.Spectrogram(...)
transform.to_yaml('transform.yaml')

it is obvious what will happen.

But to load the transform the user can do:

audobject.Object.from_yaml('transform.yaml')
audsp.Spectrogram.from_yaml('transform.yaml')
audsp.Cepstrogram.from_yaml('transform.yaml')

which all return

{'$audsp.core.spectrogram.Spectrogram': {'win_dur': 0.2, 'hop_dur': 0.1, 'power': False, 'num_fft': 4096, 'center': False, 'reflect': False, 'zero_phase': False, 'audspec': None, 'unit': 'seconds', 'channels': [0], 'mixdown': False, 'resample': False, 'sampling_rate': 16000}}

Especially audsp.Cepstrogram.from_yaml('transform.yaml') is very misleading.

I would propose that me move the static functions

  • audobject.Object.from_dict()
  • audobject.Object.from_yaml()
  • audobject.Object.from_yaml_s()
    to the module level:
  • audobject.from_dict()
  • audobject.from_yaml()
  • audobject.from_yaml_s()

Or maybe add a load, e.g. audobject.load_from_dict(), ...

In the above example the code would then read:

audobject.from_yaml('transform')

and this is the only way to load the transform.
The only downside is that the user has to import audobject on top/instead of audsp.

Sub-package for resolver classes

Now that we already have four resolver classes (and there may be more in the future):

  • FilePathResolver
  • FunctionResolver
  • TupleResolver
  • TypeResolver

I wonder if it would improve the interface if we group them in an own subpackage:

  • resolver.FilePath
  • resolver.Function
  • resolver.Tuple
  • resolver.Type

And ValueResolver we could rename to resolver.Base to make it more explicit that this is the base class.

@hagenw what's your opinion?

Do not encode value with resolver if it is None

Currently, encoding a value that is None will raise an error:

class MyObject(audobject.Object):

    @audobject.init_decorator(
        resolvers={
            'arg': audobject.resolver.Tuple,
        }
    )
    def __init__(
            self,
            arg: typing.Tuple = None,
    ):
        super().__init__()
        self.arg = arg


MyObject().to_yaml_s(include_version=False)
TypeError: 'NoneType' object is not iterable

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.