audeering / audobject Goto Github PK
View Code? Open in Web Editor NEWGeneric Python interface for serializing objects to YAML
Home Page: https://audeering.github.io/audobject/
License: Other
Generic Python interface for serializing objects to YAML
Home Page: https://audeering.github.io/audobject/
License: Other
Currently, arguments have to be stored as attributes with the same name in order to be seralized.
It might be the case that for whatever reason we use a property instead of the attribute, but might still want to store that inside the YAML file.
The big question is: does this makes sense or should arguments that are handled by properties never be stored as they are calculated on the fly anyway?
I don't thing it is urgent to answer this soon, but to not forget this discussion (see also audbenchmark!108 (comment 83805)), I created this issue.
Currently, all attributes of an object are serialized if the constructor accepts **kwargs
, e.g.:
class Object(audobject.Object):
def __init__(
self,
**kwargs,
):
self.foo = kwargs['foo']
self.bar = 'not a kwarg'
o = Object(foo='a kwarg')
o.to_yaml_s()
$__main__.Object==1.0.0:
foo: a kwarg
bar: not a kwarg
But the expected behavior is:
$__main__.Object==1.0.0:
foo: a kwarg
class Base(audobject.Object):
@audobject.init_decorator(
resolvers={
't': audobject.resolver.Tuple,
}
)
def __init__(
self,
t: typing.Tuple,
):
self.t = t
s = Base(t).to_yaml_s()
obj = audobject.from_yaml_s(s)
type(obj.t)
<class 'tuple'>
but:
class Child(Base):
def __init__(
self,
t: typing.Tuple,
):
super().__init__(t)
s = Child(t).to_yaml_s()
obj = audobject.from_yaml_s(s)
type(obj.t)
<class 'list'>
In the second case, t
is not properly decoded to a tuple. This is because we only decode keyword arguments. Hence, it works if we call the constructor of the base class using keyword arguments, but it would be nicer if this was not necessary:
class Child(Base):
def __init__(
self,
t: typing.Tuple,
):
super().__init__(t=t)
s = Child(t).to_yaml_s()
obj = audobject.from_yaml_s(s)
type(obj.t)
<class 'tuple'>
At the moment the tests are failing for Python 3.11, see #68
Serializing an object to YAML and loading it from there adds a big overhead (factor 100), which seems not reasonable to me:
import audobject
import auglib
import numpy as np
import time
def measure_wo_serialization():
start = time.time()
for _ in range(100):
signal = np.zeros((1, 16000))
transform = auglib.transform.Tone(1000)
transform(signal)
end = time.time()
print(end - start)
def measure_w_serialization():
start = time.time()
for _ in range(100):
signal = np.zeros((1, 16000))
transform = auglib.transform.Tone(1000)
transform = audobject.from_yaml_s(transform.to_yaml_s(include_version=False))
transform(signal)
end = time.time()
print(end - start)
Then we get:
>>> measure_wo_serialization()
0.17473506927490234
>>> measure_w_serialization()
18.60582685470581
Line : decorator.py:57
, throws a warning
UserWarning: TupleResolver is deprecated and will be removed with version 1.0.0. Use resolver.Tuple instead.
resolver_obj = resolve()
When we want to serialize a function we use inspect.getsource(func)
to get the source code of the function. This, however, fails for dynamically defined functions. Consequently, we get an error if we try to serialize the function of an object that has already been loaded from YAML:
class MyObjectWithFunction(audobject.Object):
@audobject.init_decorator(
resolvers={
'func': audobject.resolver.Function,
}
)
def __init__(
self,
func: typing.Callable,
):
self.func = func
def __call__(self, *args, **kwargs):
return self.func(*args, **kwargs)
def add(a, b):
return a + b
o = MyObjectWithFunction(add)
o_yaml = o.to_yaml_s()
o2 = audobject.from_yaml_s(o_yaml)
# cannot inspect code of a dynamically defined function
o2.to_yaml_s(s)
OSError: could not get source code
Currently the loading methods (e.g. audobject.from_yaml_s
) return an instance of audobject.Object
, causing the loss of specific type information for LSPs and tools like mypy.
It might be a good idea to add an optional type parameter (e.g. instance_of
) to improve static analysis and type verification.
In addition, this could ensure that the loaded object is actually an instance of the expected type, providing an additional layer of safety and reducing potential runtime errors.
This could default to instance_of=audobject.Object
such that full backwards compatibility should be provided (assuming no parameter with the same name was passed).
import audobject
__version__ = "1.0.0" # pretend we have a package version
class TestObject(audobject.Object):
def some_method(self, some_arg: str) -> None:
print(some_arg)
test_object = TestObject() # test_object is an instance of TestObject
test_object.some_method("hello") # LSP / mypy information avaliable here
test_yaml = test_object.to_yaml_s()
test_object2 = audobject.from_yaml_s(
test_yaml
) # test_object2 is an instance of TestObject but only known as audobject.Object
test_object2.some_method(123) # no LSP / mypy information available here
from typing import Type, TypeVar
T = TypeVar("T", bound=audobject.Object)
def from_yaml_s_typed(
yaml_s: str,
instance_of: Type[T] = audobject.Object,
**kwargs,
) -> T:
instance = audobject.from_yaml_s(yaml_s, **kwargs)
if not isinstance(instance, instance_of):
raise ValueError(...)
return instance
test_object3 = from_yaml_s_typed(
test_yaml,
instance_of=TestObject,
) # test_object3 is an instance of TestObject and known as such
test_object3.some_method(123) # LSP / mypy information available here
# mypy error: Argument 1 to "some_method" of "TestObject" has incompatible type "int"; expected "str" [arg-type]
With the new option to auto-install missing packages, it would be good to store the package name if it differs from the module name.
See: #35 (comment)
Currently we use it at
Lines 61 to 62 in 4a99629
and at
audobject/audobject/core/parameter.py
Lines 107 to 108 in 4a99629
Due to these lines which were recently added and I don't quite understand it is now not possible to do the following:
class Foo(audobject.Object):
@audobject.init_decorator(hide=['root'])
def __init__(self, root: str = None):
self.root = root
bar = Foo('./test')
print(bar.root) # shows './test'
bar = audobject.from_yaml_s(bar.to_yaml_s(), root='./test')
print(bar.root) # shows None
This means that root
can no longer be a hidden argument that the user would not store as YAML but define during loading. As it's kind of a common name, maybe it's not ideal to use it like this in audobject
? Anyways, I guess you guys are aware of it but it gave me some trouble recently..Maybe you want to add a warning in the docs?
Would be nice to have an example in the docstring of the TupleResolver
class.
Properly serializing functions only works under specific conditions. E.g. a function most not reference other local functions or objects. It would be good to include a more detailed discussion what is possible and what are the limitations.
When I run audb.load(...)
I get an error on MacOS that seems to be related to importlib-metadata
from this package here:
Traceback (most recent call last):
File "/Library/Python/3.9/site-packages/conans/errors.py", line 34, in conanfile_exception_formatter
yield
File "/Library/Python/3.9/site-packages/conans/client/conanfile/build.py", line 16, in run_build_method
conanfile.build()
File "/Users/vagrant/builds/Zq9KZGs5/0/project/devaice/components/cnn10-scene-model/test_package/conanfile.py", line 32, in build
self._build_utils.get_reference_data(self.conan_data["model"]["uid"], self.build_folder, get_values_func, None, self.output)
File "/tmp/ci-jobs/399674/.conan/data/devaice-build-utils/latest/audeering/develop/export/conanfile.py", line 303, in get_reference_data
download_audb_database(entry, build_folder, output=output)
File "/tmp/ci-jobs/399674/.conan/data/devaice-build-utils/latest/audeering/develop/export/conanfile.py", line 245, in download_audb_database
db = audb.load(entry["database"], version=entry["version"], media=entry["media"], full_path=False, verbose=True)
File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/audb/core/load.py", line 1047, in load
db_root = database_cache_root(name, version, cache_root, flavor)
File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/audb/core/cache.py", line 44, in database_cache_root
flavor.path(name, version),
File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/audb/core/flavor.py", line 136, in path
return os.path.join(name, version, self.short_id)
File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/audb/core/flavor.py", line 148, in short_id
return self.id[-8:]
File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/audobject/core/object.py", line 171, in id
string = self.to_yaml_s(include_version=False)
File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/audobject/core/object.py", line 337, in to_yaml_s
return yaml.dump(self.to_dict(include_version=include_version))
File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/audobject/core/object.py", line 271, in to_dict
name = utils.create_class_key(self.__class__, include_version)
File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/audobject/core/utils.py", line 34, in create_class_key
package_names = packages_distributions()
File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/importlib_metadata/__init__.py", line 967, in packages_distributions
for pkg in _top_level_declared(dist) or _top_level_inferred(dist):
File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/importlib_metadata/__init__.py", line 1010, in _top_level_inferred
opt_names = set(map(_get_toplevel_name, always_iterable(dist.files)))
File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/importlib_metadata/__init__.py", line 511, in files
return skip_missing_files(
File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/importlib_metadata/_functools.py", line 102, in wrapper
return func(param, *args, **kwargs)
File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/importlib_metadata/__init__.py", line 509, in skip_missing_files
return list(filter(lambda path: path.locate().exists(), package_paths))
File "/Users/vagrant/Library/Python/3.9/lib/python/site-packages/importlib_metadata/__init__.py", line 509, in <lambda>
return list(filter(lambda path: path.locate().exists(), package_paths))
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/pathlib.py", line 1414, in exists
self.stat()
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/pathlib.py", line 1222, in stat
return self._accessor.stat(self)
PermissionError: [Errno 13] Permission denied: '/Library/Python/3.9/site-packages/../../../../var/root/Library/Caches/com.apple.python/Library/Python/3.9/site-packages/virtualenv/__init__.cpython-39.pyc'
Python version is 3.9. I cannot reproduce it on Linux unfortunately..
Install audb: pip3 install audb
Check latest importlib-metadata
is installed (as from a fresh install of audb): pip3 install -U importlib-metadata
Run in Python:
>>> from audb.core.flavor import Flavor
>>> flavor = Flavor(sampling_rate=16000)
>>> flavor.path("test", "1.0.0")
Result is the error above.
When installing older importlib-metadata
it works fine:
pip3 install importlib-metadata==5.0.0
Run in Python:
>>> from audb.core.flavor import Flavor
>>> flavor = Flavor(sampling_rate=16000)
>>> flavor.path("test", "1.0.0")
Result is:
'test/1.0.0/9be30deb'
Since a callable function looks like a function, a user may try the following:
class MyCallableObject:
def __call__(self, x):
return x * x
func = MyCallableObject()
class MyObjectWithFunction(audobject.Object):
@audobject.init_decorator(
resolvers={
'func': audobject.resolver.Function,
}
)
def __init__(
self,
func: typing.Callable,
):
self.func = func
def __call__(self, x):
return self.func(x)
o = MyObjectWithFunction(func)
o(3)
9
However, trying to serialize o
fails:
audobject.from_yaml_s(o.to_yaml_s())
AttributeError: 'MyCallableObject' object has no attribute '__name__'
Generally, there is no solution for this and we should raise a proper error saying it's not possible to serialize a callable object.
However, there is an exception where it should be possible and that is when it derives from Object
, i.e.:
class MyCallableObject(audobject.Object):
def __call__(self, x):
return x * x
In that case we can simply dump the YAML representation of the object, which will allow us to recreate the object later on.
In addition to id
add a property short_id
to Object
, which returns only the last 8 digits of the full ID.
See https://audeering.github.io/audb/api.html#audb.Flavor.short_id
When deriving from a class that already sets borrow
, hide
or resolvers
, we lose this information in the child class.
Here's an example for borrow
:
class Parent(audobject.Object):
@audobject.init_decorator(
hide=[
'parent',
],
)
def __init__(
self,
parent: bool = False,
):
self.parent = parent
class Child(Parent):
@audobject.init_decorator(
hide=[
'child',
],
)
def __init__(
self,
child: bool = False,
parent: bool = False,
):
super().__init__(parent=parent)
self.child = child
c = Child()
print(c.to_yaml_s())
$__main__.Child==1.0.0:
child: false
We can see that child
is not hidden as we would expect.
The following is working as expected:
class MyObjectWithFunction(audobject.Object):
@audobject.init_decorator(
resolvers={
'func': audobject.resolver.Function,
}
)
def __init__(
self,
func: typing.Callable,
):
self.func = func
def __call__(self, *args, **kwargs):
return self.func(*args, **kwargs)
def add(a, b=1):
return a + b
o = MyObjectWithFunction(add)
o(1) # uses b=1
2
But after serialization the default argument for b
is lost:
o_yaml = o.to_yaml_s()
o2 = audobject.from_yaml_s(o_yaml)
o2(1)
TypeError: add() missing 1 required positional argument: 'b'
If a user does something like:
transform = audsp.Spectrogram(...)
transform.to_yaml('transform.yaml')
it is obvious what will happen.
But to load the transform the user can do:
audobject.Object.from_yaml('transform.yaml')
audsp.Spectrogram.from_yaml('transform.yaml')
audsp.Cepstrogram.from_yaml('transform.yaml')
which all return
{'$audsp.core.spectrogram.Spectrogram': {'win_dur': 0.2, 'hop_dur': 0.1, 'power': False, 'num_fft': 4096, 'center': False, 'reflect': False, 'zero_phase': False, 'audspec': None, 'unit': 'seconds', 'channels': [0], 'mixdown': False, 'resample': False, 'sampling_rate': 16000}}
Especially audsp.Cepstrogram.from_yaml('transform.yaml')
is very misleading.
I would propose that me move the static functions
audobject.Object.from_dict()
audobject.Object.from_yaml()
audobject.Object.from_yaml_s()
audobject.from_dict()
audobject.from_yaml()
audobject.from_yaml_s()
Or maybe add a load
, e.g. audobject.load_from_dict()
, ...
In the above example the code would then read:
audobject.from_yaml('transform')
and this is the only way to load the transform.
The only downside is that the user has to import audobject
on top/instead of audsp
.
We currently have no way to find out if an object was loaded from YAML. But this could be useful, e.g. to suppress deprecation warnings in that case.
Now that we already have four resolver classes (and there may be more in the future):
FilePathResolver
FunctionResolver
TupleResolver
TypeResolver
I wonder if it would improve the interface if we group them in an own subpackage:
resolver.FilePath
resolver.Function
resolver.Tuple
resolver.Type
And ValueResolver
we could rename to resolver.Base
to make it more explicit that this is the base class.
@hagenw what's your opinion?
Currently, encoding a value that is None
will raise an error:
class MyObject(audobject.Object):
@audobject.init_decorator(
resolvers={
'arg': audobject.resolver.Tuple,
}
)
def __init__(
self,
arg: typing.Tuple = None,
):
super().__init__()
self.arg = arg
MyObject().to_yaml_s(include_version=False)
TypeError: 'NoneType' object is not iterable
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.