omry / omegaconf Goto Github PK
View Code? Open in Web Editor NEWFlexible Python configuration system. The last one you will ever need.
License: BSD 3-Clause "New" or "Revised" License
Flexible Python configuration system. The last one you will ever need.
License: BSD 3-Clause "New" or "Revised" License
Python 2.7 is reaching end of life in 1/1/2020
Some yaml files are organizing as the following, could you please consider to support that ?
Like the followling from the official documentation https://pyyaml.org/wiki/PyYAMLDocumentation
Aliases
Note that PyYAML does not yet support recursive objects.
Using YAML you may represent objects of arbitrary graph-like structures. If you want to refer to the same object from different parts of a document, you need to use anchors and aliases.
Anchors are denoted by the & indicator while aliases are denoted by ``. For instance, the document
left hand: &A
name: The Bastard Sword of Eowyn
weight: 30
right hand: *A
expresses the idea of a hero holding a heavy sword in both hands.
PyYAML now fully supports recursive objects. For instance, the document
&A [ *A ]
will produce a list object containing a reference to itself.
A dict is supposed to have a .copy() method (https://docs.python.org/3/library/stdtypes.html#dict.copy). For an OmegaConf object to be compatible with a dict's API, it should also support the copy() method -
>>> from omegaconf import OmegaConf
>>> conf = OmegaConf.create({"a": 2, "b": [1,2,3]})
>>> conf
{'a': 2, 'b': [1, 2, 3]}
>>> conf.copy()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object is not callable
use case:
config.yaml
foo:
list: [1,2,3]
bar: ${foo.list}
The associated PR (#86 ) is adding support for initializing OmegaConf from class files and objects.
Details below
Config classes and objects (dataclass or attrs classes) can be converted to OmegaConf object.
The following value types are supported:
foo=II("bar")
: Foo inherits the type and value of bar at runtime.foo=II("http://${host}:${port}/")
: foo is always a string. actual value is determined by host and portfoo=II("env:USER")
: foo is evaluated on access, the type and value is determined by the function (env in this case)Example for bool types (The same is supported for int, str, float and enum):
@dataclass
class BoolConfig:
with_default: bool = True
null_default: Optional[bool] = None
mandatory_missing: bool = MISSING
interpolation: bool = II("with_default")
the typing information in them is available and acted on at runtime (composition and command line overrides are at runtime).
type annotations can be used to get static type checking.
Currently the following is supported for both dataclasses and attr classes.
Examples below are @dataclasses. attr classes are similar.
@dataclass
class Database:
name: str = "demo_app_db"
# A an untyped list. will accept any type that can be represented by OmegaConf
tables: List = field(default_factory=lambda: ["table1", "table2"])
port: int = 3306
# A list of integers, will reject anything that cannot be converted to int at runtime
admin_ports: List[int] = field(default_factory=lambda: [8080, 1080])
# A dict of string -> int, will reject any value that cannot be converted to int at runtime
error_levels: Dict[str, int] = field(
default_factory=lambda: {"info": 0, "error": 10}
)
This class, and objects of this class can be used to construct an OmegaConf object:
# Create from a class
conf1 = OmegaConf.create(Database)
# Create from an object
conf2 = OmegaConf.create(Database(port=3307))
Python type annotation can then be used on the config object to static type checking:
def foo(cfg : Database):
cfg.port = 10 # passes static type check
cfg.port = "nope" # fails static type check
conf.merge_with_dotlist(['port=30']) # success
conf.merge_with_dotlist(['port=nope']) # runtime error
conf.pork = 80 # fail
conf: Database = OmegaConf.create(Database)
conf.admin_ports[0] = 999 # okay
assert conf.admin_ports[0] == 999
conf.admin_ports[0] = "1000" # also ok!
assert conf.admin_ports[0] == 1000
with pytest.raises(ValidationError):
conf.admin_ports[0] = "fail"
with pytest.raises(ValidationError):
conf.error_levels["info"] = "fail"
class Protocol(Enum):
HTTP = 0
HTTPS = 1
@dataclass
class ServerConfig:
# Nested configs are by value.
# This will be expanded to default values for Database
db1: Database = Database()
# This will be expanded to the values in the created Database instance
db2: Database = Database(tables=["table3", "table4"])
host: str = "localhost"
port: int = 80
website: str = MISSING
protocol: Protocol = MISSING
debug: bool = False
conf: ServerConfig = OmegaConf.create(ServerConfig(protocol=Protocol.HTTP))
assert conf.protocol == Protocol.HTTP
# The HTTPS string is converted to Protocol.HTTPS per the enum name
conf.protocol = "HTTPS"
assert conf.protocol == Protocol.HTTPS
# The value 0 is converted to Protocol.HTTP per the enum value
conf.protocol = 0
assert conf.protocol == Protocol.HTTP
# Enum names are care sensitive
with pytest.raises(ValidationError):
conf.protocol = "https"
for value in (True, False, "1", "http", 1.0):
with pytest.raises(ValidationError):
conf.protocol = value
@dataclass(frozen=True)
class FrozenClass:
x: int = 10
list: List = field(default_factory=lambda: [1, 2, 3])
conf = OmegaConf.create(FrozenClass)
with pytest.raises(ReadonlyConfigError):
conf.x = 20
# Read-only flag is recursive
with pytest.raises(ReadonlyConfigError):
conf.list[0] = 20
for a nested config:
foo:
bar:
baz: 10
if foo is struct, and we do
with open_dict(cfg.foo.bar):
bar.baz = 20
the struct flag of foo.bar would be changed to True.
Since Config is a dynamic data type, I want to minimize the functions that exists on it to the bare minimum.
A user can easily create a save field that would potentially conflict with the save() function.
A future version would remove config.save() completely.
In struct mode, accessing an invalid attribute through getattr() should result in an AttributeError and not in a KeyError.
See facebookresearch/hydra#266 (Which is actually one bug in Hydra and one in OmegaConf).
c.get('not_there', 'default_value')
throws if c is in struct mode.
Change semantics of key in cfg
to be reasonable when key points to '???', or
is an interpolation that points to '???', or an interpolation that cannot be resolved.
desired behavior:
@pytest.mark.parametrize("conf,key,expected", [
({"a": 1, "b": {}}, "a", True),
({"a": 1, "b": {}}, "b", True),
({"a": 1, "b": {}}, "c", False),
({"a": 1, "b": "${a}"}, "b", True),
({"a": 1, "b": "???"}, "b", False),
({"a": 1, "b": "???", "c": "${b}"}, "c", False),
({"a": 1, "b": "${not_found}"}, "b", False),
])
def test_in_dict(conf, key, expected):
conf = OmegaConf.create(conf)
ret = key in conf
assert ret == expected
It breaks at # Access and manipulation Input yaml file:
I get an invalid syntax
error
when I use the cfg's property such as cfg.kernel_sizes, I what to compare cfg.kernel_sizes with list, tuple, or int
my code is that:
if isinstance(cfg.kernel_sizes, (list, tuple, int)):
...
but the type(cfg.kernel_sizes) is omegaconf.listconfig.ListConfig, not list, so I can't get the true judgement
I want to know how to make the types become list, dict and other standard types
OmegaConf.create(dict(foo={}))
expanding foo results in a stack trace in the debugger.
OmegaConf runs into issues when the config object is constructed by incrementally adding new OmegaConf objects.
To reproduce:
conf = OmegaConf.create()
conf.a = 1
conf.b = OmegaConf.create()
conf.b.c = "${a}"
With conf
above, trying to access conf.b.c
gets me the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/private/home/kchua13/.conda/envs/hierarchy/lib/python3.6/site-packages/omegaconf/dictconfig.py", line 72, in __getattr__
return self.get(key=key, default_value=None)
File "/private/home/kchua13/.conda/envs/hierarchy/lib/python3.6/site-packages/omegaconf/dictconfig.py", line 83, in get
return self._resolve_with_default(key=key, value=self.get_node(key), default_value=default_value)
File "/private/home/kchua13/.conda/envs/hierarchy/lib/python3.6/site-packages/omegaconf/config.py", line 131, in _resolve_with_default
return self._resolve_single(value) if isinstance(value, str) else value
File "/private/home/kchua13/.conda/envs/hierarchy/lib/python3.6/site-packages/omegaconf/config.py", line 426, in _resolve_single
return Config._resolve_value(root, match.group(1), match.group(2))
File "/private/home/kchua13/.conda/envs/hierarchy/lib/python3.6/site-packages/omegaconf/config.py", line 407, in _resolve_value
raise KeyError("{} interpolation key '{}' not found".format(inter_type, inter_key))
KeyError: "str interpolation key 'a' not found"
when converting to yaml string or container, provide an option to resolve interpolations.
dest[key] = src[key]
is resolving variables which goes against the strategy of resolving lazily.
@staticmethod
def map_merge(dest, src):
"""merge src into dest and return a new copy, does not modified input"""
assert isinstance(dest, DictConfig)
assert isinstance(src, DictConfig)
dest = dest.content
dest = copy.deepcopy(dest)
src = copy.deepcopy(src)
for key, value in src.items():
if key in dest and isinstance(dest[key], Config):
if isinstance(value, Config):
dest[key].merge_with(value)
else:
dest[key] = value
else:
dest[key] = src[key]
return dest
defaults object here is actually of type list instead of Config.
if main_cfg.defaults is None:
main_cfg.defaults = []
the following valid config throws an exception in the debugger when trying to inspect it.
defaults:
- dataset: imagenet
- model: alexnet
- optimizer: nesterov
{'a': None, 'b': '${a}'}
>>> "a" in a
True
>>> "b" in a
False
I was wondering whether we could access the path to the current config. Something likeK ${__file__}
I'm wanted to test hydra, but don't want to hardcode the base directory to my working directory as different collaborators might have a different one. I was hoping that I define the base directory in the main config.yaml
by using ${__file__}
.
Hi, thank you for this great repo!
I want to instantiate omegaconf.Config
with protocol like class to tell IDE what member the config have, that is:
class A:
a: int = 1
cfg: A = omegaconf.OmegaConf.create(A)
print(cfg.a)
1
File "/scratch/pwang01/anaconda3/envs/rl-env/lib/python3.7/site-packages/omegaconf/omegaconf.py", line 48, in create
return DictConfig(obj, parent)
File "/scratch/pwang01/anaconda3/envs/rl-env/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 15, in __init__
self.__setitem__(k, v)
File "/scratch/pwang01/anaconda3/envs/rl-env/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 25, in __setitem__
value = self._prepare_value_to_add(key, value)
File "/scratch/pwang01/anaconda3/envs/rl-env/lib/python3.7/site-packages/omegaconf/config.py", line 446, in _prepare_value_to_add
value = OmegaConf.create(value, parent=self)
File "/scratch/pwang01/anaconda3/envs/rl-env/lib/python3.7/site-packages/omegaconf/omegaconf.py", line 48, in create
return DictConfig(obj, parent)
File "/scratch/pwang01/anaconda3/envs/rl-env/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 15, in __init__
self.__setitem__(k, v)
File "/scratch/pwang01/anaconda3/envs/rl-env/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 25, in __setitem__
value = self._prepare_value_to_add(key, value)
File "/scratch/pwang01/anaconda3/envs/rl-env/lib/python3.7/site-packages/omegaconf/config.py", line 446, in _prepare_value_to_add
value = OmegaConf.create(value, parent=self)
File "/scratch/pwang01/anaconda3/envs/rl-env/lib/python3.7/site-packages/omegaconf/omegaconf.py", line 48, in create
return DictConfig(obj, parent)
File "/scratch/pwang01/anaconda3/envs/rl-env/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 10, in __init__
super(DictConfig, self).__init__()
RecursionError: maximum recursion depth exceeded while calling a Python object
Minimal reproduction config.yaml:
a: &a
b: *a
The following should not throw:
cfg = OmegaConf.create()
cfg.foo = (1,2)
Master task to release 1.4.0.
def test_deepcopy_with_flags():
c1 = OmegaConf.create({'dataset': {'name': 'imagenet', 'path': '/datasets/imagenet'}, 'defaults': []})
OmegaConf.set_struct(c1, True)
c2 = copy.deepcopy(c1)
with pytest.raises(KeyError):
c1.merge_with_dotlist(['dataset.bad_key=yes'])
with pytest.raises(KeyError):
OmegaConf.merge(c2, OmegaConf.from_dotlist(['dataset.bad_key=yes']))
test args:
optimizer.lr_steps=[1,2,3]
from omegaconf.nodes import FloatNode
x = FloatNode()
x == 1
raises
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-163-39f43fc4ed44> in <module>
2
3 x = FloatNode()
----> 4 x == 1
~/work/omegaconf/omegaconf/nodes.py in __eq__(self, other)
115 other_val = other
116
--> 117 return self.val == other_val or (math.isnan(self.val) and math.isnan(other_val))
118
119
TypeError: must be real number, not NoneType
due to the following line:
Line 88 in 7f52127
math.isnan
accepts typing.SupportsFloat
but not NoneType
.
cfg = OmegaConf.create(dict(name='???'))
cfg.get('name', 'default value')
This seems like a useful extension:
flap: 99
baz:
flaf: 100
foo:
bar: 10
a: ${foo.bar} # 10, absolute
b: ${.bar} # 10, relative
c: ${..flap} # 99, relative, level up
d: ${..baz.flaf} # 100, level up and then and then dig into baz
family, name = next(iter(default.items()))
a: 1
b: 2.0
d: ${a} - ${b}
e: ${a} + ${b}
d: ${a} * ${b}
e: ${a} / ${b}
desired behavior:
conf = OmegaConf.load("file.yaml")
assert conf.d == -1.0
assert conf.d == 3.0
assert conf.d == 2.0
assert conf.d == 0.5
This should work only if both operands are numbers (int or float). result type should be int if both are ints or float otherwise.
Also check for other missing list methods.
Similar to standing on a root directory and asking whether any of the sub-directories has files in it
Freeze is nice, but does not work for unintended reads.
It's possible that the real solution here would be to relay on node level validation.
>>> a = OmegaConf.empty()
>>> a.foo
>>> a.freeze(True)
>>> a
{}
>>> a.foo
>>>
>>> a.foo = a
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/private/home/omry/dev/omegaconf.git/omegaconf/dictconfig.py", line 47, in __setattr__
self.__setitem__(key, value)
File "/private/home/omry/dev/omegaconf.git/omegaconf/dictconfig.py", line 26, in __setitem__
raise FrozenConfigError(self.get_full_key(key))
omegaconf.errors.FrozenConfigError: foo
Currently in OmegaConf, for the following config:
server:
host: localhost
port: 80
client:
url: http://${server.host}:${server.port}/
server_port: ${server.port}
We can get conf["server"]
and conf["client"]
, but then we lose the parent component name (e.g., "server" or "client").
This feature is to be able to get complete subtrees, rooted at the specified head.
Example config:
server:
host: localhost
port: 80
client:
url: http://${server.host}:${server.port}/
server_port: ${server.port}
Usage:
print(conf.root_from("server"))
Outputs
server:
host: localhost
port: 80
OmegaConf.set_readonly(cfg.whitelisted_orgs, True)
cfg = OmegaConf.merge(cfg, cli)
This python tool.py whitelisted_orgs=['blah']
changes the readonly variable instead of throwing.
# TODO: improve exception message to contain full key a.b
def test_create_nested_dict_with_illegal_value():
When instantiating OmegaConf from from_dotlist
, yaml loader tries to convert date-formatted strings into date
type, therefore raises error when validating the type of values.
OmegaConf.from_dotlist(["my_date=2019-11-11"])
>>>
.../site-packages/omegaconf/config.py in _prepare_value_to_add(self, key, value)
448 if not Config.is_primitive_type(value):
449 full_key = self.get_full_key(key)
--> 450 raise ValueError("key {}: {} is not a primitive type".format(full_key, type(value).__name__))
451
452 if self._get_flag('readonly'):
ValueError: key my_date: date is not a primitive type
This error also blocks the overriding of configuration values by using cli arguments in hydra.
Could be solved by removing implicit timestamp resolvers in default yaml.SafeLoader
.
(Related stackoverflow: https://stackoverflow.com/questions/34667108/ignore-dates-and-times-while-parsing-yaml)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.