dynaconf / dynaconf Goto Github PK
View Code? Open in Web Editor NEWConfiguration Management for Python ⚙
Home Page: https://dynaconf.com
License: MIT License
Configuration Management for Python ⚙
Home Page: https://dynaconf.com
License: MIT License
Users of Django apps have to specify a 'DYNACONF_SETTINGS' environment variable to specify where their settings file is located. Application writers would benefit from being able to set a SETTINGS config in their settings.py which would provide a default location for a settings file. If the file is not present, dynaconf should log this, and allow django to start up normally.
Hi 👊
This is my first visit to this fine repo, but it seems you have been working hard to keep all dependencies updated so far.
Once you have closed this issue, I'll create seperate pull requests for every update as soon as I find one.
That's it for now!
Happy merging! 🤖
SSIA
In my oppionion, there should not be a constraint to define the default DYNACONF
namespace when the application doesn't need it. There should be a way to avoid loading the default namespace when changing it inside the application.
My suggestion is the to load the desired configuration when dynaconf.namespace(...)
is called or load the default configuration only when there was an attempt to read a variable from dynaconf.settings
with no previous dynaconf.namespace()
call.
One use-case might be choosing the desired environment from command-line or from other sources.
from dynaconf import settings
if sys.argv[1] == "prod":
settings.namespace("production")
elif sys.argv[1] == "devel":
settings.namespace("devel")
else:
raise RuntimeError("Invalid environment specified")
print(settings.MONGODB_HOST)
print(settings.MONGODB_PORT)
print(settings.MONGODB_DATABASE)
a dynaconf cli to be use used on shell to validate and --help as documentation
I have a settings written in linux, with utf-8, however when running on windows it is being read as cp1252 which shows invalid characters. :(
toml
by defaultI am wondering if there might be a method to define default values for configuration parameters for some values because they might not need to be required in some namespaces.
For example , I have a settings.yml file
DYNACONF:
DEVEL:
MONGODB_DATABASE: devel
MONGODB_HOST: localhost
MONGODB_PORT: 27000
PRODUCTION:
MONGODB_DATABASE: production_stuff
MONGODB_HOST: mongodb.example.com
MONGODB_PORT: 10980
MONGODB_USERNAME: efe6398127928f1b2e9ef3207fb82663
MONGODB_PASSWORD: 1f048e5fb80c559a4ab4c6e79d940708
And the username and password fields are not required in the devel namespace (can be None
).
If I attempt to get the MONGODB_USERNAME
parameter from the production
namespace, it works as expected but if I attempt to get it from the DEVEL
namespace, it raises an AttributeError
and I need to hack around it with getattr
as shown.
print(settings.MONGODB_HOST)
print(settings.MONGODB_PORT)
print(settings.MONGODB_DATABASE)
print(getattr(settings, "MONGODB_USE_SSL", False))
print(getattr(settings, "MONGODB_USERNAME", None))
print(getattr(settings, "MONGODB_PASSWORD", None))
I understand if this is not a desired feature but it might come in really handy in some situations when we need some parameters in only one environment.
Currently, if I have FOO = False
in settings.pyand I use
settings.get('FOO', cast='@Bool')`, I get:
AttributeError: 'bool' object has no attribute 'lower'
I would suggest changing https://github.com/rochacbruno/dynaconf/blob/master/dynaconf/utils/parse_conf.py#L9 to:
'@bool': lambda value: True if str(value).lower() in true_values else False,
Write a lot of tests
Dynaconf requires Flask by default, is that by mistake or is it intentionally?
File "/app/.heroku/python/lib/python3.6/site-packages/dynaconf/__init__.py", line 5, in <module>
from dynaconf.contrib import FlaskDynaconf
File "/app/.heroku/python/lib/python3.6/site-packages/dynaconf/contrib/__init__.py", line 1, in <module>
from dynaconf.contrib.flask_dynaconf import FlaskDynaconf, DynaconfConfig # noqa
File "/app/.heroku/python/lib/python3.6/site-packages/dynaconf/contrib/flask_dynaconf.py", line 2, in <module>
from flask.config import Config
ModuleNotFoundError: No module named 'flask'
Implement loaders for:
Would be nice to support a more complex settings structure like:
PRODUCTION:
DATABASE:
HOST: ....
PASSWORD: ...
And after use it like: settings.DATABASE.HOST
Currently you get a dict at DATABASE
I haven't had time to find all the Python 3 incompatibilities but there's one I just ran into here. basestring
is not a thing in Python 3 anymore.
As of today, importing dynaconf configures the handlers in the logging
root logger, which renders the basicConfig
useless:
This function does nothing if the root logger already has handlers configured for it.
Example:
>>> import logging
>>> logging.basicConfig(format="[%(asctime)s|%(levelname)s] %(message)s",
... datefmt="%Y-%m-%dT%H:%M:%S")
>>> logging.error("Something")
[2018-06-23T18:21:03|ERROR] Something
The collateral effect of importing dynaconf
:
>>> import logging
>>> import dynaconf
>>> logging.basicConfig(format="[%(asctime)s|%(levelname)s] %(message)s",
... datefmt="%Y-%m-%dT%H:%M:%S")
>>> logging.error("Something")
2018-06-23 18:21:59,723 - root - ERROR - Something
This library should support Python 3 only
If settings.py is not present, try settings.yml instead of throwing an error. This will allow projects using settings.yml on root without having to specify a env var nor calling configure function.
setup.py
specifies packages=find_packages()
, which will include tests
and example
.
Perhaps packages=find_packages(include=['dynaconf'])
would be better?
Hi Bruno
I am reading about dynaconf and it looks quite good, however, i dont know if it will fit in our use case, so i am looking for your confirmation on that
I would like to use a config server for python to retrieve external configurations (eg - a yaml file in a git repository) that i will use that later for my application. That configuration can change at some point, so it should be dinamically reloaded
I have seen it works well when pulling a configuration from a redis database or similar, but i dont know if it would work to pull a file from a git repository?
Thanks
Based on https://www.reddit.com/r/Python/comments/3h1tau/dynamic_config_load_for_python_load_settings_from/
If we have two projects using Dynaconf in the same environment we need a way to set different variables, so we need namespaces.
A namespace should be defined by __
double underline (also known as dunder)
so
$DYNACONF_DATABASE
will parse to
settings.DATABASE
and
$DYNACONF__PROJECTA_DATABASE
or
$DYNACONF__PROJECTB_DATABASE
will be accessed by settings the namespace
from dynaconf import settings
settings.DATABASE
# accessing namespace vars
settings.namespace('PROJECTA')
settings.DATABASE
settings.namespace('PROJECTB')
settings.DATABASE
I couldn't find any documentation for this project.
I suggest to generate (with sphinx, using autodoc plugin?) some documentation for the end user and for future people that are interested in contributing to this project (for example I couldn't find some hints or guides on how to create a loader from scratch).
Hi!
First of all, you rock. Great project.
Bug description:
Environment: Flask Application
When you use yaml
files, the settings.yml(yaml)
and .secret.yml(yaml)
are no loaded in the settings
When you use toml
both are loaded, e.g.:
|- other/stuff
|- .env
|- manage.py
|- settings.yml(yaml)(toml) # --> Both settings are loaded from toml
|- .secrets.yml(yaml)(toml) # No settings are loaded from yaml/yml
Raises:
(...) dynaconf/base.py", line 100, in __getattr__
return getattr(self._wrapped, name)
AttributeError: 'Settings' object has no attribute 'ATTRIBUTE'
Question: is there any way to use settings.toml and .secrets.yml?
from flask import Flask
from dynaconf.utils.flask import DynaFlask
app = Flask(__name__)
DynaFlask(app)
app.run()
Then, app.config will be replaced by DynaFlask which is a config class able to read from dynaconf
Currently, all loaders are executed again every time we
settings.using_namespace
or settings.namespace
always_fresh
For the fresh=True
and always_fresh
it makes sense to execute loaders again as it is demanded to get the fresh variable (lets say from redis or another database).
The problem is that it may cause overhead (I guess not too much) but in any case we can improve by.
fresh
examples)execute_loaders
so files and databases are not accessed twice when we are only switching namespaces and not wanting to load all again.Part of this is already implemented in loaded_by_loaders
dictionary, we just need to use this as a cache point.
Targeting plugin based applications take as example:
Configuration for general program is in export SETTINGS_FOR_DYNACONF=/etc/program/settings.ext
(ext
being any of the supported formats yaml, json, ini, toml etc)
That same program has plugins and plugin stores its settings under different path like /etc/plugin_name/settings.yaml
include
of settings files in different formats.Then /etc/program/settings.yaml
DEFAULT:
DYNACONF_INCLUDE:
- /etc/plugin_name/settings.yaml
- /etc/other_plugin_name/settings.toml
- /etc/settings_folder/*
or toml
[default]
dynaconf_include = ["path1", "path2"]
or json
{
"default": {
"dynaconf_include": ["path1", "path2"]
}
}
then All the loaders will take the special DYNACONF_INCLUDE
key then load values from each of the files listed using the normal pipeline.
I get this error when I run an exe generated from pyinstaller :
>dummy.exe
Hello World !!!
Traceback (most recent call last):
File "dummy.py", line 4, in <module>
File "site-packages\dynaconf\base.py", line 87, in __getattr__
File "site-packages\dynaconf\base.py", line 113, in _setup
File "site-packages\dynaconf\base.py", line 166, in __init__
File "site-packages\dynaconf\base.py", line 575, in execute_loaders
File "site-packages\dynaconf\base.py", line 550, in loaders
File "importlib\__init__.py", line 126, in import_module
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'dynaconf.loaders.env_loader'
[3256] Failed to execute script dummy
Here is my source script dummy.py :
import dynaconf
print('Hello World !!!')
print(dynaconf.settings.var1)
I use the latest 64 bits anaconda on a windows 7 64 bits and :
pip.exe install pyinstaller==3.3.*
pip.exe install pyyaml==3.12.* dynaconf[yaml]==1.0.*
I try to build a folder or a single exe with pyinstaller (both give the same error) :
pyinstaller.exe dummy.py
or
pyinstaller.exe --onefile dummy.py
I tried to convert README.md to README.rst using m2r
the code is on setup.py
does not works, pypi page is bas formatted: https://pypi.python.org/pypi/dynaconf
I need help to convert README to proper RsT or even rewrite README in valid .rst and drop .md.
look: https://github.com/disqus/gutter
Dynaconf should have a simplified version
from dynaconf import settings
if settings.is_active('some feature'):
# do something
and then in dynaconf.conf features will be defined with conditions
[some feature]
object = user
attribute = age
condition = >=45
Features should also be read from REDIS
Using pytest-xdist
or even threading
and multiprocessing
directly
we need to write a test which will run on travis to ensure.
Depends on #84
Traversal of nested dictionaries should also be possible in validators.
having settings.yaml
DEFAULT:
REDIS:
HOST: localhost
PORT: 1234
It is currently possible to write validator to ensure:
REDIS
key is set and it is a dict
and it has the keys HOST
and PORT
being HOST
a str
and PORT
as int
from dynaconf import settings, Validator
settings.validators.register(
Validator(
'REDIS',
must_exist=True,
is_type_of=dict.
condition=lambda v: (
'HOST' in v and 'PORT' in v
) and (
isinstance(v.HOST, int) and isinstance(v.PORT, int)
)
),
)
settings.validators.register(
Validator('REDIS', must_exist=True, is_type_of=dict),
Validator('REDIS.HOST', must_exist=True, is_type_of=str),
Validator('REDIS.PORT', must_exist=True, is_type_of=int),
)
Also the above validation can be translated to toml
validator file as in https://dynaconf.readthedocs.io/en/latest/guides/validation.html#cli-and-dynaconf-validators-toml
For flask and django the dynaconf
cli should discover the global_env FLASK_ or DJANGO_ to be able to list, write or init correctly
Run from project folder, admin to inspect config data
A Django extension when enabled should replace django.conf.settings
with Dynaconf instance
Hi Bruno,
I am trying to load configuration from multiple files in different folders
This is the folder structure that i use:
my_project
├── config1
│ ├── init.py
│ └── settings.toml
├── config2
│ ├── init.py
│ └── settings.toml
└── main.py
in each settings.toml i have 2 environment development and production ad VAR1 for config1 and VAR2 for config2
[development]
VAR1 = 'var1_dev'
[production]
VAR1 = 'var1_prod'
In the main
import os
from dynaconf import settings
from my_project.config1 import __file__ as config1_file
from my_project.config2 import __file__ as config2_file
f1 = os.path.join(os.path.dirname(config1_file), 'settings.toml')
f2 = os.path.join(os.path.dirname(config2_file), 'settings.toml')
os.environ['SETTINGS_MODULE_FOR_DYNACONF'] = "{},{}".format(f1, f2)
print('development')
print(settings.VAR1)
print(settings.VAR2)
print('production')
settings.setenv('production')
print(settings.VAR1)
print(settings.VAR2)
For default and development it works but when i try to setenv to production i have this error :
development
var1_dev
var2_dev
production
Traceback (most recent call last):
File "/home/hch/PycharmProjects/dynaconf/iia/my_project/main.py", line 13, in <module>
settings.setenv('production')
File "/home/hch/.virtualenvs/dynaconf/lib/python3.6/site-packages/dynaconf/base.py", line 459, in setenv
self.execute_loaders(env=env, silent=silent, filename=filename)
File "/home/hch/.virtualenvs/dynaconf/lib/python3.6/site-packages/dynaconf/base.py", line 572, in execute_loaders
filename=filename)
File "/home/hch/.virtualenvs/dynaconf/lib/python3.6/site-packages/dynaconf/loaders/__init__.py", line 53, in settings_loader
settings_module = settings_module or obj.settings_module
File "/home/hch/.virtualenvs/dynaconf/lib/python3.6/site-packages/dynaconf/base.py", line 413, in settings_module
if settings_module != self.SETTINGS_MODULE:
AttributeError: 'Settings' object has no attribute 'SETTINGS_MODULE'
It's very common in production to store settings in a NoSQL database like DynamoDB.
Would be nice to support these kind of sources.
Hey!
I tracked a bug in the namespace with YAML.
The problem is basically that the environment variable DYNACONF_NAMESPACE is loaded in the execute_loaders(self, namespace=None, silent=None, key=None)
AFTER the YAML as been already parsed in module_loader(obj, settings_module=None, namespace=None, silent=False)
.
And for some reason the YAML loader isn't rexecuted in the execute_loaders
, not sure why.
Thanks ;)
SSIA
Usually I create a virtualenv for my python-applications,
and I collect all config-files in an etc-directory to avoid polluting the project-directory, like this:
/opt/application-name/bin
/opt/application-name/etc/
/opt/application-name/lib
/opt/application-name/src/git-checkout
So I propose to look for config-files not only in the parent-directories, walking all the way up to root,
but also check the existence for an etc-subdirectory.
In bigger projects with lots of components, we split up the etc-dir into
.../etc/uwsgi
.../etc/supervisord
.../etc/dynaconf
As bonus we also support the global etc-directory:
/etc/settings.yml
/etc/dynaconf/settings.yml
Yes it is possible and https://coveralls.io/github/rochacbruno/dynaconf lists all files missing coverage.
Lets write more tests!!!
Lets take a settings.yaml
DEFAULT:
REDIS:
HOST: localhost
PORT: 1234
To access the port
you need to use one of:
settings.REDIS['PORT'] # may raise KeyError
settings['REDIS']['PORT'] # may raise KeyError
settings.REDIS.PORT # possible because it is a Box(), but may raise AtributeError
settings.get('REDIS').PORT # dict like get method
settings('REDIS')['PORT'] # the same as calling .get
settings.get('REDIS').get('PORT') # may raise AttributeError
But what if you are not sure that the REDIS
value is defined in settings.yaml
and you don't wanna fail?
one way is relying on the .get
default argument (dict like)
settings.get('REDIS', {}).get('PORT', 1234)
settings('REDIS.PORT') # would access the settings['REDIS']['PORT'] or raise KeyError.
settings('REDIS.PORT', 1234) # if value cannot be accessed returns 1234 by default
The .
should be parsed on LazySettings.get
method and then perform the traversing.
opt-out
Just like vault and redis
for
having a dynaconf.toml
file
dynaconf validate
must run validators
Hi, I have this folder structure:
test_dynaconf
└── subfolder
├── main.py
└── settings.toml
with main.py:
#main.py
from dynaconf import settings
print(settings["KEY1"])
and settings.toml:
#settings.toml
[default]
KEY1="hi"
If I run from subfolder, everything works:
➜ subfolder python3 main.py
hi
But if I run from test_dynaconf or another path, it breaks:
test_dynaconf python3 subfolder/main.py
Traceback (most recent call last):
File "subfolder/main.py", line 3, in <module>
print(settings["KEY1"])
File "/usr/local/lib/python3.5/dist-packages/dynaconf/utils/functional.py", line 208, in inner
return func(self._wrapped, *args)
File "/usr/local/lib/python3.5/dist-packages/dynaconf/base.py", line 193, in __getitem__
raise KeyError('{0} does not exists'.format(item))
KeyError: 'KEY1 does not exists'
dynaconf is properly showing the settings I put into /etc/pulp/settings.py
(pulp) [vagrant@pulp3 pulp]$ dynaconf list -k DATABASES
Django app detected
Working in development environment
DATABASES: {'default': {'CONN_MAX_AGE': 0,
'ENGINE': 'django.db.backends.dummy',
'HOST': 'postgres',
'NAME': 'pulp',
'PORT': '5432',
'USER': 'pulp'}}
However, django is not using the setting. It seems like Django is loading the settings for the database before dynaconf is loaded.
Change the use of term namespace
to use env
so variable names like NAMESPACE_FOR_DYNACONF
will be changed to ENV_FOR_DYNACONF
, also functions like using_namespace
will be called using_env
and a compat layer must be added to not break. The reason is that it meets the use of ENV
in other frameworks.
Rename the base DYNACONF
env to DEFAULT
, currently, the base env
is called DYNACONF
it is the default namespace for variables and must be renamed to DEFAULT
(this will be a breaking change)
Change the order of loading setting .toml
as the preferred config format.
Add to CLI
#2 a command called init
which will add a new settings.* file (defautls to toml) containing the scaffolding.
So with this change, an application config file will look like this.
settings.toml
[default]
server = 'defaulserver.com'
# value in [default] env will serve as default when it is not defined in specific env
# default is optional and can also be empty or ommited
# values from other [envs] will overrride existing default values
[development]
server = 'devserver.com'
[staging]
server = 'stagingserver.com'
user = 'staging user'
[testing]
server = 'ciserver.com'
[production]
server = 'prodserver.com'
[custom]
server = 'customserver.com'
# If it is needed to override all values in every namespace declare a [global] env
[global]
server = 'itoverrideseverything.com'
IN a .secrets
file there is no reason to have different envs
as easch environment must have its own secrets file.
.secrets.toml
[default]
password = 'SuperSecret'
The variable ENV_FOR_DYNACONF
must default to DEVELOPMENT
and when it is time to change a variable must be defined.
export ENV_FOR_DYNACONF=PRODUCTION
That is all
Programatically switching namespaces may be useful for testing and also to use Dynaconf as a Feature flag system so to switch:
with settings.using_env('staging'):
assert settings.USERNAME == 'staging user'
assert settings.SERVER == itoverrideseverything.com' # override by [global]
DYNACONF
env it works as a global
and is named DYNACONF
instead of GLOBAl to avoid conflicts with existing envvars.To overwrite values in any of the namespaces do:
export DYNACONF_SERVER='ThisOverwritesEverything.com'
Load settings from YAML file
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.