Check out the test error on https://ci.appveyor.com/project/jezdez/envdir/build/1.0.6/job/kuxa9gmcwd7l261x#L63

Would be nice to have an up-to-date release in pypi, so people don't waste time with bugs that have already been fixed since 2 years.

If a lookup happens inside a loop, this can severely affect the performance of the loop, compared to if the lookup was happening on environment variable.

Hi,

what do you think about adding support to create envdir, not only to read it?
Like this:

import envdir
envdir.write('/path/to/envs/prod', {
    'DJANGO_SETTINGS_MODULE': 'myproject.settings',
})

This would made the package useful also for administrative tasks like creating env for new application.

Here is the directory structure I have set up for this scenario:

.
├── env
│   └── PYTHONPATH
├── lib
│   └── mypackage
│       ├── __init__.py
│       └── settings.py
└── myscript.py

3 directories, 4 files

And this is the command I am using to instantiate myscript.py: envdir env python myscript.py, where myscript.py tries to import mypackage. Because mypackage is nested inside the lib folder, myscript.py cannot see the package unless the PYTHONPATH is changed.

This is the traceback I get after running the above command:

Traceback (most recent call last):
  File "myscript.py", line 1, in <module>
    from mypackage.settings import *
ImportError: No module named mypackage.settings

If I set the PYTHONPATH manually in my current shell, it works though:

export PYTHONPATH="lib:$PYTHONPATH"
python myscript.py

I thought it might have something to do with being a relative PATH so I put the absolute path to the lib folder in my PYTHONPATH environment variable file and ran envdir again but it still didn't work.

I can see that the variable is definitely getting set by running the env command.

➜   testenvdir  envdir env env | grep "PYTHONPATH"
PYTHONPATH=PYTHONPATH="/Users/johria/Development/testenvdir/lib:$PYTHONPATH"

Do you have any ideas why this might not be working?

I have this simple code:

import time

try:
    while True:
        time.sleep(0.1)
except KeyboardInterrupt:
    print 'hey'

which prints 'hey' by keyboard interruption when I run it like python test.py. But when I do the same thing with envdir like envdir envdir python test.py, it doesn't print 'hey'. I think envdir catches the interrupt and closes the application itself.

I am considering using envdir for a Django project, but get the impression that it may become more difficult to grep / search for settings and their values, given that they are split into filename and file content.

For example, you cannot just grep your settings / project folder anymore for DEBUG, EMAIL_ etc, without having to also grep (and then look into) the output of find.

Given that I am also looking into django-configurations, because it allows to have local, production and staging settings in a single file, this feels uncomfortable.

OTOH, envdir makes it easier already to have environment settings in a central location (envs/{local,prod,...}), which are scattered around otherwise (virtualenv's postactivate, wsgi server setting files, ...).

Just wanted to leave this as feedback - maybe there's an approach to make it easier to work with it?

Maybe a script / option could be provided, which only displays settings from a specific env dir in the form of NAME=VALUE.
Or when invoked in a directory with subdirs, would display it like grep as in:

$ envprint ./envs
prod: DEBUG=False
prod: FOO=1
local: DEBUG=True

$ envprint ./envs/local
DEBUG=True

This would be useful to easily override some var.

Like with DJANGO_DEBUG=False manage.py runserver, where the (custom) hook in manage.py could then look like this (overwrite would be a new kwarg to open):

thisdir = os.path.dirname(os.path.realpath(__file__))
envdir.open(thisdir + '/../envs/' + DJANGO_ENVDIR, overwrite=False)

I've just wondered if it might be a good idea to have envdir setup an environment variable itself, which might contain the dir being used.

I just fell for it, because I was using envdir envs/tests manage.py test, while having the following in manage.py, which loaded another envdir on top:

import envdir
try:
    DJANGO_ENVDIR = os.environ['DJANGO_ENVDIR']
except KeyError:
    DJANGO_ENVDIR = 'local'
logging.info('Using DJANGO_ENVDIR=%s', DJANGO_ENVDIR)
thisdir = os.path.dirname(os.path.realpath(__file__))
envdir.open(thisdir + '/../envs/' + DJANGO_ENVDIR)

Having envdir setup ENVDIR automatically would allow for easier integration/usage.

Of course, I could just define an own variable, as in ENVDIR=local envdir envs/$ENVDIR ... and use that in manage.py etc.
Or, add DJANGO_ENVDIR to the envs/.. dirs itself.

Hello,

I have a tmuxp profile as follows:

session_name: TEST
shell_command_before: envshell ./envars
windows:
- window_name: TEST
  layout: tiled
  panes:
    - shell_command:
      - docker exec -it $ODOOCONT /bin/bash
    - shell_command:
      - docker exec -it $ODOOCONT /bin/bash

As you can see for each pane I am loading with envshell the envars folder, which contains one variable as follows:

cat envars/ODOOCONT
test_container_19

I would like to do something like:
ODOOCONT=$ODOOCONTAINER

Where $ODOOCONTAINER is defined on my local shell session before executing tmuxp. This doesn't work but is there any other way to achieve this?

Thank you!

If you run "python -m envdir" it will complain about the order in which the modules are loaded. I opened a PR to fix this #70. Is it possible to get it merged?

Thanks,
Vlad

I am using the Python API in manage.py as in

envdir.open(DJANGO_ENVDIR)

The problem appears to be that Runner sets up the signal listener in init:

signal.signal(signal.SIGTERM, self.terminate)

The simple fix would be to have a setsignals method, which is then called by run() and shell() only.

More details:
The command:

sudo env DJANGO_ENVDIR="$(DJANGO_ENVDIR)" $(PYTHON) apps/manage.py supervisor $(SUPERVISOR_ARGS)

supervisor starts Django's runserver this way:

command={{ PYTHON }} {{ PROJECT_DIR }}/manage.py runserver --insecure

Traceback/Log:

^C2014-01-21 17:09:25,711 WARN received SIGINT indicating exit request
2014-01-21 17:09:25,712 DEBG killing webserver (pid 20210) with signal SIGTERM
2014-01-21 17:09:25,712 INFO waiting for webserver to die
2014-01-21 17:09:25,713 DEBG 'webserver' stdout output:
envdir: terminating child with SIGTERM

2014-01-21 17:09:25,713 DEBG 'webserver' stdout output:
Traceback (most recent call last):
  File "/apps/manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "virtualenv/src/django-stable/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "virtualenv/src/django-stable/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "virtualenv/src/django-stable/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "virtualenv/src/django-stable/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "virtualenv/src/django-stable/django/core/management/commands/runserver.py", line 79, in handle
    self.run(*args, **options)
  File "virtualenv/src/django-stable/django/core/management/commands/runserver.py", line 88, in run
    autoreload.main(self.inner_run, args, options)
  File "virtualenv/src/django-stable/django/utils/autoreload.py", line 181, in main
    reloader(wrapped_main_func, args, kwargs)
  File "virtualenv/src/django-stable/django/utils/autoreload.py", line 153, in python_reloader
    exit_code = restart_with_reloader()
  File "virtualenv/src/django-stable/django/utils/autoreload.py", line 140, in restart_with_reloader
    exit_code = os.spawnve(os.P_WAIT, sys.executable, args, new_environ)
  File "virtualenv/lib/python2.7/os.py", line 573, in spawnve
    return _spawnvef(mode, file, args, env, execve)
  File "virtualenv/lib/python2.7/os.py", line 546, in _spawnvef
    wpid, sts = waitpid(pid, 0)
  File "virtualenv/src/envdir/envdir/runner.py", line 135, in terminate
    self.quit(signum)
  File "virtualenv/src/envdir/envdir/runner.py", line 151, in quit
    if self.process.poll() is None:
AttributeError: 'Runner' object has no attribute 'process'

2014-01-21 17:09:25,746 DEBG fd 8 closed, stopped monitoring <POutputDispatcher at 63828336 for <Subprocess at 63097096 with name webserver in state STOPPING> (stdout)>
2014-01-21 17:09:25,746 INFO stopped: webserver (exit status 1)
2014-01-21 17:09:25,747 DEBG received SIGCLD indicating a child quit
make: *** [supervisor] Interrupt

Command terminated

I have experimented with circus and chaussette, using the following command for the django workers:

cmd = envdir $(circus.env.envdir) chaussette --fd $(circus.sockets.django) project.wsgi.application

The close_fds=not is_windows setting that envdir uses with Popen apparently breaks this:

2014-01-11 17:03:53 [9328] [INFO] Serving on fd://8
2014-01-11 17:03:53 [9328] [INFO] Using <class chaussette.backend._wsgiref.ChaussetteServer at 0xf6c6d0> as a backend
Traceback (most recent call last):
  File "/venv/bin/chaussette", line 9, in <module>
    load_entry_point('chaussette==1.0', 'console_scripts', 'chaussette')()
  File "/venv/lib/python2.7/site-packages/chaussette/server.py", line 201, in main
    inner()
  File "/venv/lib/python2.7/site-packages/chaussette/server.py", line 182, in inner
    socket_type=_SOCKET_TYPE[args.socket_type])
  File "/venv/lib/python2.7/site-packages/chaussette/server.py", line 35, in make_server
    server = server_class((host, port), app, **server_class_kwargs)
  File "/venv/lib/python2.7/site-packages/chaussette/backend/_wsgiref.py", line 38, in __init__
    backlog=backlog)
  File "/venv/lib/python2.7/site-packages/chaussette/util.py", line 261, in create_socket
    sock = socket.fromfd(fd, family, type)
socket.error: [Errno 9] Bad file descriptor

Hi Jannis

I tried to run the tests but they keep failing and I can't figure out what I'm doing wrong. Maybe you could help me on this one. This is what I have done so far:

• New virtualenv envdir created with virtualenvwrapper
• Installed cram into the virtualenv (pip install cram)
• envdir repo cloned to /path/to/the/cloned/envdir/
• Added the path containing the package to the Python path (/path/to/the/cloned/envdir/)
• Run the tests with python setup.py test

Now each test throws and envdir: command not found or envshell: command not found error.

Do you have an idea what I'm doing wrong?

Each envdir can optionally contain a file named inherits, which contains names of other env dirs from which this envdir is inherited from.

So my envs folder can contain base, prod, dev, and test, and each of prod, dev and test contains inherits file containing "base", and most of my environment variables stored in base.

The from .__main__ import runner statement in __init__.py masks the name of the envdir.runner module. This is especially problematic as envdir.runner module contains the exception class Response.

This seems to mean that to use the API one must not only have two import statements, but you must also pollute the local namespace.

from envdir.runner import Response
import envdir

I would like to share some environment variables between stages and the easiest way to do that seems to be to symlink from my local (i.e. default) environment into my production and staging environments. In the example below, I would like to share the DJANGO_SETTINGS_MODULE and PYTHONPATH variables between all stages.

Currently envdir doesn't follow symlinks so the variables will be missing if symlinked.

This seems like a feature that's just enough in scope @jezdez :) Let me know if you want me to give a shot at a PR!

➜   envs [master] ⚡ tree
.
├── local
│   ├── DATABASE_URL
│   ├── DEBUG
│   ├── DJANGO_SETTINGS_MODULE
│   ├── PYTHONPATH
│   └── STAGE
├── production
│   ├── DATABASE_URL
│   ├── DEBUG
│   ├── DJANGO_SETTINGS_MODULE
│   ├── PYTHONPATH
│   └── STAGE
└── staging
    ├── DATABASE_URL
    ├── DEBUG
    ├── DJANGO_SETTINGS_MODULE
    ├── PYTHONPATH
    └── STAGE

Thanks!

cc @blueyed

And only if its not there should it read from file. That way I can do

DB=somedb python app.py

and my DB setting will supersede whats stored in envs/prod/DB, which I think is what the end user is trying to do.

First of all, thank you for this neat little tool.

I think I found a little bug in the read() method at https://github.com/jezdez/envdir/blob/master/envdir/__main__.py#L55 :

If an environment variables is already set, envdir will not overwrite it.

That's because of os.environ.setdefault(name, value) in
https://github.com/jezdez/envdir/blob/master/envdir/__main__.py#L63 not setting the value, if the key namealready exists in the os.environ dict:

def setdefault(self, key, failobj=None):
    if key not in self:
       self[key] = failobj
    return self[key]

hi @jezdez, it seems envdir is modifying the PATH in some unexpected way when using virtualenv. daemontools seems to be working as expected.

I have a pretty vanilla set up using pyenv and virtualenvwrapper. Nothing fancy.

setup:

$ deactivate
$ mkvirtualenv test
$ deactivate

with jezdez/envdir:

$ workon test
$ which python
/Users/johria/.virtualenvs/test/bin/python
$ envdir envs/prod which python
/Users/johria/.pyenv/versions/3.6.1/bin/python

^^^^ error is here ^^^^

uninstall jezdez/envdir, install daemontools:

$ deactivate
$ pip uninstall envdir
$ brew install daemontools

with daemontools:

$ workon test
$ which python
/Users/johria/.virtualenvs/test/bin/python
$ /usr/local/bin/envdir envs/prod which python
/Users/johria/.virtualenvs/test/bin/python

To debug errors, I'm often running code locally using envs/prod to test something and have to override DEBUG to true.

It would be helpful to be able to do something like this:

DEBUG=true envdir envs/prod make runserver

to easily override a single variable. Not quite sure on the syntax or implementation details though.

Is there any interest in something like this? Do most people just create a separate environment like prod-debug or prod-local?

I was standing in my envdir when my psql session decided to do a coredump, creating a .core file in the envdir. This later resulted in the rather nondescript error An error occurred: [Errno 7] Argument list too long, as I had failed to remove the .core-file.

I could look into this and submit a patch if you think it makes sense / is possible to handle this type of error in a sensible way?

The 0.7.0 release is missing a .pyz download

• existing env var STAGE=local
• create file config/dev/STAGE with value 'dev'
• run "envshell config/dev"
• run "echo $STAGE"
• result is 'local' when i was expecting 'dev'

hi @jezdez, this may be completely out of scope as the project is literally named envdir, but I was wondering if its possible to provide a single file with all of the variables for each environment. so like:

prod
  - .env
dev
  - .env

this is mainly because I run programs both locally and with docker and docker allows me to specify a --env argument with a single file of environment variables. this way I can use the same file of environment variables for locally running a program and docker

thanks!

Simply install the launcher from its site (downloads) and you're ready to follow the rest of the instructions below.

is there anyway to use envdir with shell aliases? for example I have a couple aliases for django-admin as da or docker-compose as dc and I would love to use them as envdir envs/prod da migrate

In bash, you can do things like: export PATH=$PATH:/some/new/path

It would be nice if envdir supported similar expansion.

The original envdir.c doesn't support this, so you may be reluctant. However, in trying to do 12-factor apps, I've found it useful to have, for example, resource prefixes so that you can have jenkins builds like so:

DATABASE_URL=postgresql://localhost/db_$JOB_NAME

Thoughts?

As reminded by #57 we should automate the creation of pyz files on tagging via Travis CI

Trying to run envdir:

'/root/envs' does not exist

But the directory exists:

root@heroku-i-0b5b786a2e16326d2 [prod-usw2-prod1] ~/envs # pwd
/root/envs

Issue appears to be here: https://github.com/jezdez/envdir/blob/master/envdir/runner.py#L29

I tried to output the value of real_path but it looks good:

envdir '/root/envs' does not exist real path = '/root/envs'

In the sentence "to simplify maintaing complicated environments"

s/maintaing/maintaining

We yesterday discovered an issue where for some reason SIGTERM is not cleanly propagated to the child process.

For instance, if you launch a gunicorn instance with two workers

$ envdir env python manage.py run_gunicorn -w 2
2013-12-20 08:51:00 [9004] [INFO] Starting gunicorn 18.0
2013-12-20 08:51:00 [9004] [INFO] Listening at: http://127.0.0.1:8000 (9004)
2013-12-20 08:51:00 [9004] [INFO] Using worker: sync
2013-12-20 08:51:00 [9007] [INFO] Booting worker with pid: 9007
2013-12-20 08:51:00 [9008] [INFO] Booting worker with pid: 9008

$ % ps aux | grep envdir
zerok            9104   0,0  0,1  2449096   8436 s000  S+    8:52am   0:00.08 /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python /usr/local/bin/envdir env python manage.py run_gunicorn -w 2

And then send SIGTERM to the envdir process, you end up with only this process terminating:

$ kill -TERM 9104
$  ps aux | grep gunicorn
zerok            9165   0,0  0,0  2442000    576 s002  R+    8:54am   0:00.00 grep gunicorn
zerok            9109   0,0  0,5  2508164  44144 s000  S     8:52am   0:00.49 python manage.py run_gunicorn -w 2
zerok            9108   0,0  0,5  2498948  43704 s000  S     8:52am   0:00.51 python manage.py run_gunicorn -w 2
zerok            9105   0,0  0,3  2483984  27540 s000  S     8:52am   0:00.34 python manage.py run_gunicorn -w 2

I couldn't reproduce this directly with deamontools' envdir because they seem to completely replace the envdir process once the environment is set up.

I'm using envdir to populate the environment of a Django app (Django 1.6.1, Python 2.7.5, mod_wsgi 3.4, Apache 2.2.25) and I'm seeing the following (very lightly anonymized) warning in my logs. A little digging led me to a section on mod_wsgi's wiki about registered signal handlers conflicting with Apache's signal handling.

[Wed Jan 22 21:30:55 2014] [notice] Apache/2.2.25 (Unix) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Wed Jan 22 21:31:00 2014] [warn] mod_wsgi (pid=1744): Callback registration for signal 15 ignored.
[Wed Jan 22 21:31:00 2014] [warn]   File "/home/user/webapps/production/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 187, in __call__
[Wed Jan 22 21:31:00 2014] [warn]     self.load_middleware()
[Wed Jan 22 21:31:00 2014] [warn]   File "/home/user/webapps/production/lib/python2.7/site-packages/django/core/handlers/base.py", line 46, in load_middleware
[Wed Jan 22 21:31:00 2014] [warn]     for middleware_path in settings.MIDDLEWARE_CLASSES:
[Wed Jan 22 21:31:00 2014] [warn]   File "/home/user/webapps/production/lib/python2.7/site-packages/django/conf/__init__.py", line 54, in __getattr__
[Wed Jan 22 21:31:00 2014] [warn]     self._setup(name)
[Wed Jan 22 21:31:00 2014] [warn]   File "/home/user/webapps/production/lib/python2.7/site-packages/django/conf/__init__.py", line 49, in _setup
[Wed Jan 22 21:31:00 2014] [warn]     self._wrapped = Settings(settings_module)
[Wed Jan 22 21:31:00 2014] [warn]   File "/home/user/webapps/production/lib/python2.7/site-packages/django/conf/__init__.py", line 128, in __init__
[Wed Jan 22 21:31:00 2014] [warn]     mod = importlib.import_module(self.SETTINGS_MODULE)
[Wed Jan 22 21:31:00 2014] [warn]   File "/home/user/webapps/production/lib/python2.7/site-packages/django/utils/importlib.py", line 40, in import_module
[Wed Jan 22 21:31:00 2014] [warn]     __import__(name)
[Wed Jan 22 21:31:00 2014] [warn]   File "/home/user/webapps/production/project/proj/settings/__init__.py", line 2, in <module>
[Wed Jan 22 21:31:00 2014] [warn]     from .base import *
[Wed Jan 22 21:31:00 2014] [warn]   File "/home/user/webapps/production/project/proj/settings/base.py", line 7, in <module>
[Wed Jan 22 21:31:00 2014] [warn]     import envdir
[Wed Jan 22 21:31:00 2014] [warn]   File "/home/user/webapps/production/lib/python2.7/site-packages/envdir/__init__.py", line 1, in <module>
[Wed Jan 22 21:31:00 2014] [warn]     from .__main__ import runner, go
[Wed Jan 22 21:31:00 2014] [warn]   File "/home/user/webapps/production/lib/python2.7/site-packages/envdir/__main__.py", line 5, in <module>
[Wed Jan 22 21:31:00 2014] [warn]     runner = Runner()
[Wed Jan 22 21:31:00 2014] [warn]   File "/home/user/webapps/production/lib/python2.7/site-packages/envdir/runner.py", line 33, in __init__
[Wed Jan 22 21:31:00 2014] [warn]     signal.signal(signal.SIGTERM, self.terminate)

Some digging into envdir's source revealed that envdir.open == Runner().open but using Env directly would circumvent the issue if not for from __main__ import runner in __init__.py.

Is there a known workaround for the situation I've described above? I've been looking for a way to refactor the modules in order to access Env without registering the signal handler in Runner.__init__ while also maintaining backwards compatibility.

Thanks for the help!

Using envdir with ipython showed that SIGINT / KeyboardInterrupt should sometimes only forwarded to the wrapped program, but not terminate it: ipython is not meant to be exited / terminated via SIGINT.

Currently envdir will send SIGTERM instantly.
My older pull request (#18) changes it to forward SIGINT, but when the program keeps running will terminate it also.

Is there a way to detect if the program handled SIGINT itself?

Maybe all signals should just get passed on to the subprocess?

See also Supervisor's pidproxy program for this: https://github.com/Supervisor/supervisor/blob/master/supervisor/pidproxy.py

I've been running rake tasks with envdir like so:

envdir envs/prod rake blahblahblah

For some reason, rake stops autocompleting when used with envdir.

Do you think this has to do with envdir or the way the autocomplete script is written?

It seems that envdir will fail completely if the user does not have read permission on a file in the envdir. It may or may not be desirable for envdir to ignore unreadable files.

There are at least two cases in which ignoring unreadable files is desirable:

• it's potentially fairly easy for an unreadable file to be created by accident; I don't my system to break if I accidentally leave a file owned by me or root mode 600 in that directory.
• to allow e.g. a django management command to access values which should not be accessible to the webserver user.

I'd like envdir to ignore files it can't read. I guess some may desire unreadable files to cause execution to abort - maybe if a value is not set then some existing programs will not behave correctly (although I'd suggest explicitly checking such things) - so perhaps behaviour needs to be configurable?