Django development server not shutting down promptly. about s2i-python-container HOT 16 CLOSED

Graham, I think this is due to the threading model that is used in django dev server now.

However, is this something that anyone cares about, since someone could deploy in prod w/o Openshift, and still cause themselves trouble.

J

from s2i-python-container.

processes under openshift get a sigterm which isn't the same as cntrl-c, i assume if you sent a sigterm to a django process you'd see similar behavior.

from s2i-python-container.

Sending a SIGTERM to the Django development server on MacOS X causes it to die immediately.

$ python manage.py runserver
Performing system checks...

System check identified no issues (0 silenced).
April 08, 2016 - 04:38:15
Django version 1.9.5, using settings 'mydjangosite.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
Terminated: 15

from s2i-python-container.

not sure why the openshift container isn't dying then. any indication in the logs as to what it's doing for those 30s?

from s2i-python-container.

that is odd, since on Fedora 23 it does the same.

I checked the logs as well, and the only thing that happens after 30s is a
messages about force-ably shutting down the container happens. Whereas
with the gunicorn implementation it's immediate.

It only seems to happen after any requests have been handled, so that's why
i think its around threads.

Just my .02

On Thu, Apr 7, 2016 at 9:51 PM, Ben Parees [email protected] wrote:

not sure why the openshift container isn't dying then. any indication in
the logs as to what it's doing for those 30s?

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#93 (comment)

from s2i-python-container.

that is odd, since on Fedora 23 it does the same.

@jfmatth sorry, it does the same as what? and are you referring to when running in a container or not in a container?

from s2i-python-container.

@bparees Sorry Ben, I tested it under BASH and sent the signal to python.

I was going to ask, is there a way to download the STI built container and test against that locally. I don't have a full STI dev environment :(

J

from s2i-python-container.

@jfmatth i'm afraid we don't have a built application image on dockerhub that i'm aware of. maybe @GrahamDumpleton does have one though.

but in anycase if the behavior is the same when run directly (not in a container) i'm not sure we'd do anything here.

from s2i-python-container.

@bparees I think I'll try and build a local container then and run the dev server and see what happens then. It's too bad we can't turn up debugging on the environment on when it shuts down containers, but it is pretty clear about waiting the 30s and then forcing their destruction :)

J

from s2i-python-container.

This issue has got nothing to do with either use of threading or the automatic reloading mechanism in the Django development server. If you supply the --noreload and/or --nothreading options it makes no difference.

The only thing that results in it working is not to run it as process ID 1.

So long as it is running as process ID 1, deleting the pod, or getting a shell terminal in the container and sending process ID 1 a TERM signal, doesn't result in it exiting. If the pod is deleted, it only gets killed after 60 seconds when the delete times out and destroys the container.

If however you run it with a mini init process manager, even the shell script:

trap 'kill -TERM $PID' TERM INT

python manage.py runserver 0.0.0.0:8080 &

PID=$!
wait $PID
trap - TERM INT
wait $PID
STATUS=$?
exit $STATUS

then it will shutdown properly when a TERM signal is received.

Interestingly, it is only the TERM signal which is a problem when it is running as process ID 1. If you send process ID 1 an INT signal from within the container, it will shutdown straight away.

So there is something mighty strange about the Django development server code. Not only doesn't it reap zombie processes when run as process ID 1, it doesn't shutdown properly shutdown when it receives a TERM signal either.

from s2i-python-container.

Could it be python in General? I know you've seen the code, and its a call to the stdlib so maybe its Python?

from s2i-python-container.

So there is something mighty strange about the Django development server code. Not only doesn't it reap zombie processes when run as process ID 1, it doesn't shutdown properly shutdown when it receives a TERM signal either.

thanks for the detailed investigation @GrahamDumpleton, sounds like we need to move this to an upstream(django) issue. We do not want to start writing mini init scripts and doing signal forwarding/etc.

from s2i-python-container.

I opened an upstream Django PR to get this fixed: django/django#9338

from s2i-python-container.

Unfortunately, this is still a problem. The mentioned upstream issue was closed because there was a discussion/plan about gunicorn as a default development web server in Django but nothing happened and I cannot find any up-to-date info about that.

Should we either reconsider any implementation in the Python container or reopen the upstream issue?

from s2i-python-container.

I going through all issues in this repository and it seems that this one can be solved by implementing init wrapper as discussed in #94 and proposed in #233

from s2i-python-container.

Optional init wrapper implemented in #322

from s2i-python-container.