Django manage.py migrate will cause corruption to databases. about s2i-python-container HOT 14 CLOSED

Here is my current rambling thoughts on all of this after having actually gone through the work of deploying some decently complicated Python web applications as well as simple ones. No doubt you will get lost and wonder what I am raving about, but I can try. :-)

The first of the more complicated applications was the Wagtail CMS product for Django. This was deployed with PostgreSQL for database and Redis for key/value store.

The second was Kallithea. This is implemented using Pylons. This was deployed with PostgreSQL for the database. In this case Kallithea also needed its own persistent volume for direct data storage.

I had two variations on the Kallithea deployment. One which had the Kallithea and PostgreSQL in separate pods. This allowed the Kallithea web front end to be scalable, but used two separate persistent volume claims. The other deployment was a lite version. This had Kallithea and PostgreSQL as separate containers in the one pod. This was not scalable in any way, but allowed them to both share the one persistent volume claim, and work within the one pod memory limit. This meant less resources being used if you didn't need a scalable solution.

Both of these were more or less set up as what you would call instant starter apps. That is, you deploy it and it immediately works. In the case of the Wagtail example you would clone an initial repo and were still able to customise the code afterwords to enhance it, where as Kallithea was a SaaS like product which you used as is.

What was really interesting about the Kallithea example, as far as the discussion here about migration goes, was that you had to use different strategies for each variant

Looking at Kallithea first. This is not a Django application, it is implemented in Pylons.

For this application, database initialisation and database migration are separate commands. This is different to Django where it is just the one command.

Further, the database initialisation command in Kallithea, which is:

paster setup-db .... lots of arguments

is destructive. That is, you run it a second time and it will drop your database tables and recreate them. It isn't like Django migrate command where it looks to see if stuff already exists and sees something and that you are up to date and leaves it alone.

The actual migrate command is:

paster upgrade-db .... lots of arguments

So the first issue is that the default S2I builder only attempts to support Django and nothing else.

The lack of an action hooks mechanism though makes it a pain to actually attempt to automate in any way triggering of even the database migration for something other than Django based application automatically as part of deployment.

In order to do that you have to create your own .s2i/bin/run script which runs your deploy step and then invokes the original S2I run script. In doing this, you hope that the run script hasn't set any environment variables or done anything else that you might need to be setup first, cause your stuff comes in before that. One such thing which is a problem is the libnss_wrapper stuff, which because it isn't done in BASH_ENV and ENV (see #96), means your Python scripts could blow up if not designed to run as a UID with no user account.

Even though I really think there should be a proper action hooks mechanism, and one which in my experience needs to go beyond what OpenShift 2 provided as far as hook points, is it even safe to be doing migrations in a deploy hook? The short answer to that is that in most cases no.

The Django database migration at this point in time may be sort of safe if run more than once at the same time, but it will still cause problems in a Rolling deployment because you will have old and new code running at the same time in different pods. Even if database migration is successful in itself, you can break old instances of the application, which could have unknown effects and result in data being stuffed up in some way.

The problem here is that Rolling deployments is the default. Thus the default behaviour is 'we may break your system'.

Now lets look at that paster upgrade-db command. What do we know about how that works. The answer is absolutely nothing. We don't know if it is safe to run multiple times at once. The same goes for any other Python application database migration scripts.

What we do not want to do is set an example which shows using a deploy hook to handle a database migration is somehow always okay. People will get the idea that that is fine and copy it for whatever they may be using and then break things. You must be really up front and say it isn't safe to do it that way if you don't want users hating you.

What can be done which is safe? First off, you shouldn't use Rolling deploys as a default strategy if you have database migrations needing to be done. You should use Recreate strategy. The database migration should then be run as a mid deployment hook.

Two problems with this though. The first is that they can't just run the migration command directly, for example as paster upgrade-db from the mid hook definition. This is because that is a Python script and will not trigger BASH_ENV and ENV mechanisms for setting up a shell environment.

So they have to write their shell script and add it to their code and invoke that.

The second problem arises in the lite variant of Kallithea I described above. In this case the PostgreSQL database and the Kallithea front end were in the same pod.

Even though you can set up the mid deployment hook under the Recreate strategy, in this arrangement with multiple containers in the same pod, the database will not be running when the mid hook is run. The only thing which will exist is the transient mid hook container which would be using the Kallithea image. This is because Recreate applies across whole pod, so database is also shutdown on new deployments.

The mantra seems to be that you should only ever have one container per pod, but as I use OSE more I am starting to disagree with that blanket stance. There are a number of use cases where it is convenient to have multiple containers per pod, mostly for side car containers, but as a way of keeping down resource allocations for small instant apps is another. It makes it easier to deal with the front end and database as one unit.

In this situation you don't have a choice but to still do it from a deploy action hook in the front end container prior to starting the web application. You still want to be using Recreate strategy though.

Since I have one code base for Kallithea for the two different deployment modes, this means having an environment variable to optionally trigger the migration from the deploy action hook. At the same time I have the separate migration script which can be triggered from a mid deployment hook when database and front end are in separate pods.

Now we get to database creation.

The only way one can automate this is by having a persistent data marker which can be checked before running the paster setup-db command. That is, check to see if marker exists, if it doesn't then create the marker and then run database initialisation.

There obviously is a race condition in here if you aren't careful and the only way to deal with that at a higher level is not to run that script more than once at the same time.

Another problem though is where do you store the persistent marker. Luckily in the Kallithea example the front end needed access to a persist volume, so that can be used.

For the single pod scenario, one can thus run database setup from the deploy hook as well, prior to the migration step, you just need to check that initialisation marker and not run it a second time if it exists. Because Recreate strategy is being used and it isn't a scalable application, it is safe to do this.

For the multi pod variant, you can trigger database setup from the pre hook. This works, because the pre hook is still run even on the very first deploy. For that very first deploy there will be no existing instances of the application running. You could also technically have done it in the mid hook as well so long as only gets done on first deployment, but pre hook is better if that isn't used for anything else because you can after the first deploy then remove the pre hook definition from the deployment config and avoid any risk of it being rerun if someone deletes that marker indicating it was run.

Anyway, that is one example, with two variants and it isn't simple. This requires templates. There is no way you can do it from a plain oc new-app with the S2I builder directly.

The direct oc new-app invocation of the S2I builder case all falls apart with databases anyway due to the mess involved with having to set up database environment variables for both front end and database, especially where different names are required making anything automatic impossible.

You basically can't avoid using templates which as a starting point when want to try and have an instant app involving both a front end and database. Even then a template in itself is useless as you are most likely going to need a starter Git repository for the framework with all the changes in the code to look for the correct database environment variables anyway.

If you are going to have to have both a template and a Git repository for cloning to start with, have automatic database migration as part of the builder gets you nothing for the initial deployment. It is going to be of no real value for simple case where you don't use templates as you are going to still have lots of manual steps to do anyway.

Now back to the Wagtail example. This was a Django application and so there is only the one migrate command which can both create database as well as migrate.

For that application no persistent volume was needed for Wagtail itself. This meant one couldn't use a marker file in the file system to gate running the migrate script. You could try and get elaborate and actually connect to the database and query for existence of certain tables to determine if should run.

This highlights issues though which has relevance to even what was done above for the custom case where no inbuilt support for migration relied on. The first is that you have no idea what database type is being used. You can't write a script to check database availability before running a database migration, let alone checks to see if certain tables existed. Plus what database tables should be checked for to gate database creation are going to be application specific.

So what you end up with is that you can't even do database setup/migration reliably for Django where you know the command, as that isn't going to check whether the database exists and keep trying, perhaps indefinitely, but perhaps with timeout.

For the Kallithea instant app case, because you can't be certain the database may have started up in time, you had to before any attempt to setup or migrate the database, try and do some innocuous query against the database to check it works. I had for example a Python script which would first connect to PostgreSQL and do a SELECT 1 query before attempting setup or migrate. If not, if the database doesn't work, your deploy breaks. Better to try waiting at that point rather than outright failing.

The other thing with Django database setup, is that this invariably has to be followed up by creating a super user account. Django is a bit of a pain in this respect as it can't be easily scripted as the createsuperuser admin command expects to be run interactively. You would have to create a special Python script which import Django and uses its data model classes to add a user and commit it.

So more fiddling if you want to automate it so it can pull the initial user details from environment variables.

How much of all these variations are you going to try and cater for. Are you going to have only Django as a first class citizen with everyone else left out because adding action hooks is a pain.

For me I just don't believe having the S2I scripts by default try and do database initialisation and migration is worthwhile in the greater scheme of things. I believe that for anything but non database hello world applications, things are going to be driven by templates which have been setup by knowledgeable people, along with starter Git repositories, or instructions of how to first modify an existing application to meet any requirements.

What helps though in this is a proper action hooks mechanism. This needs to be more developed than the OpenShift 2 incarnation.

Two types of action hooks it should have are setup and migrate. These are for database and other application setup, such as user creation, plus database migration.

Although the hooks are provided, neither assemble or run would trigger them by itself.

What the image should provide though is an easy way of calling into the image to trigger those scripts. That way they can be easily called from lifecycle hooks with a built in wrapper script ensuring all the environment variables etc are set up correctly before they are run and not be dependent on the BASH_ENV and ENV fiddle.

For the single pod case I had, presuming flag enabled for my case, then setup and migrate would be run from the deploy action hook in the container.

So rather than muck around trying to work out how to embed database migration into the S2I builders, you should add an advanced action hooks mechanism to make it easy for users to define themselves what the setup and migration steps are and then make it easy to trigger them from life cycle hooks, other action hooks, or even manually using oc rsh.

Think action hooks is too complicated or because it is felt then that all builders must agree on how it is done, I will not be overly concerned as I will keep using my own S2I builder scripts anyway as I have all this stuff and much more already. :-)

from s2i-python-container.

I've been running migrations in the "pre" Lifecycle Hook. Makes a lot more sense to me.

• Runs just once instead of every pod
• Able to abort the deploy if fails
• Decreases start time for server pods - which makes it easier to have deploys with no downtime

I did notice I had to have the command be set to something like ./run_migration.sh instead of simply ./manage migrate --noinput. No idea why. Running the command directly would result in python 2 running (I'm using 3.4).

Anyway I'd suggest the default be not running at the start of the pods and updating django-ex to have a pre deploy hook that runs migrate.

from s2i-python-container.

Doesn't the pre lifecycle hook run while the existing pods are still running?

There is now a mid hook, which if you use a Recreate strategy for deployment will be done after pods are stopped and before new ones are started. This would be the most sensible place to do migration.

An issue with the mid hook is that if it fails, it will rollback to previous build of application. If the failure occurred due to database being mucked up and so left in an inconsistent state, the previous version of application could fail and further problems could occur.

from s2i-python-container.

It does run while existing pods are running. I'm using rolling with multiple pods - so no matter what I do the database could be out of sync for some of the pods. But I get no downtime assuming I plan my migrations carefully (or simply don't care about statistically unlikely scenarios). I'm using Postgres with transactional DDL so the rollbacks should work perfectly in my setup.

If not using a transactional DDL - you'd have problems no matter what. If a migration fails half way through at the start of the pod you are still in a broken state where the current and previous deploys don't match the actual schema.

Perhaps there is no default that will please everyone. Some people might prefer a mid hook scenario as you describe where slight downtime is better than out of sync pods and schemas. I don't think on pod start makes sense for most people though.

Heroku for comparison doesn't run migrations at all by default. I like the idea that the default examples just work though - which would be nicer than Heroku.

from s2i-python-container.

I've discussed with folks in a couple of occasions in the past about running migrations as part of the run script, pre/post (and now mid) deployment hooks and not running them at all.

My personal conclusion is similar to what I read from the discussion above, quoting @bufke:

Perhaps there is no default that will please everyone.

One of the aspects not mentioned yet is the experience of doing oc new-app https://github.com/ponies/unicorn to instantiate a new Django application in an OpenShift cluster.

A user provides source code, and oc new-app does do its magic for language detection (or you can explicitly provide a builder image), but it won't magically figure out deployment hooks, and I don't think we should try to do that.

So using a deployment hook means that you'd need to use a template, and oc new-app https://github.com/openshift/django-ex would not work out-of-the-box as it does today...

The safe side, I agree with @GrahamDumpleton, would be to disable automatic migration altogether, and encourage users to follow careful migration steps as @bufke mentions.

@bufke I share the pain of using the images with oc exec and docker exec to run commands like ./manage.py migration without going through hops to get it to work.
Here's a PR that might be of your interest, as it would make the simple and obvious command every Django developer would try to use just work: #108

This topic on running database migrations does go beyond sti-python. @bparees what are your current thoughts on this?

from s2i-python-container.

@rhcarvalho first a question: if we do not run migrations by default, I assume the oc new-app flow won't work at all (the app won't come up because the initial migration wasn't executed)?

I think i can see an argument for changing the default to not migrate (but still controlling it via an env variable), since you can still have a new-app flow that way (but you would need to add a -e DISABLE_MIGRATION=false arg to new-app). At least it's still a one-liner. Forcing users down the path of defining deployment hooks feels burdensome, though it is the "right thing to do" in terms of how we'd really recommend someone use the system.

i'm just worried about scaring off the initial user who sees that they can't even get a basic django app running without either a template, or having to go edit their deploymentconfig (i guess soon they'd be able to "oc set deploymenthook", but it's still not trivial to know how to define one properly).

tl;dr: i think i'd vote for changing the default but leaving it env variable controlled. That's still going to break users who use new-app though, unless they know to pass the "-e" arg.

so my actual first vote might be to leave it, until we know django is going to break this, because users who genuinely care about migrations are going to dig into the platform/scripts and understand what it's doing anyway. Our out of box examples are not production ready and never will be.

from s2i-python-container.

@bparees if I remember well the documentation in Heroku, and what the Django Girls tutorial recommends for folks deploying their apps in PythonAnywhere is to run the migration step manually after deploying the files to the cloud.

So disabling the auto migration in run script would not "break users" in the sense of making the demo project completely unusable, but it would change the experience to what they get in other places...

Once we get oc set deployment-hook, documentation on how to configure the platform to run migrations automatically might get more palatable, while will still always require users to "know what they're doing", so they can opt-in consciously depending on their deployment strategy, app, and other constraints.

from s2i-python-container.

So disabling the auto migration in run script would not "break users" in the sense of making the demo project completely unusable, but it would change the experience to what they get in other places...

right, but it would break users compared to the experience we've been presenting them with today. and i'd argue that what heroku/etc recommend is less than ideal, seems like if we can offer a better/more seamless experience, we should.

from s2i-python-container.

if we can offer a better/more seamless experience, we should.

I agree. But how can we make it so that this experience goes beyond the hello world demo?

That's where I think automatically running schema migrations gets hairy.

Different companies, different projects, etc, have different strategies, I'm not aware of anything that's a good default for the no-user-intervention case.

from s2i-python-container.

Quick question. I will follow up with more detailed response on some issue related to this afterwards.

What is oc set deployment-hook going to set up? Will that be a way of defining a hook (like one can for pre), but where it will only be run on the very first deployment? Thus purpose built for initial database setup.

Noting that if this is the case, it is sort of a useless command given that application is likely already deployed before you can run it. Having such a hook in config for use in templates would be handy though.

from s2i-python-container.

What is oc set deployment-hook going to set up? Will that be a way of defining a hook (like one can for pre), but where it will only be run on the very first deployment? Thus purpose built for initial database setup.

no, it'll just replace you running "oc edit dc mydc" and typing in a deployment hook definition. much like oc set env, etc.

from s2i-python-container.

@GrahamDumpleton this is the Trello card I was referring to: https://trello.com/c/YpZLKSAX/

from s2i-python-container.

Another issue to note if doing database migrations automatically is that some application frameworks based on Django actually include scaffolding such that when you run python manage.py runserver, on basis that only used for development, that a side effect of runserver will be that migrate will be run first automatically. This will result in migrate being run twice if the S2I run script also does it.

Am mentioning this just in case someone sees this and is confused about why they see it running twice.

Package have seen that do this are:

• http://opal.openhealthcare.org.uk/docs/

from s2i-python-container.

As several posters mentioned above, this issue is a balancing act between ease of use for beginners and experienced users. As experienced users running multiple pods on openshift are more likely to read and understand the documentation, we decided in PR #230 to leave the migration on by default but document the option better.

from s2i-python-container.