klen / peewee_migrate Goto Github PK
View Code? Open in Web Editor NEWSimple migration engine for Peewee
License: MIT License
Simple migration engine for Peewee
License: MIT License
Hey, I'm trying to change a column like so...
migrator.change_fields('user', email=pw.CharField(max_length=100, null=True))
But I end up with the following error...
Traceback (most recent call last):
File "scripts/reload_local.py", line 108, in <module>
router.run()
File "/usr/local/lib/python3.5/dist-packages/peewee_migrate/router.py", line 158, in run
self.run_one(mname, migrator, fake=fake, force=fake)
File "/usr/local/lib/python3.5/dist-packages/peewee_migrate/router.py", line 129, in run_one
migrate(migrator, self.database, fake=fake)
File "<string>", line 46, in migrate
File "/usr/local/lib/python3.5/dist-packages/peewee_migrate/migrator.py", line 102, in wrapper
return method(migrator, migrator.orm[model], *args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/peewee_migrate/migrator.py", line 184, in change_columns
self.ops.append(self.migrator.change_column(
AttributeError: 'MySQLMigrator' object has no attribute 'change_column'
I'm using MySQL.
Thanks for any help!
Cheers
So in one of my models I have a field like this;-
bank_balance= DecimalField(max_digits=20,decimal_places=2,auto_round=True,null=True)
The migration creates a line as follows;-
bank_balance = pw.DecimalField(auto_round=True, decimal_places=2, max_digits=20, null=True, rounding=ROUND_HALF_EVEN)
Which is correct. However the migration needs a line like this;-
from decimal import ROUND_HALF_EVEN
to import the correct enumeration symbol. I presume if one of the other rounding options is chosen, that would need to be imported too.
On a side note, I notice the default= option doesnt seem to perculate across to the migrations (Specifically in my created_on=DateTimeField(datetime.datetime.now)
field) Not sure if thats documented, will perhaps investigate later if I get a chance.
Best regards, and thanks for this great little library.
The --auto
flag to scaffold a set of models and generate a migration appears broken when a solution lacks an example
and tests
module.
Based on what I've read in router.py
it appears that the culprit is here: https://github.com/klen/peewee_migrate/blob/develop/peewee_migrate/router.py#L81
In this case, I was missing an example
module; once I added this, then --auto
appears to work. Currently it appears both a example
and tests
module are required in order for --auto
to work.
I'm happy to submit a PR and remove this line if it's desired, but, I'm wondering whether there's something I'm missing concerning this line of code.
Thanks.
Since migrations/
directory will typically contain only Python files generated by the migrator (with certain formatting rules), it would be helpful if migrations/.editorconfig
was automatically created with formatting rules explicitly stated:
[*.py]
indent_style = space
indent_size = 4
More on the format: http://editorconfig.org/
models.py
class Group(BaseModel):
name = pw.CharField(max_length=255, null=False, unique=True)
parent = pw.ForeignKeyField('self', null=True, related_name='children')
description = pw.CharField(max_length=255, null=True)
def __str__(self):
return self.name
migrations/001_auth.py
def migrate(migrator, database, fake=False, **kwargs):
@migrator.create_model
class Group(pw.Model):
name = pw.CharField(max_length=255, unique=True)
parent = pw.ForeignKeyField(db_column='parent_id', rel_model=migrator.orm['group'], related_name='children', to_field='id')
description = pw.CharField(max_length=255, null=True)
class Meta:
db_table = "group"
errors
➜ ams-api git:(master) ✗ python manage.py migrate
Traceback (most recent call last):
File "manage.py", line 71, in <module>
cli()
File "/Users/tux/.pyenv/versions/3.6.1/envs/sanic-ams-api/lib/python3.6/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/Users/tux/.pyenv/versions/3.6.1/envs/sanic-ams-api/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/Users/tux/.pyenv/versions/3.6.1/envs/sanic-ams-api/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/tux/.pyenv/versions/3.6.1/envs/sanic-ams-api/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/tux/.pyenv/versions/3.6.1/envs/sanic-ams-api/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "manage.py", line 32, in migrate
migrations = router.run(name, fake=False)
File "/Users/tux/.pyenv/versions/3.6.1/envs/sanic-ams-api/lib/python3.6/site-packages/peewee_migrate/router.py", line 170, in run
self.run_one(mname, migrator, fake=fake, force=fake)
File "/Users/tux/.pyenv/versions/3.6.1/envs/sanic-ams-api/lib/python3.6/site-packages/peewee_migrate/router.py", line 140, in run_one
migrate(migrator, self.database, fake=fake)
File "<string>", line 94, in migrate
File "<string>", line 96, in Group
KeyError: 'group'
Command: pw_migrate create --auto my.models --database postgresql://localhost:5432/name auto-generated.py
migrations = diff_many(models, source, migrator, reverse=reverse)
File "env/lib/python3.6/site-packages/peewee_migrate/auto.py", line 115, in diff_many
models1 = pw.sort_models_topologically(models1)
File "playhouse/_speedups.pyx", line 345, in playhouse._speedups.sort_models_topologically (playhouse/_speedups.c:7541)
File "stringsource", line 67, in cfunc.to_py.__Pyx_CFunc_tuple____object___to_py.wrap (playhouse/_speedups.c:8690)
File "playhouse/_speedups.pyx", line 327, in playhouse._speedups._sort_key (playhouse/_speedups.c:7101)
AttributeError: type object 'Model' has no attribute '_meta'
Is this a Python 3 issue?
Or it's related to inheritance issues?
class BaseModel(Model):
class Meta:
database = db
class User(BaseModel):
pass
I have so-called base model for easy db linking:
class Model(peewee.Model):
class Meta:
abstract = True
database = db
Creating migrations produces:
@migrator.create_model
class Model(pw.Model):
id = pw.AutoField()
class Meta:
table_name = "model"
Hi there,
first of all thank you so much for taking the time and publishing this library, appreciate it a lot since peewee doesn't come with automatic migrations included.
I'm trying to use the cli to migrate my model, but load_models
raises an Exception, saying that it can't import the models module (the one at the --auto path).
This is the failing command:
$ pw_migrate create --auto ~/myproject/model/ --database "postgres://zudibytvipthsa:c8c2d7f0blabla" --directory ~/myproject/model/ mymigration
Could it be that peewee_migrate doesn't resolve the database connection URL with playhouse.db_url.connect
but expects something else instead? This isn't really clear from the help text.
My model module is structured as follows:
model/
basemodel.py <-- containing:
class BaseModel(Model):
class Meta:
database = db
.. more modules that inherit from BaseModel
I would appreciate getting some hints as to what might be causing this :)
I made init migration file and i tried to run migrate on it but there was error: peewee.OperationalError: unable to open database file.
I used this command: pw_migrate migrate --name=001_mymigration.py --database=sqlite:C:\Users\zurek\Documents\BackBone\Alto\afs\data\database.db --directory=C:\Users\zurek\Documents\BackBone\Alto\afs\data\migrations
where --name is name of migration file, database is directory to database.db sqlite file and directory where migrations are. I use windows 10. Do you know where the problem might be? Some proposals? Thank you very much.
In my standard use case, I have no need for rollback and no need for peewee's abstractions, really, but I would like to keep that as an option. To reduce boilerplate ever so slightly, I would like to load raw .sql
files from directory, as an alternative to python scripts.
Would you be willing to merge something like this if I created a PR:
class SqlRouter(Router):
filemask = re.compile(r"[\d]{3}_[^\.]+\.(py|sql)$")
@property
def todo(self):
"""Scan migrations in file system."""
if not os.path.exists(self.migrate_dir):
self.logger.warn('Migration directory: %s does not exists.', self.migrate_dir)
os.makedirs(self.migrate_dir)
return sorted(
f.rpartition('.')[0] for f in os.listdir(self.migrate_dir) if self.filemask.match(f))
def read(self, name):
"""Read migration from SQL or python file."""
sql_file = os.path.join(self.migrate_dir, name + '.sql')
if os.path.isfile(sql_file):
data = ''
with open(sql_file) as f:
for line in f.readlines():
if line and not line.startswith('--'):
data += line
return lambda mg, db, *args, **kwargs: db.execute_sql(data), VOID
return super().read(name)
Note that the only thing needing change in todo()
is from ''.join(f[:-3])
to f.rpartition('.')[0]
to support different length of file suffix. That could be improved in Router
already.
We are using the BinaryJSONField
from the Playhouse module for PostgreSQL (see http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#postgres-ext).
The migrations file contains then the following:
@migrator.create_model
class Result(pw.Model):
energy = pw.DoubleField()
task = pw.ForeignKeyField(db_column='task_id', rel_model=Task, to_field='id')
filename = pw.CharField(max_length=255, null=True)
data = pw.BinaryJSONField(index=True, null=True)
while it should maybe be something like:
import playhouse.postgres_ext as pw_pext
[...]
@migrator.create_model
class Result(pw.Model):
energy = pw.DoubleField()
task = pw.ForeignKeyField(db_column='task_id', rel_model=Task, to_field='id')
filename = pw.CharField(max_length=255, null=True)
data = pw_pext.BinaryJSONField(index=True, null=True)
In the original models file I have:
from playhouse.postgres_ext import BinaryJSONField
The documentation is very lacking so it's not clear at all.
Firstly, in the def migrate(migrator, database, fake=False, **kwargs):
method, migrator
is of type peewee_migrate.migrator.Migrator
, so I have to use migrator.migrator
to access the peewee_migrate.migrator.SqliteMigrator
so it doesn't crashes. Is this the intended behavior?
Secondly, I'm using migrator.migrator.drop_not_null('user', 'username')
and nothing is happening. Is drop_not_null
supported or not?
It always creates empty migrate
and rollback
methods.
If you have a model with some field that does not have a default value, and you then modify that model to include a default value, the generated migration for that will fail.
It fails because the resulting PSQL query is an invalid ALTER
statement.
Following the formatting in #87
models.py
:class Test(Model):
class Meta:
database = DB
only_save_dirty = True
id = AutoField()
name = CharField(max_length=123, index=True)
$ pw_migrate create --auto --directory test/migrations --database $DATABASE_URL --auto-source test test1
$ pw_migrate migrate --directory test/migrations --database $DATABASE_URL --name 001_test1
models.py
so that the name field now has a default:name = CharField(max_length=123, index=True, default='D')
Result: the generated PSQL query will be this invalid statement:
ALTER TABLE "test" ALTER COLUMN "name" VARCHAR(123) TYPE DEFAULT 'D'
Something like this PSQL statement would work and is would be more appropriate.
ALTER TABLE "test" ALTER COLUMN "name" SET DEFAULT 'D';
Version:
$ pip list | grep peewee
peewee 3.9.3
peewee-migrate 1.1.5
Thanks!
I run functions:
`router = Router(chat_app_db)
# Create migration
router.create('migration_name')
# Run migration/migrations
router.run('migration_name')
# Run all unapplied migrations
router.run()`
but it just create table migratehistory
I have a migration with the following code:
def migrate(migrator, database, fake=False, **kwargs):
"""Write your migrations here."""
MyModel = migrator.orm['my_table']
type_new = pw.CharField(null=True, constraints=[pw.Check(
"type_new in ('type1', 'type2', 'type3')")])
migrator.add_fields(MyModel, type_new=type_new)
migrator.sql('UPDATE my_table SET type_new=type')
migrator.remove_fields(MyModel, 'type')
migrator.rename_field(MyModel, 'type_new', 'type')
migrator.add_not_null(MyModel, 'type')
And when appllying it I get following error:
ValueError: type_new is not null but has no default
I'm explicitly passing null=True
, but peewee-migrate claims this field to be non-null, how it is possible? Is it some bug or am I missing something?
I want to change the default for ArrayField
. I have following lines in my migration (simplified):
def migrate(migrator, db, fake=False, **kwargs):
"""Write your migrations here."""
_, _ = fake, kwargs
class Postfilter(BaseModel):
actions = ArrayField(IntegerField, default=[2, 4, 11])
db_proxy.initialize(db)
migrator.orm['postfilter'] = Postfilter
migrator.add_default('postfilter', 'actions', [10, 2, 4, 11])
Migration fails:
...
peewee.ProgrammingError: column "actions" is of type integer[] but expression is of type record
LINE 1: UPDATE "postfilter" SET "actions" = (10, 2, 4, 11)
^
HINT: You will need to rewrite or cast the expression.
pw_migrate migrate --name THIS_FILES_DOES_NOT_EXIST.py --database $POSTGRES_URI
No error message at all. I believe there is some try/except somewhere. We should see an error like "this file does not exist"
If we use Peewee without peewee_migrate, we would just execute this: db.create_tables([Model1, Model2])
So, if we use peewee_migrate, we should avoid this command and only use peewee_migrate, right?
Also, does Peewee_Migrate use Peewee (http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#migrate) for migrations, right? Any doc about how to create the migration files?
Thanks
How I can customize migration template and include my custom fields?
Migration generated using pw_migrate create --auto models --database mysql://user:pass@localhost/test --directory migrations test
fails to apply with the error peewee.IntegrityError: (1215, 'Cannot add foreign key constraint')
.
Database: MySQL
mysql> SHOW VARIABLES LIKE "%version%";
+-------------------------+-------------------------+
| Variable_name | Value |
+-------------------------+-------------------------+
| innodb_version | 5.7.22 |
| protocol_version | 10 |
| slave_type_conversions | |
| tls_version | TLSv1,TLSv1.1 |
| version | 5.7.22-0ubuntu0.16.04.1 |
| version_comment | (Ubuntu) |
| version_compile_machine | x86_64 |
| version_compile_os | Linux |
+-------------------------+-------------------------+
models.py
class BaseModel(Model):
class Meta:
database = database
class Model1(BaseModel):
id = BigAutoField()
class Meta:
table_name = 'tb_name1'
class Model2(BaseModel):
id = BigAutoField()
site = ForeignKeyField(column_name='model1_id', field='id', model=Model1, null=True)
class Meta:
table_name = 'tb_name2'
migrations/001_test.py
import datetime as dt
import peewee as pw
try:
import playhouse.postgres_ext as pw_pext
except ImportError:
pass
def migrate(migrator, database, fake=False, **kwargs):
"""Write your migrations here."""
@migrator.create_model
class BaseModel(pw.Model):
id = pw.AutoField()
class Meta:
table_name = "basemodel"
@migrator.create_model
class Model1(pw.Model):
id = pw.BigAutoField()
class Meta:
table_name = "tb_name1"
@migrator.create_model
class Model2(pw.Model):
id = pw.BigAutoField()
site = pw.ForeignKeyField(backref='model2_set', column_name='model1_id', field='id', model=migrator.orm['tb_name1'], null=True)
class Meta:
table_name = "tb_name2"
def rollback(migrator, database, fake=False, **kwargs):
"""Write your rollback migrations here."""
migrator.remove_model('tb_name2')
migrator.remove_model('tb_name1')
migrator.remove_model('basemodel')
Output of SHOW ENGINE INNODB STATUS;
:
------------------------
LATEST FOREIGN KEY ERROR
------------------------
2018-05-30 10:27:24 0x7fdfd835c700 Error in foreign key constraint of table bench_test/tb_name2:
FOREIGN KEY (`model1_id`) REFERENCES `tb_name1` (`id`)):
Cannot find an index in the referenced table where the
referenced columns appear as the first columns, or column types
in the table and the referenced table do not match for constraint.
Note that the internal storage type of ENUM and SET changed in
tables created with >= InnoDB-4.1.12, and such columns in old tables
cannot be referenced by such columns in new tables.
Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-foreign-key-constraints.html for correct foreign key definition.
Given the following simple model, generated migration has a typo:
class User(BaseModel):
active = BooleanField(default=True)
Migration I get (the relevant part):
@migrator.create_model
class User(pw.Model):
id = pw.AutoField()
active = pw.BooleanField(constraints=[SQL("DEFAULT True")])
This fails, but replacing SQL with pw.SQL works like a charm. I can't find the code which generates it, so if you help me debug this issue, I'll be happy to send PR.
I added a "multi-column index unique" definition according to the PeeWee documentation: http://docs.peewee-orm.com/en/latest/peewee/models.html#indexes-and-constraints but when I run peewee_migrate it doesn't generate the statements for the multi-column index.
I was taking a look at your code (auto.py) and looks like you are not considering that case. It would be possible to add support for multicolumn index/unique ?
Adding manually the sentence using migrator.add_index(model, *col_names, unique=False)
works but this means that I have to manually maintain autogenerated scripts and the other one for multi column indexes.
This is my model:
class Dog(BaseModel):
id = PrimaryKeyField()
user = ForeignKeyField(User, to_field='id', related_name='dogs', db_column='user_id')
name = CharField(null=False)
description = CharField(null=False)
class Meta:
indexes = (
(('user', 'name'), True), # This should generated a unique constraint of those two columns
)
Hello!
In the docs you state as only dependency: "python 2.7,3.3,3.4", which I accomplish.
But there's also a dependency of peewee >= 2.8.5, as specified in requirements.txt. I'm using peewee 2.6 so I can't use this library.
Do you think you could lower your peewee requirement to 2.6.0 or it is a must to have it set on 2.8.5?
In that case, perhaps it would be a good idea to write it down on the requirements section of the docs page :).
Thank you very much,
Miguel.
I noticed, we can write conf file in migration directory and setup settings about DB for example - https://github.com/klen/peewee_migrate/blob/develop/peewee_migrate/cli.py#L26
If I create migrations/conf.py with
DATABASE = 'postgresql://user:pass@host/db'
it works fine.
But if I want to confugure DATABASE variable from settings.py (for example) like this
import settings
DATABASE = 'postgresql://{DB_USER}:{DB_PASSWORD}@{DB_HOST}/{DB_NAME}'.format(
DB_HOST=settings.DB_HOST,
DB_NAME=settings.DB_NAME,
DB_USER=settings.DB_USER,
DB_PASSWORD=settings.DB_PASSWORD,
)
I got
ModuleNotFoundError: No module named 'settings'
Are there any solutions of this problem?
When I try to run all unapplied migrations, I get the following error:
peewee.ProgrammingError: relation "migratehistory" already exists
My python script to run all migrations:
db_connection.init_database()
db_connection.connect()
database = db_connection.get_database()
router = Router(database)
# Run all unapplied migrations
router.run()
After quick investigation to the source of peewee_migrate I found this from BaseRouter
:
@cached_property
def model(self):
"""Initialize and cache MigrationHistory model."""
MigrateHistory._meta.database = self.database
MigrateHistory._meta.db_table = self.migrate_table
MigrateHistory.create_table(True)
return MigrateHistory
Looks like it tries to create the table always because of the last line before return which then fails if I have already migrated something to the db and the table exists. Why is that? Am I using this wrong somehow?
My peewee and peewee_migrate versions:
peewee==2.8.5
peewee-migrate==0.12.3
Using Python 3.5
And full stack trace:
Traceback (most recent call last):
File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/peewee.py", line 3768, in execute_sql
cursor.execute(sql, params or ())
psycopg2.ProgrammingError: relation "migratehistory" already exists
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/tuomas/Code/KarjalaProject/karelian-db/tasks/migrate.py", line 12, in <module>
router.run()
File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/peewee_migrate/router.py", line 163, in run
diff = self.diff
File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/peewee_migrate/router.py", line 57, in diff
done = set(self.done)
File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/peewee_migrate/router.py", line 52, in done
return [mm.name for mm in self.model.select()]
File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/cached_property.py", line 26, in __get__
value = obj.__dict__[self.func.__name__] = self.func(obj)
File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/peewee_migrate/router.py", line 42, in model
MigrateHistory.create_table(True)
File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/peewee.py", line 4975, in create_table
db.create_table(cls)
File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/peewee.py", line 3852, in create_table
return self.execute_sql(*qc.create_table(model_class, safe))
File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/peewee.py", line 3775, in execute_sql
self.commit()
File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/peewee.py", line 3598, in __exit__
reraise(new_type, new_type(*exc_args), traceback)
File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/peewee.py", line 135, in reraise
raise value.with_traceback(tb)
File "/home/tuomas/Code/KarjalaProject/karelian-db/karelian-db_venv/lib/python3.5/site-packages/peewee.py", line 3768, in execute_sql
cursor.execute(sql, params or ())
peewee.ProgrammingError: relation "migratehistory" already exists
When auto-generating a migration the ForeignKeyField rel_model argument appears to be performing an str() on the related model class instead of an name.
Start migrations
invalid syntax (<string>, line 50)
Traceback (most recent call last):
File ".../venv/lib/python2.7/site-packages/peewee_migrate/router.py", line 111, in run_one
migrate, rollback = self.read(name)
File ".../venv/lib/python2.7/site-packages/peewee_migrate/router.py", line 209, in read
exec_in(code, scope)
File "<string>", line 3, in exec_in
File "<string>", line 50
organization = pw.ForeignKeyField(db_column='organization_id', rel_model=<class 'clams.blueprints.organization.models.Organization'>, related_name='organizationuser_set')
^
SyntaxError: invalid syntax
Migration failed: 001_initial
Traceback (most recent call last):
File "manage.py", line 63, in <module>
manager.run()
File "/Users/kolanos/Web/clams/venv/lib/python2.7/site-packages/flask_script/__init__.py", line 412, in run
result = self.handle(sys.argv[0], sys.argv[1:])
File "/Users/kolanos/Web/clams/venv/lib/python2.7/site-packages/flask_script/__init__.py", line 383, in handle
res = handle(*args, **config)
File "/Users/kolanos/Web/clams/venv/lib/python2.7/site-packages/flask_script/commands.py", line 216, in __call__
return self.run(*args, **kwargs)
File "manage.py", line 41, in migrate
migrations = router.run(name, fake=fake)
File "/Users/kolanos/Web/clams/venv/lib/python2.7/site-packages/peewee_migrate/router.py", line 156, in run
self.run_one(mname, migrator, fake=fake, force=fake)
File "/Users/kolanos/Web/clams/venv/lib/python2.7/site-packages/peewee_migrate/router.py", line 111, in run_one
migrate, rollback = self.read(name)
File "/Users/kolanos/Web/clams/venv/lib/python2.7/site-packages/peewee_migrate/router.py", line 209, in read
exec_in(code, scope)
File "<string>", line 3, in exec_in
File "<string>", line 50
organization = pw.ForeignKeyField(db_column='organization_id', rel_model=<class 'models.Organization'>, related_name='organizationuser_set')
^
SyntaxError: invalid syntax
In requirements.txt
:
cached_property == 1.3.0
click == 6.4
mock == 1.3.0
peewee >= 2.8.0
Would it be possible to depend from a minimum only version (as with peewee)? Requiring exact versions is pretty annoying in projects, that happen to want to use a newer version.
Migration doesn't set default value for text field if use add_fields
migrator.add_fields(
'table_name',
text=pw.CharField(default='N', constraints=[SQL('DEFAULT "N"')]))
But if I create a model with same field always is good :-/
@migrator.create_model
class TableName(pw.Model):
text=pw.CharField(default='N', constraints=[SQL('DEFAULT "N"')]))
Where is a mistake?
I am trying to add a not-null field in a migration. The way to do it is to add the fill with null=True
, fill it and then add the not-null constraint.
I came up with the following:
# add the field with null=True
migrator.add_fields(
'mymodel1',
format = pw.CharField(max_length=255, null=True)
)
# initialize the format attribute
migrator.sql("""
UPDATE mymodel1 SET format = CASE WHEN (somerelatedtable.name LIKE '%-FOO') THEN 'FOO' ELSE 'BAR' END
FROM somerelatedtable WHERE mymodel1.somerelatedtable_id = somerelatedtable.id;""")
# add the not-null constraint
migrator.add_not_null('mymodel1', 'format')
which fails with the following message:
Starting migrations
Running "002_pseudo_format"
add_column ('pseudopotential', 'format', <peewee.CharField object at 0x7fe224bb8518>)
format is not null but has no default
Traceback (most recent call last):
File "/usr/lib64/python3.5/site-packages/peewee_migrate/router.py", line 128, in run_one
migrator.run()
File "/usr/lib64/python3.5/site-packages/peewee_migrate/migrator.py", line 108, in run
opn.run()
File "/usr/lib64/python3.5/site-packages/playhouse/migrate.py", line 144, in run
getattr(self.migrator, self.method)(*self.args, **kwargs))
File "/usr/lib64/python3.5/site-packages/playhouse/migrate.py", line 152, in inner
return fn(self, *args, **kwargs)
File "/usr/lib64/python3.5/site-packages/playhouse/migrate.py", line 222, in add_column
raise ValueError('%s is not null but has no default' % column_name)
ValueError: format is not null but has no default
Migration failed: 002_pseudo_format
Traceback (most recent call last):
File "/usr/lib/python-exec/python3.5/pw_migrate", line 9, in <module>
load_entry_point('peewee-migrate==0.5.9', 'console_scripts', 'pw_migrate')()
File "/usr/lib64/python3.5/site-packages/click/core.py", line 716, in __call__
return self.main(*args, **kwargs)
File "/usr/lib64/python3.5/site-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/usr/lib64/python3.5/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib64/python3.5/site-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib64/python3.5/site-packages/click/core.py", line 534, in invoke
return callback(*args, **kwargs)
File "/usr/lib64/python3.5/site-packages/peewee_migrate/cli.py", line 57, in migrate
migrations = router.run(name, fake=fake)
File "/usr/lib64/python3.5/site-packages/peewee_migrate/router.py", line 156, in run
self.run_one(mname, migrator, fake=fake, force=fake)
File "/usr/lib64/python3.5/site-packages/peewee_migrate/router.py", line 128, in run_one
migrator.run()
File "/usr/lib64/python3.5/site-packages/peewee_migrate/migrator.py", line 108, in run
opn.run()
File "/usr/lib64/python3.5/site-packages/playhouse/migrate.py", line 144, in run
getattr(self.migrator, self.method)(*self.args, **kwargs))
File "/usr/lib64/python3.5/site-packages/playhouse/migrate.py", line 152, in inner
return fn(self, *args, **kwargs)
File "/usr/lib64/python3.5/site-packages/playhouse/migrate.py", line 222, in add_column
raise ValueError('%s is not null but has no default' % column_name)
ValueError: format is not null but has no default
Removing the add_not_null
makes it run through, but I should be able to have it completely in one migration, shouldn't I?
It is possible now to emulate a migration run through router.run(fake=True), but can't do it for rollbacks
Traceback (most recent call last):
File "/Users/v/.venv/s/bin/pw_migrate", line 7, in
from peewee_migrate.cli import cli
File "/Users/v/.venv/s/lib/python3.6/site-packages/peewee_migrate/init.py", line 36, in
from .router import Migrator, Router # noqa
File "/Users/v/.venv/s/lib/python3.6/site-packages/peewee_migrate/router.py", line 14, in
from peewee_migrate.migrator import Migrator
File "/Users/v/.venv/s/lib/python3.6/site-packages/peewee_migrate/migrator.py", line 2, in
from playhouse.migrate import (
ImportError: cannot import name 'Clause'
The rename_field method fails with a KeyError on trying to get the old field from model._meta_fields, i.e. field = model._meta.fields[old_name]
. As far as I can see, this will always fail since the model field has been renamed already and the old field name key is gone, or am I missing something?
Version: 1.0.0
Steps to reproduce:
migrator.rename_field(FooModel, 'old_foo', 'new_bar')
Expected: Success
Actual: the following exception:
File "/usr/local/lib/python3.6/site-packages/peewee_migrate/migrator.py", line 260, in rename_column
field = model._meta.fields[old_name]
KeyError: 'old_foo'
So this appears to make things completely unusable as the NotImplementedError of BaseRouter.todo always raises NotImlementedError
router.py#L41-L43
Where are you going with all that? Do you need me to implement something there?
Nick
When using pw_migrate with the --auto
flag to add a new CharField with a default value to a table it produces code like this:
@migrator.create_model
class Color(pw.Model):
id = pw.AutoField()
name = pw.CharField(constraints=[SQL("DEFAULT New Color")], max_length=255)
The last line will lead to an Operational error
because there are quotation marks missing around the default values.
Hello
I am using peewee to run raw MySQL queries. Sometimes, I have to use some multi-lines and they take some space on python code files. May I read an external SQL file? Or a Python file with multiliners?
Thank you
I tried to use the --auto
option to generate the migration files. The result is:
def migrate(migrator, database, fake=False, **kwargs):
"""Write your migrations here."""
@migrator.create_model
class BaseModel(pw.Model):
@migrator.create_model
class BasissetFamily(pw.Model):
name = pw.CharField(max_length=255, unique=True)
@migrator.create_model
class MyModel1(pw.Model):
family = pw.ForeignKeyField(db_column='family_id', rel_model=MyModel1, to_field='id')
element = pw.CharField(max_length=255)
basis = pw.TextField()
@migrator.create_model
class Model(pw.Model):
@migrator.create_model
class MyModel2(pw.Model):
name = pw.CharField(max_length=255, unique=True)
The problem are the empty bodies for the BaseModel
and Model
classes which is invalid Python.
I am trying to add an index as follows:
migrator.add_index('structuresetstructure', 'structure', 'set', unique=True)
I get the following error:
[...]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python-exec/python3.5/pw_migrate", line 9, in <module>
load_entry_point('peewee-migrate==0.5.10', 'console_scripts', 'pw_migrate')()
File "/usr/lib64/python3.5/site-packages/click/core.py", line 716, in __call__
return self.main(*args, **kwargs)
File "/usr/lib64/python3.5/site-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/usr/lib64/python3.5/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib64/python3.5/site-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib64/python3.5/site-packages/click/core.py", line 534, in invoke
return callback(*args, **kwargs)
File "/usr/lib64/python3.5/site-packages/peewee_migrate/cli.py", line 57, in migrate
migrations = router.run(name, fake=fake)
File "/usr/lib64/python3.5/site-packages/peewee_migrate/router.py", line 156, in run
self.run_one(mname, migrator, fake=fake, force=fake)
File "/usr/lib64/python3.5/site-packages/peewee_migrate/router.py", line 128, in run_one
migrator.run()
File "/usr/lib64/python3.5/site-packages/peewee_migrate/migrator.py", line 107, in run
opn.run()
File "/usr/lib64/python3.5/site-packages/playhouse/migrate.py", line 144, in run
getattr(self.migrator, self.method)(*self.args, **kwargs))
File "/usr/lib64/python3.5/site-packages/playhouse/migrate.py", line 133, in _handle_result
self.execute(result)
File "/usr/lib64/python3.5/site-packages/playhouse/migrate.py", line 129, in execute
self.migrator.database.execute_sql(sql, params)
File "/usr/lib64/python3.5/site-packages/playhouse/postgres_ext.py", line 380, in execute_sql
self.commit()
File "/usr/lib64/python3.5/site-packages/peewee.py", line 3316, in __exit__
reraise(new_type, new_type(*exc_args), traceback)
File "/usr/lib64/python3.5/site-packages/peewee.py", line 127, in reraise
raise value.with_traceback(tb)
File "/usr/lib64/python3.5/site-packages/playhouse/postgres_ext.py", line 373, in execute_sql
cursor.execute(sql, params or ())
peewee.ProgrammingError: FEHLER: Relation „structuresetstructure_structure_id_set_id“ existiert bereits
I have another add_index
call for a different table in an earlier (already applied) migration file.
pw_migrate migrate
Add a foreign key to table ,produce the following error
role_id is not null but has no default
001_add_role.py as follows
def migrate(migrator, database, fake=False, **kwargs):
"""Write your migrations here."""
try:
migrator.add_fields(
'host',
role = pw.ForeignKeyField(db_column='role_id', rel_model='role', to_field='id'))
except Exception as e:
print(e)
I have a model with a few fields. In the next version of my program, I want to add another field. I add it to the model class and write a migration with migrator.add_fields
call. Everything is fine if I run the program with an empty database or if I migrate the database from the previous version. But I run migrations on the database of the last version which had no migrations I get the exception:
peewee.ProgrammingError: column of relation already exists
How do I handle this situation? Is there any way to suppress the exception or to check the field existence without executing SQL (I want my migrations to be DBMS independent)?
Lets say there are migrations 001 and 002. 001 succeeds but 002 does not. When you re-run migrations, library tries to re-run 001 even though it's already applied in the database.
In BaseRouter.migrator
there is:
for name in self.done:
self.run_one(name, migrator)
I'm not sure why that is there, what is the logic, but removing the block fixes the issue.
I have peewee version 3.7.1 installed and latest peewee-migrate
when I try to create a migration
pw_migrate create -v --database deploy_services --directory migrations/ --auto-source /path/to/model/new_table.py migration_name
/lib/python2.7/site-packages/playhouse/db_url.py line 97 in connect parsed.scheme)
RuntimeError: Unrecognized or unsupported schema: "".
I get the below error
Happens with peewee-migrate version 1.0.0 (tested against multiple peewee versions from the latest one (3.6.4) backwards, didn't find any where this doesn't occur).
Error:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/peewee_migrate/router.py", line 144, in run_one
migrator.run()
File "/usr/local/lib/python2.7/site-packages/peewee_migrate/migrator.py", line 129, in run
op.run()
File "/usr/local/lib/python2.7/site-packages/playhouse/migrate.py", line 141, in run
self._handle_result(method(*self.args, **kwargs))
File "/usr/local/lib/python2.7/site-packages/playhouse/migrate.py", line 149, in inner
return fn(self, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/peewee_migrate/migrator.py", line 36, in change_column
operations = [self.alter_change_column(table, column_name, field)]
File "/usr/local/lib/python2.7/site-packages/peewee_migrate/migrator.py", line 81, in alter_change_column
field_clause = clause.nodes[-1]
AttributeError: 'Context' object has no attribute 'nodes'
Other details:
OS: Ubuntu 16.04
Setup: docker version 18.06.0-ce; docker-compose version 1.8.0; Python 2.7; postgresql version 9.6
Steps:
class PgTest(Model):
class Meta:
database = psql_client.db
table_function = lambda model_cls: '{model_name}'.format(model_name=model_cls.__name__.lower())
id = AutoField()
name = CharField(max_length=100, index=True)
Create migration, and migrate it
$ pw_migrate create --auto test.models --directory test/migrations --database postgresql://postgres@postgres:5432/postgres test
$ pw_migrate migrate --directory test/migrations --database postgresql://postgres@postgres:5432/datalake --name 001_test
Back to the model, alter max_length
on the name
field, generate new migration
Migrate the new migration, result is the traceback above.
It must start with 3 numbers, like 001_something.py
Invoking create --auto
on the following model:
class Domain(peewee.Model):
name = peewee.CharField()
user = peewee.CharField()
class Meta:
db_table = 'domains'
generates these migrations:
def migrate(migrator, database, fake=False, **kwargs):
"""Write your migrations here."""
@migrator.create_model
class Domain(pw.Model):
name = pw.CharField(max_length=255)
user = pw.CharField(max_length=255)
def rollback(migrator, database, fake=False, **kwargs):
"""Write your rollback migrations here."""
migrator.remove_model('domains')
As expected, this applies normally (creating domain
table despite db_table
meta option) but fails to rollback.
Hello, I currently have a situation where I have multiple models.py files separated into their own domains and folders. I was wondering how I can get pw_migrate
to auto-generate the migrations. I have tried the following with no such luck.
File Structue:
- api
- - migrations
- - auth
- - - auth
- - - - models.py
- - groups
- - - groups
- - - - models.py
I am running the following command:
pw_migrate create --auto -v --database=${DB_URL} "initial"
from the /api/ folder.. and it comes back with this error message:
Can't import models module: True
Does anyone have any suggestions or ideas?
I would be okay having migration folders/files like so:
- api
- - auth
- - - auth
- - - - models.py
- - - migrations
- - groups
- - - groups
- - - - models.py
- - - migrations
However, I would need all the migrations to be able to be run against the same database.
Line https://github.com/klen/peewee_migrate/blob/develop/peewee_migrate/__init__.py#L21 adds a StreamHandler()
to peewee_migrate
logger. IMO it should be up to the user of the library to configure the logging as per platform requirements.
In our case we end up doing del(logging.getLogger('peewee_migrate').handlers[:])
to avoid having output on stderr which doesn't obey the logging configuration.
Proposed solution: remove lines adding handler and setting log level.
File "/Users/anaconda3/lib/python3.6/site-packages/peewee_migrate/router.py", line 277, in _import_submodules
if set(package.path) & passed:
AttributeError: module 'model' has no attribute 'path'
peewee_migrate/peewee_migrate/router.py
Line 73 in 99e7de8
It is better to raise this exception (ImportError) to show the real error, instead of showing a beautiful message but that is hard to debug.
Nice initiative of the library by the say! it is pretty cool!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.