Comments (8)
Also the TimescaleSchemaEditor
extends PostGISSchemaEditor
instead of django.db.backends.postgresql.schema.DatabaseSchemaEditor
.
Should this be fixed?
from django-timescaledb.
I think we can handle "migration to hypertable" properly (autodetected by the migration framework).
- For new table - the current implementation works
- For existing table - there are two cases
- new "time" field added and use it as partition column
- migration autodetect feature - will create
AddField
migration operation - which internally calls
schema_editor.add_field(from_model, field)
- we override this method and call
super.add_field(from_model, field)
- then we check if new field is type
TimeScaleDateTimeField
- if that is true we trigger "migrate_to_hypertabe" - procedure for it defined below
- migration autodetect feature - will create
- use "existing datetime" field as partition column
- migration autdetect feature - will create
AlterField
migration operation - which internally calls
schema_editor.alter_field(from_model, from_field, to_field)
- we override this method and call
schema_editor.alter_field(from_model, from_field, to_field)
- we check if the "type" has changed
- if that is ture, we trigger "migrate_to_hyertable" procedure
- migration autdetect feature - will create
- new "time" field added and use it as partition column
"migrate_to_hypertable" producre
There are two way we can do this:
- [#16]
create new hypertable "like" the old one - insert all rows from old to new; this is what is recommened herethis one is recommended as it does not lock the old table, good for large tableone issue here is - deleting old(non-hyper)tableshould this be done automatically - risking data lose? (if implmented correctly wont happen, but cant say!)or we let it be and tell user to "drop" it when he thinks the migration was perfect
- create hypertable "with" existing table - with
migrate_data => true
option when callingcreate_hypertable
; as described here- this is relatively simpler to implement and safe (as timescaledb takes the responsibility for "migration")
- as mentioned in the docs (check below) - this could lock the table and this could be an issue for large tables
WARNING:The use of the migrate_data argument to convert a non-empty table can lock the table for a significant amount of time, depending on how much data is in the table. It can also run into deadlock if foreign key constraints exist to other tables.
from django-timescaledb.
Am I doing anything wrong?
from django-timescaledb.
@schlunsen let me know your views, so that I can start working on it?
Do we have test framework(I am new to automatic testing)? so that we can check various conditions for migrations ?
from django-timescaledb.
Hi @daadu
Thanks for taking the time to look into this, really nice with some movement on this project! I really appreciate that you have documented and highlighted the issues regarding the migration so well.
Would be awesome to have migrations working, which as you correctly observed isn't the case at the moment.
I think both use cases are valid and it could be nice if the we with relatively ease could change the strategy of the migration. Maybe this could be a configurable setting for the migration?
I lean mostly towards having option 2 that you mention as a default and using migrate_data => true
when creating the hypertable.
from django-timescaledb.
Hi @daadu,
Really cool input on this! Looks like a really positive change. As for testing I would recommend we use pytest and the standard Django test framework and use a GitHub actions to setup as a CI. I agree with @schlunsen that option 2 is the best option, my use cases are only greenfield, does the migrate_data => true have any impact on new tables or is that ignored by timescale? I can't see anything in the docs.
from django-timescaledb.
@bencleary for new table (create_model
call in schema
), the migration passes migrate_data => false
, so does not affect that.
from django-timescaledb.
@bencleary Looks like you have self-assigned #1 , have begin the work for it? where can I check it. would love to add to for migrations.
from django-timescaledb.
Related Issues (20)
- Roadmap Proposal HOT 3
- FR: Add support for time_bucket offset and origin parameters HOT 2
- Failure in `time_bucket()` invocation on TimescaleQuerySet HOT 6
- FR: LTTB feature implementation
- Cannot drop constraint because other objects depend on it
- Getting psycopg2.DatabaseError: cannot create a unique index without the column "time" (used in partitioning) HOT 3
- FR: Support for time_bucket_ng
- Add partitioning_column
- rename variable in sql_add_hypertable string
- Add auto_now_add for time field? HOT 1
- time_bucket_gapfill does not support running without datapoints param. HOT 1
- Custom Admin to plot time-series
- Is it absolutely necessary to name the field "time" ? HOT 2
- Implement interpolate() functionality
- Error with compression and django pytests
- Incorrect function location
- TimeBucket does not allow for extra arguments allowed by timescaledb HOT 1
- Settings not found in python manage.py migrate HOT 1
- pkey name overflow happened if table name length is more than 58 symbols
- primary key / unique behavior and index creation
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from django-timescaledb.