Giter VIP home page Giter VIP logo

django-postgres-tracked-model's Introduction

Django Postgres Tracked Model

This is a Django implementation of Cognite's txid-syncing. It's an approach to reliabliy sync data between Postgres and other systems, for example ElasticSearch. To keep track of changes a version table is added that tracks which transaction last updated each row in a tracked table.

Usage

Caution

This library is very much work-in-progress. I'd recommend not using it in anything production critical

  1. First you need to install the application in your Django settings file
    INSTALLED_APPLICATIONS = [
        ...,
        "tracked_model",
    ]
  2. Next you add the decorator to the model(s) you want to sync/track
    from django.db import models
    from tracked_model import tracked
    
    @tracked()
    class MyModel(models.Model):
        ...
  3. Create new migrations. This will create a new model with metadata about the syncing
    ./manage.py makemigrations
  4. Create a new migration and add triggers and backfill data
    ./manage.py makemigrations --empty --name add_tracking my_app
    Edit the migration and add operations to track changes and backfill for existing rows
    from tracked_model.operations import AddVersionTracking, BackfillModelVersion
    
    class Migration(migrations.Migration):
        operations = [
            # Note: You'll probably want to do this in two separate migrations
            AddVersionTracking(tracked_model="MyModel", version_model="MyModelVersion"),
            BackfillModelVersion(tracked_model="MyModel"),
        ]

Now you can start streaming changes to your models:

import time
from tracked_model import get_changed_objects

qs = MyModel.objects.all()
changes, cursor = get_changed_objects(cursor=None, limit=10, queryset=qs)
while True:
    if not changes:
        time.sleep(5)
    else:
        print(changes)

    changes, cursor = get_changed_objects(cursor=cursor, limit=10, queryset=qs)

You can send in any queryset you want. The changes return value will be a list of objects returned from the queryset. You can send in any kind of queryset, e.g. using .values(), depending on what you want to have out.

django-postgres-tracked-model's People

Contributors

ljodal avatar

Stargazers

Pål Karlsrud avatar

Watchers

Alex Brasetvik avatar  avatar

django-postgres-tracked-model's Issues

Add migration auto-detection

This requires monkey patching the migration detector. django-pgtriggers has nice inspiration for how to do this

Track related model changes

Add a good interface for tracking changes to related models, for example if the changes API endpoint contains joined fields that we also want to ensure are up to date. One possible interface could be something like this:

from tracked_model import tracked

@tracked()
class Author(models.Model):
    name = models.TextField()

@tracked(parent=Author)
class Book(models.Model):
    title = models.TextField()
    author = models.ForeignKey(Author, on_delete=models.CASCADE)

where we update the Author's version rather than create a separate version model for the Book model. This should work for "simpler" cases, but I'm unsure if it'll work for more complex cases.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.