Giter VIP home page Giter VIP logo

Comments (7)

clokep avatar clokep commented on July 17, 2024

@theromis I've attempted to optimize the internal iteration, but there's likely room for improvement still!

FWIW you don't need to call order_by('-date') on each internal QuerySet if you're calling it on QuerySetSequence, it automatically recurses into each QuerySet. (I don't think this will speed up your query at all, however.)

I have noticed it's slower than just feeding everything into chain, but it's necessary to get the QuerySet API, e.g. I've used this with DRF pagination. I'll take a look and see if I can profile some of this later!

from django-querysetsequence.

clokep avatar clokep commented on July 17, 2024

I've actually been unable to reproduce this with the following code:

from itertools import chain
import random
import string
from time import time

from django.db.models.query import Q
from django.test import TestCase

from queryset_sequence import QuerySetSequence

from .models import (Article, Author, BlogPost, Book, OnlinePublisher,
                     PeriodicalPublisher, Publisher)


class TimeIt(object):
    def __enter__(self):
        self.time = time()

    def __exit__(self, *args):
        print(time() - self.time)


class TestPerformance(TestCase):
    def test_foo(self):
        """Set-up some data to be tested against."""
        alice = Author.objects.create(name="Alice")
        bob = Author.objects.create(name="Bob")

        mad_magazine = PeriodicalPublisher.objects.create(name="Mad Magazine")
        big_books = Publisher.objects.create(name="Big Books",
                                             address="123 Street")

        # Generate a bunch of books.
        for i in xrange(20000):
            N = 8
            title = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(N))
            author = random.choice([alice, bob])

            Article.objects.create(title=title, author=author, publisher=mad_magazine)
        for i in xrange(1000):
            N = 8
            title = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(N))
            author = random.choice([alice, bob])

            Book.objects.create(title=title, author=author, publisher=big_books, pages=random.randint(100, 100000))

        # Time this.
        print("CHAIN")
        for i in xrange(10):
            with TimeIt():
                data = sorted(
                    chain(
                        Article.objects.filter(Q(title__contains='Q') | Q(author__in=[alice, bob])),
                        Book.objects.filter(Q(pages__gt=1500))
                    ),
                    key=lambda instance: instance.title, reverse=True)

        print("QUERYSETSEQUENCE")
        for i in xrange(10):
            with TimeIt():
                new_data = QuerySetSequence(
                    Article.objects.filter(Q(title__contains='Q') | Q(author__in=[alice, bob])),
                    Book.objects.filter(Q(pages__gt=1500))
                ).order_by('-title')

This yields results:

CHAIN
0.714089870453
0.870369911194
0.727043867111
0.732348918915
0.761646032333
0.712174892426
0.727970838547
0.755318880081
0.736012935638
0.709211826324

QUERYSETSEQUENCE
0.00103998184204
0.000744104385376
0.000738859176636
0.000655889511108
0.000715970993042
0.000702142715454
0.000705003738403
0.000704050064087
0.000705003738403
0.000693082809448

from django-querysetsequence.

theromis avatar theromis commented on July 17, 2024

yes, I can't reproduce my production with your modified test:

from itertools import chain
import random
import string
from time import time

from django.db.models.query import Q
from django.test import TestCase

from queryset_sequence import QuerySetSequence
from lonje.account.models import User
from lonje.social.models import Feed, Like
from django.core.paginator import Paginator

class TimeIt(object):
    def __enter__(self):
        self.time = time()

    def __exit__(self, *args):
        print(time() - self.time)


class TestPerformance(TestCase):
    def setUp(self):
        """Set-up some data to be tested against."""
        a = User.objects.create(email='[email protected]', first_name="a", last_name="b")
        c = User.objects.create(email='[email protected]', first_name="c", last_name="d")

        print("Gen FEEDS")
        for i in xrange(20000):
            N = 8
            title = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(N))
            user = random.choice([a, c])

            Feed.objects.create(title=title, text=title, user=user)
            if i%1000==0:
                print i

        print("Gen Likes")
        for i, feed in enumerate(Feed.objects.all().order_by('?')[:1000]):
            user = random.choice([a, c])

            feed_list=user.feeds.all()
            Like.objects.create(user=user, feed=feed)
            if i%100==0:
                print i

    def test_foo(self):
        # Time this.
        print("CHAIN")
        for i in xrange(10):
            with TimeIt():
                data = sorted(
                    chain(
                      Feed.objects.all(),
                      Like.objects.all(),
                    ),
                    key=lambda instance: instance.date, reverse=True)
                p = Paginator(data, 20)

        print("QUERYSETSEQUENCE")
        for i in xrange(10):
            with TimeIt():
                data = QuerySetSequence(
                      Feed.objects.all(),
                      Like.objects.all(),
                ).order_by('-date')
                p = Paginator(data, 20)

got good results for queryset as well:

CHAIN
0.670628070831
0.722599029541
0.714858055115
0.723458051682
0.721124172211
0.729364156723
0.723715782166
0.651701927185
0.699700117111
0.707785129547
QUERYSETSEQUENCE
0.0142028331757
0.000411987304688
0.000228881835938
0.000220775604248
0.000217914581299
0.000216960906982
0.000217914581299
0.000221967697144
0.00021505355835
0.000216007232666

can it be UWSGI related issue?

from django-querysetsequence.

clokep avatar clokep commented on July 17, 2024

Possibly! I also tried doing more complicate queries with Q? Could be related to that...

from django-querysetsequence.

theromis avatar theromis commented on July 17, 2024

ok, seems like I found mistake in queryset.
in my case I never reached any nodes, because Paginator generates only wrapper/iterator.
So after adding p.page(1) results became real, but they still looks good.
But I'm using different Paginator class in my production code, have to research this.

CHAIN
0.672379970551
0.731985092163
0.721895933151
0.728050947189
0.726105928421
0.756543159485
0.742374897003
0.671122789383
0.713405847549
0.719001054764
QUERYSETSEQUENCE
0.797764778137
0.791668891907
0.790395975113
0.791607141495
0.795854091644
0.85275220871
0.823861122131
0.815613031387
0.8164498806
0.819106817245

PS: I think in your testcase you need to do something similar.

from django-querysetsequence.

clokep avatar clokep commented on July 17, 2024

Ah! You're right. I'm not evaluating the QuerySet, adding a new_data = list(new_data) produces results that are 1.5x - 2.5x slower. I'll check this out! Thanks for reporting.

from django-querysetsequence.

theromis avatar theromis commented on July 17, 2024

Thank you sir, hope will be not hard to improve processing speed.

from django-querysetsequence.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.