Comments (7)
@theromis I've attempted to optimize the internal iteration, but there's likely room for improvement still!
FWIW you don't need to call order_by('-date')
on each internal QuerySet
if you're calling it on QuerySetSequence
, it automatically recurses into each QuerySet
. (I don't think this will speed up your query at all, however.)
I have noticed it's slower than just feeding everything into chain
, but it's necessary to get the QuerySet
API, e.g. I've used this with DRF pagination. I'll take a look and see if I can profile some of this later!
from django-querysetsequence.
I've actually been unable to reproduce this with the following code:
from itertools import chain
import random
import string
from time import time
from django.db.models.query import Q
from django.test import TestCase
from queryset_sequence import QuerySetSequence
from .models import (Article, Author, BlogPost, Book, OnlinePublisher,
PeriodicalPublisher, Publisher)
class TimeIt(object):
def __enter__(self):
self.time = time()
def __exit__(self, *args):
print(time() - self.time)
class TestPerformance(TestCase):
def test_foo(self):
"""Set-up some data to be tested against."""
alice = Author.objects.create(name="Alice")
bob = Author.objects.create(name="Bob")
mad_magazine = PeriodicalPublisher.objects.create(name="Mad Magazine")
big_books = Publisher.objects.create(name="Big Books",
address="123 Street")
# Generate a bunch of books.
for i in xrange(20000):
N = 8
title = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(N))
author = random.choice([alice, bob])
Article.objects.create(title=title, author=author, publisher=mad_magazine)
for i in xrange(1000):
N = 8
title = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(N))
author = random.choice([alice, bob])
Book.objects.create(title=title, author=author, publisher=big_books, pages=random.randint(100, 100000))
# Time this.
print("CHAIN")
for i in xrange(10):
with TimeIt():
data = sorted(
chain(
Article.objects.filter(Q(title__contains='Q') | Q(author__in=[alice, bob])),
Book.objects.filter(Q(pages__gt=1500))
),
key=lambda instance: instance.title, reverse=True)
print("QUERYSETSEQUENCE")
for i in xrange(10):
with TimeIt():
new_data = QuerySetSequence(
Article.objects.filter(Q(title__contains='Q') | Q(author__in=[alice, bob])),
Book.objects.filter(Q(pages__gt=1500))
).order_by('-title')
This yields results:
CHAIN
0.714089870453
0.870369911194
0.727043867111
0.732348918915
0.761646032333
0.712174892426
0.727970838547
0.755318880081
0.736012935638
0.709211826324
QUERYSETSEQUENCE
0.00103998184204
0.000744104385376
0.000738859176636
0.000655889511108
0.000715970993042
0.000702142715454
0.000705003738403
0.000704050064087
0.000705003738403
0.000693082809448
from django-querysetsequence.
yes, I can't reproduce my production with your modified test:
from itertools import chain
import random
import string
from time import time
from django.db.models.query import Q
from django.test import TestCase
from queryset_sequence import QuerySetSequence
from lonje.account.models import User
from lonje.social.models import Feed, Like
from django.core.paginator import Paginator
class TimeIt(object):
def __enter__(self):
self.time = time()
def __exit__(self, *args):
print(time() - self.time)
class TestPerformance(TestCase):
def setUp(self):
"""Set-up some data to be tested against."""
a = User.objects.create(email='[email protected]', first_name="a", last_name="b")
c = User.objects.create(email='[email protected]', first_name="c", last_name="d")
print("Gen FEEDS")
for i in xrange(20000):
N = 8
title = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(N))
user = random.choice([a, c])
Feed.objects.create(title=title, text=title, user=user)
if i%1000==0:
print i
print("Gen Likes")
for i, feed in enumerate(Feed.objects.all().order_by('?')[:1000]):
user = random.choice([a, c])
feed_list=user.feeds.all()
Like.objects.create(user=user, feed=feed)
if i%100==0:
print i
def test_foo(self):
# Time this.
print("CHAIN")
for i in xrange(10):
with TimeIt():
data = sorted(
chain(
Feed.objects.all(),
Like.objects.all(),
),
key=lambda instance: instance.date, reverse=True)
p = Paginator(data, 20)
print("QUERYSETSEQUENCE")
for i in xrange(10):
with TimeIt():
data = QuerySetSequence(
Feed.objects.all(),
Like.objects.all(),
).order_by('-date')
p = Paginator(data, 20)
got good results for queryset as well:
CHAIN
0.670628070831
0.722599029541
0.714858055115
0.723458051682
0.721124172211
0.729364156723
0.723715782166
0.651701927185
0.699700117111
0.707785129547
QUERYSETSEQUENCE
0.0142028331757
0.000411987304688
0.000228881835938
0.000220775604248
0.000217914581299
0.000216960906982
0.000217914581299
0.000221967697144
0.00021505355835
0.000216007232666
can it be UWSGI related issue?
from django-querysetsequence.
Possibly! I also tried doing more complicate queries with Q
? Could be related to that...
from django-querysetsequence.
ok, seems like I found mistake in queryset.
in my case I never reached any nodes, because Paginator generates only wrapper/iterator.
So after adding p.page(1)
results became real, but they still looks good.
But I'm using different Paginator class in my production code, have to research this.
CHAIN
0.672379970551
0.731985092163
0.721895933151
0.728050947189
0.726105928421
0.756543159485
0.742374897003
0.671122789383
0.713405847549
0.719001054764
QUERYSETSEQUENCE
0.797764778137
0.791668891907
0.790395975113
0.791607141495
0.795854091644
0.85275220871
0.823861122131
0.815613031387
0.8164498806
0.819106817245
PS: I think in your testcase you need to do something similar.
from django-querysetsequence.
Ah! You're right. I'm not evaluating the QuerySet
, adding a new_data = list(new_data)
produces results that are 1.5x - 2.5x slower. I'll check this out! Thanks for reporting.
from django-querysetsequence.
Thank you sir, hope will be not hard to improve processing speed.
from django-querysetsequence.
Related Issues (20)
- Retrieving related values doen't work after order_by() HOT 3
- Need help! How to pass the serializers with the QuerySetSequence? HOT 2
- Slicing issue HOT 6
- DJ1.10: TypeError: not_impl() got an unexpected keyword argument 'using' HOT 4
- NotImplementedError: QuerySequence does not implement get_compiler() HOT 8
- Implement distinct() HOT 8
- 'QuerySetSequence' object has no attribute '_cloning' HOT 3
- Using QuerySets from different databases is not supported HOT 7
- Broken support for DjangoFilterBackend?
- New Version HOT 3
- Support Django 3.0 ( / drop support for Django 1.11 / Python 2.7) HOT 5
- SyntaxWarning: "is not" with a literal. Did you mean "!="? HOT 6
- pip installing this package is still using django.utils import six HOT 1
- 'QuerySetSequence' object has no attribute 'query' HOT 1
- AttributeError: 'QuerySetSequence' object has no attribute '_prefetch_related_lookups' HOT 10
- Support using `models.F` in `QuerySetSequence.order_by()` HOT 1
- Support asynchronous ORM methods
- Support Django 4.1 fields for `bulk_create()`
- Support prefetching in `iterator()`
- Support order_by with None value HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from django-querysetsequence.