clokep / django-querysetsequence Goto Github PK
View Code? Open in Web Editor NEWChain multiple (disparate) QuerySets in Django
Home Page: https://django-querysetsequence.readthedocs.io/
License: ISC License
Chain multiple (disparate) QuerySets in Django
Home Page: https://django-querysetsequence.readthedocs.io/
License: ISC License
I'm getting this error from Django 3.0
from django.utils import six
ImportError: cannot import name 'six' from 'django.utils'
How should we fix it?
the low_mark
comparison is not obviously an error.
I get this warning running:
python manage.py runserver
/var/venv/lib/python3.8/site-packages/queryset_sequence/init.py:255: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if low_mark is not 0:
For this reason I did that change, maybe, not strictly necessary.
Originally posted by @alfredo-ardito in #58 (comment)
QuerySet.bulk_create()
now supports updating fields when a row insertion fails uniqueness constraints. This is supported on MariaDB, MySQL, PostgreSQL, and SQLite 3.24+.
The
update_conflicts
,update_fields
, andunique_fields
parameters were added to support updating fields when a row insertion fails on conflict.
Since the private API changed in 6394e19, this required a fix in dal's form field yourlabs/django-autocomplete-light@7ec8106
For the form field (#2), we need to validate a tuple of ctype id and object id against the QuerySetSequence. The current implementation deconstructs the QuerySetSequence back into querysets to validate the object id against the queryset corresponding to the ctype.
Is there any way to achieve this goal with the public API ?
Hi,
try to replace construction:
return sorted(
chain(
Feed.objects.filter(Q(user=self) | Q(user__in=self.following_users.all()) | Q(business__in=self.following_business.all())),
Like.objects.filter(user__in=self.following_users.all()),
Dislike.objects.filter(user__in=self.following_users.all()),
),
key=lambda instance: instance.date, reverse=True)
by:
return QuerySetSequence(
Feed.objects.filter(Q(user=self) | Q(user__in=self.following_users.all()) | Q(business__in=self.following_business.all())).order_by('-date'),
Like.objects.filter(user__in=self.following_users.all()).order_by('-date'),
Dislike.objects.filter(user__in=self.following_users.all()).order_by('-date'),
).order_by('-date')
Feeds.count: about 22500, Likes.count: 230, Dislikes.count: 70
original paginated result takes 1.7 sec with QuerySetSequence
takes 2.9 sec.
I need this wrapper for DRF CursorPagination.
Is there any potential internal optimization?
PS: May be I can build data structure to have one one data type in queryset, in this case I will avoid using extra inmemory transformations?
Hi
This error tiger when enter to admin module list page
`
Request Method: | GET |
---|---|
http://127.0.0.1:8000/admin/shop/product/?e=1 | |
3.0.5 | |
AttributeError | |
'QuerySetSequence' object has no attribute 'query' |
`
Describe the bug
I've tried to use QuerySetSequence.order_by()
with models.F
, but it fails with the following error message
Internal Server Error: /backend/planning/plans/
Traceback (most recent call last):
File "/Users/tui/Library/Caches/pypoetry/virtualenvs/fwlap-backend-vuErbnoo-py3.9/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/Users/tui/Library/Caches/pypoetry/virtualenvs/fwlap-backend-vuErbnoo-py3.9/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/Users/tui/Library/Caches/pypoetry/virtualenvs/fwlap-backend-vuErbnoo-py3.9/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/Users/tui/Library/Caches/pypoetry/virtualenvs/fwlap-backend-vuErbnoo-py3.9/lib/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
return self.dispatch(request, *args, **kwargs)
File "/Users/tui/Library/Caches/pypoetry/virtualenvs/fwlap-backend-vuErbnoo-py3.9/lib/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
response = self.handle_exception(exc)
File "/Users/tui/Library/Caches/pypoetry/virtualenvs/fwlap-backend-vuErbnoo-py3.9/lib/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception
self.raise_uncaught_exception(exc)
File "/Users/tui/Library/Caches/pypoetry/virtualenvs/fwlap-backend-vuErbnoo-py3.9/lib/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
raise exc
File "/Users/tui/Library/Caches/pypoetry/virtualenvs/fwlap-backend-vuErbnoo-py3.9/lib/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
File "/Users/tui/Desktop/develop/work/pttep/projects/fwlap/fwlap-backend/planning/views/combined_plan_view.py", line 47, in list
response_objects = plan_list_service.list(
File "/Users/tui/Desktop/develop/work/pttep/projects/fwlap/fwlap-backend/planning/services/combined_plan_list_service.py", line 22, in list
combined_plan_queryset = CombinedPlanQuerySet(
File "/Users/tui/Desktop/develop/work/pttep/projects/fwlap/fwlap-backend/planning/models/plan.py", line 41, in order_by_date_range
return self.order_by(
File "/Users/tui/Library/Caches/pypoetry/virtualenvs/fwlap-backend-vuErbnoo-py3.9/lib/python3.9/site-packages/queryset_sequence/__init__.py", line 780, in order_by
_, filtered_fields = self._separate_fields(*fields)
File "/Users/tui/Library/Caches/pypoetry/virtualenvs/fwlap-backend-vuErbnoo-py3.9/lib/python3.9/site-packages/queryset_sequence/__init__.py", line 645, in _separate_fields
if field.startswith("#") or field.startswith("-#"):
AttributeError: 'OrderBy' object has no attribute 'startswith'
Versions
django: 4.0
django-querysetsequence: 0.16
To Reproduce
Given two models with a common field.
from django.db import models
class ModelA(models.Model):
date = models.DateField(null=True)
class ModelB(models.Model):
date = models.DateField(null=True)
Then calling order_by
on QuerySetSequence
with models.F
will result in error
combined_queryset = QuerySetSequence(
ModelA.objects.all(),
ModelB.objects.all(),
).order_by(
models.F("date").asc(nulls_first=True)
)
print(list(combined_queryset))
Expected behavior
QuerySetSequence.order_by()
should work with models.F
since it's a valid Django code.
QuerySet.iterator()
now supports prefetching related objects as long as thechunk_size
argument is provided. In older versions, no prefetching was done.
#26 Doing just QuerySetSequence(Group.objects.all())
in the manage.py shell of dal's test_project works great with Django 1.8 and 1.9:
[dal_env] 04/04 2017 17:40:32 jpic@lue ~/env/src/dal/test_project (fix_tests)
$ ../.tox/base-py27-django19-sqlite/bin/python manage.py shell
In [1]: from django.contrib.auth.models import Group
In [2]: from queryset_sequence import QuerySetSequence
In [3]: QuerySetSequence(Group.objects.all())
Out[3]: (0.001) SELECT COUNT(*) AS "__count" FROM "auth_group"; args=()
(0.000) SELECT "auth_group"."id", "auth_group"."name" FROM "auth_group" LIMIT 21; args=()
[]
But starting Django 1.10, it doesn't work for me anymore, I really wonder how tests make it to pass ....
[dal_env] 04/04 2017 17:38:30 jpic@lue ~/env/src/dal/test_project (fix_tests)
$ ../.tox/base-py27-django110-sqlite/bin/python manage.py shell
In [1]: from queryset_sequence import QuerySetSequence
In [2]: from django.contrib.auth.models import Group
In [3]: QuerySetSequence(Group.objects.all())
Out[3]: ---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
/home/jpic/env/src/dal/.tox/base-py27-django110-sqlite/lib/python2.7/site-packages/IPython/core/formatters.pyc in __call__(self, obj)
670 type_pprinters=self.type_printers,
671 deferred_pprinters=self.deferred_printers)
--> 672 printer.pretty(obj)
673 printer.flush()
674 return stream.getvalue()
/home/jpic/env/src/dal/.tox/base-py27-django110-sqlite/lib/python2.7/site-packages/IPython/lib/pretty.pyc in pretty(self, obj)
381 if callable(meth):
382 return meth(obj, self, cycle)
--> 383 return _default_pprint(obj, self, cycle)
384 finally:
385 self.end_group()
/home/jpic/env/src/dal/.tox/base-py27-django110-sqlite/lib/python2.7/site-packages/IPython/lib/pretty.pyc in _default_pprint(obj, p, cycle)
501 if _safe_getattr(klass, '__repr__', None) not in _baseclass_reprs:
502 # A user-provided repr. Find newlines and replace them with p.break_()
--> 503 _repr_pprint(obj, p, cycle)
504 return
505 p.begin_group(1, '<')
/home/jpic/env/src/dal/.tox/base-py27-django110-sqlite/lib/python2.7/site-packages/IPython/lib/pretty.pyc in _repr_pprint(obj, p, cycle)
699 """A pprint that just redirects to the normal repr function."""
700 # Find newlines and replace them with p.break_()
--> 701 output = repr(obj)
702 for idx,output_line in enumerate(output.splitlines()):
703 if idx:
/home/jpic/env/src/dal/.tox/base-py27-django110-sqlite/lib/python2.7/site-packages/django/db/models/query.py in __repr__(self)
225
226 def __repr__(self):
--> 227 data = list(self[:REPR_OUTPUT_SIZE + 1])
228 if len(data) > REPR_OUTPUT_SIZE:
229 data[-1] = "...(remaining elements truncated)..."
/home/jpic/env/src/dal/.tox/base-py27-django110-sqlite/lib/python2.7/site-packages/django/db/models/query.py in __iter__(self)
249 - Responsible for turning the rows into model objects.
250 """
--> 251 self._fetch_all()
252 return iter(self._result_cache)
253
/home/jpic/env/src/dal/.tox/base-py27-django110-sqlite/lib/python2.7/site-packages/django/db/models/query.py in _fetch_all(self)
1101 def _fetch_all(self):
1102 if self._result_cache is None:
-> 1103 self._result_cache = list(self._iterable_class(self))
1104 if self._prefetch_related_lookups and not self._prefetch_done:
1105 self._prefetch_related_objects()
/home/jpic/env/src/dal/.tox/base-py27-django110-sqlite/lib/python2.7/site-packages/django/db/models/query.py in __iter__(self)
48 queryset = self.queryset
49 db = queryset.db
---> 50 compiler = queryset.query.get_compiler(using=db)
51
52 # Execute the query. This will also fill compiler.select, klass_info,
TypeError: not_impl() got an unexpected keyword argument 'using'
In [4]:
when installed with pip, in the init file of the package there is still an import from django.utils import six; it is not compatible with django 3.0+
Hi !
So, I put together a couple of form fields for django-querysetsequence. As I just saw that you implemented .none(), I thought maybe you were going for Model{,Multiple}ChoiceField compatibility, in which case I was wondering if you wanted to maintain this as part of django-querysetsequence, for other users who are non dal v3 users to enjoy too.
As a side note, I also want to say I really like this app ^^ I wonder if apps like django-generic-m2m or django-gm2m could use it too, you might be curious and check it out.
I'm getting an exception when using QuerySetSequence with django-cacheops:
>>> from models import TestModel
>>> from queryset_sequence import QuerySetSequence
>>> q = QuerySetSequence(TestModel.objects.all())
>>> q.all()
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 777, in all
return self._clone()
File "/usr/local/lib/python3.5/site-packages/cacheops/query.py", line 227, in _clone
if self._cloning:
File "/usr/local/lib/python3.5/site-packages/queryset_sequence/_inheritance.py", line 71, in __getattribute__
(name, attr))
AttributeError: 'QuerySetSequence' object has no attribute '_cloning'
Cacheops monkeypatches QuerySet and adds some new properties and methods to it: https://github.com/Suor/django-cacheops/blob/master/cacheops/query.py#L542
What I'm trying to achieve is autocomplete field for GenericForeignKey using django-autocomplete-light (https://django-autocomplete-light.readthedocs.io/en/master/gfk.html), but with caching.
I have tried to add '_cloning' and other patched fields to QuerySetSequence's descendant INHERITED_ATTRS, but with no result.
Could you propose a workaround for it?
Having 2 models SomeModel
and OtherModel
, each is in its own database.
aqs = SomeModel.objects.all()
print(aqs.count()) # 12
bqs = OtherModel.objects.all()
print(bqs.count()) # 128
qs = QuerySetSequence(aqs, bqs)
print(aqs.count())
# ProgrammingError: relation "tbl_ets_auctions" does not exist
does it modify original queryset? Please add a note to the examples about different databases (seems to be not supported?)
Hello, I'm trying to integrate django-querysetsequence with django-datatable-view (https://github.com/pivotal-energy-solutions/django-datatable-view)
When this component executes a search a NotImplementedError: QuerySetSequence does not implement distinct() error arrises.
So I would like to know if you plan to implement distinct in the future.
By the way, I fix it so when I use querysetsequence it doesn't use distinct, but of course, there are some duplicated results.
Thanks a lot for the help.
Mauro.
Hey!
When I'm using QuerySetSequence with debug turned on, Django debug toolbar complains that:
AttributeError at /
class _meta has no attribute 'app_label'
env/local/lib/python2.7/site-packages/debug_toolbar/panels/templates/panel.py in _store_template_info
value.model._meta.app_label, value.model.__name__)
I've added app_label = 'QuerySetSequenceModel'
to _meta class of QuerySetSequenceModel and it works. But maybe there is a better approach.
First of all, thanks about this amazing lib! I've found a little issue where values()
doesn't work after calling order_by()
.
Code to replicate the problem:
class Question(models.Model):
name = models.CharField(max_length=350)
class Answer(models.Model):
question = models.ForeignKey(Question, on_delete=models.CASCADE)
class Answer2(models.Model):
question = models.ForeignKey(Question, on_delete=models.CASCADE)
query = QuerySetSequence(Answer.objects.all(), Answer2.objects.all())
query.values('question__name') # This works
query.order_by('question__name').values('question__name') # This does NOT work
The error thrown is the following:
File "/my_path/venv/lib/python3.10/site-packages/queryset_sequence/init.py", line 565, in iter
self._fetch_all()
File "/my_path/venv/lib/python3.10/site-packages/queryset_sequence/init.py", line 542, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/my_path/venv/lib/python3.10/site-packages/queryset_sequence/init.py", line 367, in iter
yield from super().iter()
File "/my_path/venv/lib/python3.10/site-packages/queryset_sequence/init.py", line 210, in _ordered_iterator
iterables = sorted(iterables, key=comparator)
File "/my_path/venv/lib/python3.10/site-packages/queryset_sequence/init.py", line 194, in comparator
return _comparator(tuple_1[2], tuple_2[2])
File "/my_path/venv/lib/python3.10/site-packages/queryset_sequence/init.py", line 143, in comparator
v1 = cls._get_fields(i1, *field_names)
File "/my_path/venv/lib/python3.10/site-packages/queryset_sequence/init.py", line 379, in _get_fields
return itemgetter(*field_names)(obj)
KeyError: 'question.name'
I think calling QuerySetSequence(...).iterator()
doesn't perform the nice Django optimizations. It seems to just end up in the same iteration code as normal.
Pretty much, we always seem to call iter()
or directly iterate through the QuerySets
. This shouldn't be too difficult to implement, it involves touching QuerySetSequence(...).iterator()
; QuerySeqeunce.__iter__()
and QuerySequence._ordered_iterator
.
In my Django project, I have a FeedSerializer
that contains two nested serializers (ArticleSerializer and SubjectSerializer)
. The goal is to use QuerySetSequence
in the FeedListApiView
in views.py to combine data from multiple models (Article and Ticker)
and pass it to the FeedSerializer
.
Here's my code:
serializers.py
class FeedSerializer(serializers.Serializer):
article = ArticleSerializer(many=True, read_only=True)
subject = SubjectSerializer(many=True, read_only=True)
class Meta:
fields = '__all__'
views.py
class FeedListApiView(ModelViewSet):
"""
Feed list API View.
"""
serializer_class = FeedSerializer
pagination_class = FeedPagination
def get_queryset(self):
query = QuerySetSequence(
Article.objects.all(),
Ticker.objects.all(),
)
return query
The docs are long enough now that we should probably move the documentation to an actual docs
directory using Sphinx.
Any idea why ModelChoiceIterator does not work with QSS anymore ? Will look further into this anyway, apparently it broke django-autocomplete-light
self = <django.forms.models.ModelChoiceIterator object at 0x7f801689ced0>
def __iter__(self):
if self.field.empty_label is not None:
yield ("", self.field.empty_label)
queryset = self.queryset
# Can't use iterator() when queryset uses prefetch_related()
> if not queryset._prefetch_related_lookups:
E AttributeError: 'QuerySetSequence' object has no attribute '_prefetch_related_lookups'
../.tox/base-py37-django21-sqlite/lib/python3.7/site-packages/django/forms/models.py:1137: AttributeError
See Django 4.1 release notes and the corresponding documentation and the list of methods.
Hello,
I've found something weird.
objs = Objs.objects.all()
length = len(objs[1:4])
# length equals 3
objs = QuerySetSequence(Objs.objects.all())
length = len(objs[1:4])
# length equals 4
Actually, it looks like QuerySetSequence ignores start index and it always equals 0.
As result, it breaks Django pagination. I found a workaround for that - convert QSS to a list, and pass that list to Paginator class. But maybe it should not be like that?
The get_queryset() command is very helpful. Can you create a new version for that?
I try to sort with an annotate field that possible None value. But look like django-querysetsequence doesn't suport it
Reproduce step:
room_options is a JSON field that contains branding
room = Room.objects.all().annotate(brand=F('room_options__branding'))
ticket = Ticket.objects.all().annotate(brand=F('room_options__branding'))
room_ticket = QuerySetSequence(room, ticket, model=Room).order_by('brand')
room_ticket[0]
=> TypeError: '>' not supported between instances of 'NoneType' and 'NoneType'
Hi there,
after installing django-autocomplete-light which uses QuerySequence I've been getting this error: NotImplementedError: QuerySequence does not implement get_compiler()
; I started an issue under django-autocomplete-light originally. Here is the traceback:
(InteractiveConsole)
>>> from queryset_sequence import QuerySetSequence
>>> from .models import *
>>> cities = City.objects.all()
>>> qs = QuerySetSequence(cities)
>>> qs
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "C:\Users\Areopagitics\Development\env-dev\lib\site-packages\django\db\models\query.py", line 226, in __repr__
data = list(self[:REPR_OUTPUT_SIZE + 1])
File "C:\Users\Areopagitics\Development\env-dev\lib\site-packages\django\db\models\query.py", line 250, in __iter__
self._fetch_all()
File "C:\Users\Areopagitics\Development\env-dev\lib\site-packages\django\db\models\query.py", line 1105, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "C:\Users\Areopagitics\Development\env-dev\lib\site-packages\django\db\models\query.py", line 50, in __iter__
compiler = queryset.query.get_compiler(using=db)
File "C:\Users\Areopagitics\Development\env-dev\lib\site-packages\queryset_sequence\_inheritance.py", line 43, in not_impl
(name, attr))
NotImplementedError: QuerySequence does not implement get_compiler()
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.