clokep / django-querysetsequence Goto Github PK

First of all, thanks about this amazing lib! I've found a little issue where values() doesn't work after calling order_by().

Code to replicate the problem:

class Question(models.Model):
	name = models.CharField(max_length=350)

class Answer(models.Model):
	question = models.ForeignKey(Question, on_delete=models.CASCADE)


class Answer2(models.Model):
	question = models.ForeignKey(Question, on_delete=models.CASCADE)


query = QuerySetSequence(Answer.objects.all(), Answer2.objects.all())

query.values('question__name')  # This works
query.order_by('question__name').values('question__name')  # This does NOT work

The error thrown is the following:

File "/my_path/venv/lib/python3.10/site-packages/queryset_sequence/init.py", line 565, in iter
self._fetch_all()
File "/my_path/venv/lib/python3.10/site-packages/queryset_sequence/init.py", line 542, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/my_path/venv/lib/python3.10/site-packages/queryset_sequence/init.py", line 367, in iter
yield from super().iter()
File "/my_path/venv/lib/python3.10/site-packages/queryset_sequence/init.py", line 210, in _ordered_iterator
iterables = sorted(iterables, key=comparator)
File "/my_path/venv/lib/python3.10/site-packages/queryset_sequence/init.py", line 194, in comparator
return _comparator(tuple_1[2], tuple_2[2])
File "/my_path/venv/lib/python3.10/site-packages/queryset_sequence/init.py", line 143, in comparator
v1 = cls._get_fields(i1, *field_names)
File "/my_path/venv/lib/python3.10/site-packages/queryset_sequence/init.py", line 379, in _get_fields
return itemgetter(*field_names)(obj)
KeyError: 'question.name'

Since the private API changed in 6394e19, this required a fix in dal's form field yourlabs/django-autocomplete-light@7ec8106

For the form field (#2), we need to validate a tuple of ctype id and object id against the QuerySetSequence. The current implementation deconstructs the QuerySetSequence back into querysets to validate the object id against the queryset corresponding to the ctype.

Is there any way to achieve this goal with the public API ?

I'm getting an exception when using QuerySetSequence with django-cacheops:

>>> from models import TestModel
>>> from queryset_sequence import QuerySetSequence
>>> q = QuerySetSequence(TestModel.objects.all())                                                                                                                                                                                                                  
>>> q.all()                                                                                                                                                                                                                                                            
Traceback (most recent call last):                                                                                                                                                                                                                                     
  File "<console>", line 1, in <module>                                                                                                                                                                                                                                
  File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 777, in all                                                                                                                                                                            
    return self._clone()                                                                                                                                                                                                                                               
  File "/usr/local/lib/python3.5/site-packages/cacheops/query.py", line 227, in _clone                                                                                                                                                                                 
    if self._cloning:                                                                                                                                                                                                                                                  
  File "/usr/local/lib/python3.5/site-packages/queryset_sequence/_inheritance.py", line 71, in __getattribute__                                                                                                                                                        
    (name, attr))                                                                                                                                                                                                                                                      
AttributeError: 'QuerySetSequence' object has no attribute '_cloning'                                                                                                                                                                                                  

Cacheops monkeypatches QuerySet and adds some new properties and methods to it: https://github.com/Suor/django-cacheops/blob/master/cacheops/query.py#L542

What I'm trying to achieve is autocomplete field for GenericForeignKey using django-autocomplete-light (https://django-autocomplete-light.readthedocs.io/en/master/gfk.html), but with caching.

I have tried to add '_cloning' and other patched fields to QuerySetSequence's descendant INHERITED_ATTRS, but with no result.

Could you propose a workaround for it?

the low_mark comparison is not obviously an error.
I get this warning running:
python manage.py runserver

/var/venv/lib/python3.8/site-packages/queryset_sequence/init.py:255: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if low_mark is not 0:

For this reason I did that change, maybe, not strictly necessary.

Originally posted by @alfredo-ardito in #58 (comment)

See Django 4.1 release notes and the corresponding documentation and the list of methods.

I just noticed that v0.6 added support for DjangoFilterBackend (see #6), but v0.9 dropped support for setting the model when changing to using only external APIs (see #38).

Hi there,
after installing django-autocomplete-light which uses QuerySequence I've been getting this error: NotImplementedError: QuerySequence does not implement get_compiler() ; I started an issue under django-autocomplete-light originally. Here is the traceback:

(InteractiveConsole)
>>> from queryset_sequence import QuerySetSequence
>>> from .models import *
>>> cities = City.objects.all()
>>> qs = QuerySetSequence(cities)
>>> qs
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "C:\Users\Areopagitics\Development\env-dev\lib\site-packages\django\db\models\query.py", line 226, in __repr__
    data = list(self[:REPR_OUTPUT_SIZE + 1])
  File "C:\Users\Areopagitics\Development\env-dev\lib\site-packages\django\db\models\query.py", line 250, in __iter__
    self._fetch_all()
  File "C:\Users\Areopagitics\Development\env-dev\lib\site-packages\django\db\models\query.py", line 1105, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "C:\Users\Areopagitics\Development\env-dev\lib\site-packages\django\db\models\query.py", line 50, in __iter__
    compiler = queryset.query.get_compiler(using=db)
  File "C:\Users\Areopagitics\Development\env-dev\lib\site-packages\queryset_sequence\_inheritance.py", line 43, in not_impl
    (name, attr))
NotImplementedError: QuerySequence does not implement get_compiler()

See Django 4.1 release notes:

QuerySet.bulk_create() now supports updating fields when a row insertion fails uniqueness constraints. This is supported on MariaDB, MySQL, PostgreSQL, and SQLite 3.24+.

The update_conflicts, update_fields, and unique_fields parameters were added to support updating fields when a row insertion fails on conflict.

Hi

This error tiger when enter to admin module list page
`

`

Hey!

When I'm using QuerySetSequence with debug turned on, Django debug toolbar complains that:

AttributeError at /
class _meta has no attribute 'app_label'

env/local/lib/python2.7/site-packages/debug_toolbar/panels/templates/panel.py in _store_template_info
value.model._meta.app_label, value.model.__name__) 

I've added app_label = 'QuerySetSequenceModel' to _meta class of QuerySetSequenceModel and it works. But maybe there is a better approach.

Hi !

So, I put together a couple of form fields for django-querysetsequence. As I just saw that you implemented .none(), I thought maybe you were going for Model{,Multiple}ChoiceField compatibility, in which case I was wondering if you wanted to maintain this as part of django-querysetsequence, for other users who are non dal v3 users to enjoy too.

As a side note, I also want to say I really like this app ^^ I wonder if apps like django-generic-m2m or django-gm2m could use it too, you might be curious and check it out.

See Django 4.1 release notes:

QuerySet.iterator() now supports prefetching related objects as long as the chunk_size argument is provided. In older versions, no prefetching was done.

Hi,

try to replace construction:

      return sorted(
          chain(
              Feed.objects.filter(Q(user=self) | Q(user__in=self.following_users.all()) | Q(business__in=self.following_business.all())),
              Like.objects.filter(user__in=self.following_users.all()),
              Dislike.objects.filter(user__in=self.following_users.all()),
            ),
          key=lambda instance: instance.date, reverse=True)

by:

      return QuerySetSequence(
          Feed.objects.filter(Q(user=self) | Q(user__in=self.following_users.all()) | Q(business__in=self.following_business.all())).order_by('-date'),
          Like.objects.filter(user__in=self.following_users.all()).order_by('-date'),
          Dislike.objects.filter(user__in=self.following_users.all()).order_by('-date'),
        ).order_by('-date')

Feeds.count: about 22500, Likes.count: 230, Dislikes.count: 70

original paginated result takes 1.7 sec with QuerySetSequence takes 2.9 sec.
I need this wrapper for DRF CursorPagination.

Is there any potential internal optimization?
PS: May be I can build data structure to have one one data type in queryset, in this case I will avoid using extra inmemory transformations?

Hello,
I've found something weird.

objs = Objs.objects.all()
length = len(objs[1:4])
# length equals 3

objs = QuerySetSequence(Objs.objects.all())
length = len(objs[1:4])
# length equals 4

Actually, it looks like QuerySetSequence ignores start index and it always equals 0.

As result, it breaks Django pagination. I found a workaround for that - convert QSS to a list, and pass that list to Paginator class. But maybe it should not be like that?

In my Django project, I have a FeedSerializer that contains two nested serializers (ArticleSerializer and SubjectSerializer). The goal is to use QuerySetSequence in the FeedListApiView in views.py to combine data from multiple models (Article and Ticker) and pass it to the FeedSerializer.

Here's my code:

serializers.py

class FeedSerializer(serializers.Serializer):
    article = ArticleSerializer(many=True, read_only=True)
    subject = SubjectSerializer(many=True, read_only=True)

    class Meta:
        fields = '__all__'

views.py

class FeedListApiView(ModelViewSet):
    """
    Feed list API View.
    """

    serializer_class = FeedSerializer
    pagination_class = FeedPagination

    def get_queryset(self):
        query = QuerySetSequence(
            Article.objects.all(),
            Ticker.objects.all(),
        )

        return query

when installed with pip, in the init file of the package there is still an import from django.utils import six; it is not compatible with django 3.0+

I'm getting this error from Django 3.0
from django.utils import six
ImportError: cannot import name 'six' from 'django.utils'
How should we fix it?

Any idea why ModelChoiceIterator does not work with QSS anymore ? Will look further into this anyway, apparently it broke django-autocomplete-light

self = <django.forms.models.ModelChoiceIterator object at 0x7f801689ced0>

    def __iter__(self):

        if self.field.empty_label is not None:

            yield ("", self.field.empty_label)

        queryset = self.queryset

        # Can't use iterator() when queryset uses prefetch_related()

>       if not queryset._prefetch_related_lookups:

E       AttributeError: 'QuerySetSequence' object has no attribute '_prefetch_related_lookups'

../.tox/base-py37-django21-sqlite/lib/python3.7/site-packages/django/forms/models.py:1137: AttributeError

#26 Doing just QuerySetSequence(Group.objects.all()) in the manage.py shell of dal's test_project works great with Django 1.8 and 1.9:

[dal_env] 04/04 2017 17:40:32 jpic@lue ~/env/src/dal/test_project  (fix_tests)
$ ../.tox/base-py27-django19-sqlite/bin/python manage.py shell
In [1]: from django.contrib.auth.models import Group
In [2]: from queryset_sequence import QuerySetSequence
In [3]: QuerySetSequence(Group.objects.all())
Out[3]: (0.001) SELECT COUNT(*) AS "__count" FROM "auth_group"; args=()
(0.000) SELECT "auth_group"."id", "auth_group"."name" FROM "auth_group" LIMIT 21; args=()
[]

But starting Django 1.10, it doesn't work for me anymore, I really wonder how tests make it to pass ....

[dal_env] 04/04 2017 17:38:30 jpic@lue ~/env/src/dal/test_project  (fix_tests)
$ ../.tox/base-py27-django110-sqlite/bin/python manage.py shell
In [1]: from queryset_sequence import QuerySetSequence

In [2]: from django.contrib.auth.models import Group

In [3]: QuerySetSequence(Group.objects.all())
Out[3]: ---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/home/jpic/env/src/dal/.tox/base-py27-django110-sqlite/lib/python2.7/site-packages/IPython/core/formatters.pyc in __call__(self, obj)
    670                 type_pprinters=self.type_printers,
    671                 deferred_pprinters=self.deferred_printers)
--> 672             printer.pretty(obj)
    673             printer.flush()
    674             return stream.getvalue()

/home/jpic/env/src/dal/.tox/base-py27-django110-sqlite/lib/python2.7/site-packages/IPython/lib/pretty.pyc in pretty(self, obj)
    381                             if callable(meth):
    382                                 return meth(obj, self, cycle)
--> 383             return _default_pprint(obj, self, cycle)
    384         finally:
    385             self.end_group()

/home/jpic/env/src/dal/.tox/base-py27-django110-sqlite/lib/python2.7/site-packages/IPython/lib/pretty.pyc in _default_pprint(obj, p, cycle)
    501     if _safe_getattr(klass, '__repr__', None) not in _baseclass_reprs:
    502         # A user-provided repr. Find newlines and replace them with p.break_()
--> 503         _repr_pprint(obj, p, cycle)
    504         return
    505     p.begin_group(1, '<')

/home/jpic/env/src/dal/.tox/base-py27-django110-sqlite/lib/python2.7/site-packages/IPython/lib/pretty.pyc in _repr_pprint(obj, p, cycle)
    699     """A pprint that just redirects to the normal repr function."""
    700     # Find newlines and replace them with p.break_()
--> 701     output = repr(obj)
    702     for idx,output_line in enumerate(output.splitlines()):
    703         if idx:

/home/jpic/env/src/dal/.tox/base-py27-django110-sqlite/lib/python2.7/site-packages/django/db/models/query.py in __repr__(self)
    225 
    226     def __repr__(self):
--> 227         data = list(self[:REPR_OUTPUT_SIZE + 1])
    228         if len(data) > REPR_OUTPUT_SIZE:
    229             data[-1] = "...(remaining elements truncated)..."

/home/jpic/env/src/dal/.tox/base-py27-django110-sqlite/lib/python2.7/site-packages/django/db/models/query.py in __iter__(self)
    249                - Responsible for turning the rows into model objects.
    250         """
--> 251         self._fetch_all()
    252         return iter(self._result_cache)
    253 

/home/jpic/env/src/dal/.tox/base-py27-django110-sqlite/lib/python2.7/site-packages/django/db/models/query.py in _fetch_all(self)
   1101     def _fetch_all(self):
   1102         if self._result_cache is None:
-> 1103             self._result_cache = list(self._iterable_class(self))
   1104         if self._prefetch_related_lookups and not self._prefetch_done:
   1105             self._prefetch_related_objects()

/home/jpic/env/src/dal/.tox/base-py27-django110-sqlite/lib/python2.7/site-packages/django/db/models/query.py in __iter__(self)
     48         queryset = self.queryset
     49         db = queryset.db
---> 50         compiler = queryset.query.get_compiler(using=db)
     51 
     52         # Execute the query. This will also fill compiler.select, klass_info,

TypeError: not_impl() got an unexpected keyword argument 'using'
In [4]:                          

I try to sort with an annotate field that possible None value. But look like django-querysetsequence doesn't suport it

Reproduce step:

room_options is a JSON field that contains branding 
room = Room.objects.all().annotate(brand=F('room_options__branding'))
ticket = Ticket.objects.all().annotate(brand=F('room_options__branding'))
room_ticket =  QuerySetSequence(room, ticket, model=Room).order_by('brand')
room_ticket[0]

=> TypeError: '>' not supported between instances of 'NoneType' and 'NoneType'

Hello, I'm trying to integrate django-querysetsequence with django-datatable-view (https://github.com/pivotal-energy-solutions/django-datatable-view)

When this component executes a search a NotImplementedError: QuerySetSequence does not implement distinct() error arrises.

So I would like to know if you plan to implement distinct in the future.

By the way, I fix it so when I use querysetsequence it doesn't use distinct, but of course, there are some duplicated results.

Thanks a lot for the help.
Mauro.

The get_queryset() command is very helpful. Can you create a new version for that?

Having 2 models SomeModel and OtherModel, each is in its own database.

aqs = SomeModel.objects.all()
print(aqs.count())  # 12
bqs = OtherModel.objects.all()
print(bqs.count())  # 128
qs = QuerySetSequence(aqs, bqs)
print(aqs.count())
# ProgrammingError: relation "tbl_ets_auctions" does not exist

does it modify original queryset? Please add a note to the examples about different databases (seems to be not supported?)

Describe the bug
I've tried to use QuerySetSequence.order_by() with models.F, but it fails with the following error message

Internal Server Error: /backend/planning/plans/
Traceback (most recent call last):
  File "/Users/tui/Library/Caches/pypoetry/virtualenvs/fwlap-backend-vuErbnoo-py3.9/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/Users/tui/Library/Caches/pypoetry/virtualenvs/fwlap-backend-vuErbnoo-py3.9/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/Users/tui/Library/Caches/pypoetry/virtualenvs/fwlap-backend-vuErbnoo-py3.9/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/Users/tui/Library/Caches/pypoetry/virtualenvs/fwlap-backend-vuErbnoo-py3.9/lib/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/Users/tui/Library/Caches/pypoetry/virtualenvs/fwlap-backend-vuErbnoo-py3.9/lib/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/Users/tui/Library/Caches/pypoetry/virtualenvs/fwlap-backend-vuErbnoo-py3.9/lib/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/Users/tui/Library/Caches/pypoetry/virtualenvs/fwlap-backend-vuErbnoo-py3.9/lib/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/Users/tui/Library/Caches/pypoetry/virtualenvs/fwlap-backend-vuErbnoo-py3.9/lib/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/Users/tui/Desktop/develop/work/pttep/projects/fwlap/fwlap-backend/planning/views/combined_plan_view.py", line 47, in list
    response_objects = plan_list_service.list(
  File "/Users/tui/Desktop/develop/work/pttep/projects/fwlap/fwlap-backend/planning/services/combined_plan_list_service.py", line 22, in list
    combined_plan_queryset = CombinedPlanQuerySet(
  File "/Users/tui/Desktop/develop/work/pttep/projects/fwlap/fwlap-backend/planning/models/plan.py", line 41, in order_by_date_range
    return self.order_by(
  File "/Users/tui/Library/Caches/pypoetry/virtualenvs/fwlap-backend-vuErbnoo-py3.9/lib/python3.9/site-packages/queryset_sequence/__init__.py", line 780, in order_by
    _, filtered_fields = self._separate_fields(*fields)
  File "/Users/tui/Library/Caches/pypoetry/virtualenvs/fwlap-backend-vuErbnoo-py3.9/lib/python3.9/site-packages/queryset_sequence/__init__.py", line 645, in _separate_fields
    if field.startswith("#") or field.startswith("-#"):
AttributeError: 'OrderBy' object has no attribute 'startswith'

Versions
django: 4.0
django-querysetsequence: 0.16

To Reproduce
Given two models with a common field.

from django.db import models

class ModelA(models.Model):
    date = models.DateField(null=True)

class ModelB(models.Model):
    date = models.DateField(null=True)

Then calling order_by on QuerySetSequence with models.F will result in error

combined_queryset = QuerySetSequence(
    ModelA.objects.all(),
    ModelB.objects.all(),
).order_by(
    models.F("date").asc(nulls_first=True)
)
print(list(combined_queryset))

Expected behavior
QuerySetSequence.order_by() should work with models.F since it's a valid Django code.

I think calling QuerySetSequence(...).iterator() doesn't perform the nice Django optimizations. It seems to just end up in the same iteration code as normal.

Pretty much, we always seem to call iter() or directly iterate through the QuerySets. This shouldn't be too difficult to implement, it involves touching QuerySetSequence(...).iterator(); QuerySeqeunce.__iter__() and QuerySequence._ordered_iterator.

The docs are long enough now that we should probably move the documentation to an actual docs directory using Sphinx.