django-nonrel / django-dbindexer Goto Github PK

Taken from bitbucket: https://bitbucket.org/Wilfred/django-dbindexer/issue/6/crash-on-nullable-values

Here's a minimal test case:

models.py:

from django.db import models

class Test(models.Model):
    display_name = models.CharField(max_length=400, null=True)

index:

register_index(Test, {'display_name': 'istartswith'})

traceback:

$ python2.5 manage.py shell
>>> from artproject_artwork.models import Test
>>> Test.objects.create()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/wilfred/work/gap2/django/db/models/manager.py", line 138, in create
    return self.get_query_set().create(**kwargs)
  File "/home/wilfred/work/gap2/django/db/models/query.py", line 362, in create
    obj.save(force_insert=True, using=self.db)
  File "/home/wilfred/work/gap2/django/db/models/base.py", line 462, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)
  File "/home/wilfred/work/gap2/django/db/models/base.py", line 573, in save_base
    result = manager._insert(values, return_id=update_pk, using=using)
  File "/home/wilfred/work/gap2/django/db/models/manager.py", line 195, in _insert
    return insert_query(self.model, values, **kwargs)
  File "/home/wilfred/work/gap2/django/db/models/query.py", line 1438, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/home/wilfred/work/gap2/dbindexer/compiler.py", line 37, in execute_sql
    resolver.convert_insert_query(self.query)
  File "/home/wilfred/work/gap2/dbindexer/resolver.py", line 39, in convert_insert_query
    backend.convert_insert_query(query)
  File "/home/wilfred/work/gap2/dbindexer/backends.py", line 53, in convert_insert_query
    self._convert_insert_query(query, lookup)
  File "/home/wilfred/work/gap2/dbindexer/backends.py", line 64, in _convert_insert_query
    value = lookup.convert_value(value)
  File "/home/wilfred/work/gap2/dbindexer/lookups.py", line 54, in convert_value
    value = self._convert_value(value)
  File "/home/wilfred/work/gap2/dbindexer/lookups.py", line 175, in _convert_value
    return value.lower()
AttributeError: 'NoneType' object has no attribute 'lower'

When I define an ExtraField lookup on a model, proxy models for that model raise an error when trying to save.

'ProxyModel' object has no attribute 'idxf_team_name_l_icontains'

Patch that allows indexed dates to be None:

diff --git a/dbindexer/lookups.py b/dbindexer/lookups.py
index c72b8a36971d1c9c6911cf418df0b2bf6764ea60..5773fe0b1ee97b70702c9567400369e6d038b786 100644
--- a/dbindexer/lookups.py
+++ b/dbindexer/lookups.py
@@ -87,25 +87,37 @@ class Day(DateLookup):
     lookup_types = 'day'

     def _convert_value(self, value):
-        return value.day
+        if value:
+            return value.day
+        else:
+            return None

 class Month(DateLookup):
     lookup_types = 'month'

     def _convert_value(self, value):
-        return value.month
+        if value:
+            return value.month
+        else:
+            return None

 class Year(DateLookup):
     lookup_types = 'year'

     def _convert_value(self, value):
-        return value.year
+        if value:
+            return value.year
+        else:
+            return None

 class Weekday(DateLookup):
     lookup_types = 'week_day'

     def _convert_value(self, value):
-        return value.isoweekday()
+        if value:
+            return value.isoweekday()
+        else:
+            return None

 class Contains(ExtraFieldLookup):
     lookup_types = 'contains'

I am getting issues setting up django-nonrel.
I followed steps as http://djangoappengine.readthedocs.org/en/latest/
and then I run this command from django-tesapp folder
$ python manage.py runserver
and I get this :
Traceback (most recent call last):
File "manage.py", line 14, in
execute_manager(settings)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/core/management/init.py", line 438, in execute_manager
utility.execute()
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/core/management/init.py", line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/core/management/init.py", line 261, in fetch_command
klass = load_command_class(app_name, subcommand)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/core/management/init.py", line 67, in load_command_class
module = import_module('%s.management.commands.%s' % (app_name, name))
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/utils/importlib.py", line 35, in import_module
import(name)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/djangoappengine/management/commands/runserver.py", line 5, in
from django.db import connections
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/db/init.py", line 78, in
connection = connections[DEFAULT_DB_ALIAS]
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/db/utils.py", line 94, in getitem
conn = backend.DatabaseWrapper(db, alias)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/dbindexer/base.py", line 54, in DatabaseWrapper
return Wrapper(merged_settings, _args, *_kwargs)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/dbindexer/base.py", line 37, in init
super(BaseDatabaseWrapper, self).init(_args, *_kwargs)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/djangoappengine/db/base.py", line 290, in init
self.ops = DatabaseOperations(self)
TypeError: init() takes exactly 1 argument (2 given)
pradeep@pradeep:~/projects/python/gae_django_nonrel_app/django-testapp$ python manage.py createsuperuser
Traceback (most recent call last):
File "manage.py", line 14, in
execute_manager(settings)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/core/management/init.py", line 438, in execute_manager
utility.execute()
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/core/management/init.py", line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/core/management/init.py", line 261, in fetch_command
klass = load_command_class(app_name, subcommand)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/core/management/init.py", line 67, in load_command_class
module = import_module('%s.management.commands.%s' % (app_name, name))
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/utils/importlib.py", line 35, in import_module
import(name)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/contrib/auth/management/init.py", line 5, in
from django.contrib.auth import models as auth_app
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/contrib/auth/models.py", line 7, in
from django.db import models
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/db/init.py", line 78, in
connection = connections[DEFAULT_DB_ALIAS]
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/db/utils.py", line 94, in getitem
conn = backend.DatabaseWrapper(db, alias)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/dbindexer/base.py", line 54, in DatabaseWrapper
return Wrapper(merged_settings, _args, *_kwargs)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/dbindexer/base.py", line 37, in init
super(BaseDatabaseWrapper, self).init(_args, *_kwargs)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/djangoappengine/db/base.py", line 290, in init
self.ops = DatabaseOperations(self)
TypeError: init() takes exactly 1 argument (2 given)

HELP

Introduced in #5 . Apologies folks.

The latest version (currently at 041da37) doesn't work with Python 2.5 due to an ImportError trying to do:

from collections import Mapping

Mapping doesn't exist in Python 2.5. Running manage.py will give an error.

Given the models:

from django.db import models

class Foo(models.Model):
    name = models.CharField(max_length=100, blank=True, null=True)

class PointsToFoo(models.Model):
    foo = models.ForeignKey(Foo, null=True)

And given the following settings (note the FkNullFix):

DATABASES['default'] = {'ENGINE': 'dbindexer', 'TARGET': 'native'}

DBINDEXER_BACKENDS = (
    'dbindexer.backends.BaseResolver',
    'dbindexer.backends.FKNullFix',
    'dbindexer.backends.InMemoryJOINResolver',
)

Querying for null foreign keys generates the following crash:

from django.test import TestCase
from artproject_debug.models import PointsToFoo

class DBIndexerNullableFK(TestCase):
    def test_dbindexer(self):
        filtered_queryset = PointsToFoo.objects.filter(foo=None)

        # force query to occur (yes, twice is necessary)
        list(filtered_queryset[:500])
        list(filtered_queryset[:500])

Traceback:

======================================================================
ERROR: test_dbindexer (artproject_debug.tests.DBIndexerNullableFK)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/wilfred/work/gap2/artproject_debug/tests.py", line 10, in test_dbindexer
    list(filtered_queryset[:500])
  File "/home/wilfred/work/gap2/django/db/models/query.py", line 85, in __len__
    self._result_cache.extend(self._iter)
  File "/home/wilfred/work/gap2/django/db/models/query.py", line 276, in iterator
    for row in compiler.results_iter():
  File "/home/wilfred/work/gap2/dbindexer/compiler.py", line 27, in results_iter
    self.convert_filters()
  File "/home/wilfred/work/gap2/dbindexer/compiler.py", line 19, in convert_filters
    resolver.convert_filters(self.query)
  File "/home/wilfred/work/gap2/dbindexer/resolver.py", line 31, in convert_filters
    backend.convert_filters(query)
  File "/home/wilfred/work/gap2/dbindexer/backends.py", line 68, in convert_filters
    self._convert_filters(query, query.where)
  File "/home/wilfred/work/gap2/dbindexer/backends.py", line 75, in _convert_filters
    self._convert_filters(query, child)
  File "/home/wilfred/work/gap2/dbindexer/backends.py", line 78, in _convert_filters
    self.convert_filter(query, filters, child, index)
  File "/home/wilfred/work/gap2/dbindexer/backends.py", line 167, in convert_filter
    self.fix_fk_null_filter(query, constraint)
  File "/home/wilfred/work/gap2/dbindexer/backends.py", line 184, in fix_fk_null_filter
    self.unref_alias(query, alias)
  File "/home/wilfred/work/gap2/dbindexer/backends.py", line 170, in unref_alias
    unref_alias(query, alias)
  File "/home/wilfred/work/gap2/dbindexer/backends.py", line 145, in unref_alias
    query.table_map[table_name].remove(alias)
ValueError: list.remove(x): x not in list

i've been working on dbindexer to get it ready for django 1.4. i've fixed a couple of issues already, one related to threading changes with connection objects which caused the whole thing to break on appengine. but i've run into a bigger problem with the insert changes in 1.4.

a quick primer on dbindexer for those of you unfamiliar with it. say you want to do a case insensitive query on field 'name', your nonrel db can't do this for you, so dbindexer will secretly add another field to your model called 'name_iexact'. when you insert that model, dbindexer runs before the real insert compiler and copies the value from the 'name' field into the 'name_iexact' field (and converting it to lower case). so now your db row contains an additional field. when you query, it secretly rewrites the query to use this additional field.

in 1.3, insert compilers would get access to the list of already prepared values, so dbindexer could copy and convert those values. with 1.4, the insert query no longer contains prepped values, but instead a list of objects that are meant to be prepared by the compiler itself.

to get around this issue, i have dbindexer replace its hidden fields with proxy objects that look like the original non-dbindexer field but have a different column name. when the real insert compiler runs, it goes through the list of fields for each model instance and uses each field to prepare the value for insertion. the proxied dbindexer field will look up the original value and then store it in the new column.

its a bit of a hack. but you can see it here 657331c

for some reason its flaky, sometimes working, sometimes not. i'm still refining. if anyone has ideas, i'd sure appreciate it.