django-nonrel / django-dbindexer Goto Github PK
View Code? Open in Web Editor NEWAdds support for SQL features like JOINs to non-relational Django backends
License: BSD 3-Clause "New" or "Revised" License
Adds support for SQL features like JOINs to non-relational Django backends
License: BSD 3-Clause "New" or "Revised" License
Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. All documentation is in the "docs" directory and online at http://docs.djangoproject.com/en/dev/. If you're just getting started, here's how we recommend you read the docs: * First, read docs/intro/install.txt for instructions on installing Django. * Next, work through the tutorials in order (docs/intro/tutorial01.txt, docs/intro/tutorial02.txt, etc.). * If you want to set up an actual deployment server, read docs/howto/deployment/index.txt for instructions. * You'll probably want to read through the topical guides (in docs/topics) next; from there you can jump to the HOWTOs (in docs/howto) for specific problems, and check out the reference (docs/ref) for gory details. * See docs/README for instructions on building an HTML version of the docs. Docs are updated rigorously. If you find any problems in the docs, or think they should be clarified in any way, please take 30 seconds to fill out a ticket here: http://code.djangoproject.com/newticket To get more help: * Join the #django channel on irc.freenode.net. Lots of helpful people hang out there. Read the archives at http://django-irc-logs.com/. * Join the django-users mailing list, or read the archives, at http://groups.google.com/group/django-users. To contribute to Django: * Check out http://www.djangoproject.com/community/ for information about getting involved. To run Django's test suite: * Follow the instructions in the "Unit tests" section of docs/internals/contributing.txt, published online at https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/unit-tests/#running-the-unit-tests
Taken from bitbucket: https://bitbucket.org/Wilfred/django-dbindexer/issue/6/crash-on-nullable-values
Here's a minimal test case:
models.py:
from django.db import models
class Test(models.Model):
display_name = models.CharField(max_length=400, null=True)
index:
register_index(Test, {'display_name': 'istartswith'})
traceback:
$ python2.5 manage.py shell
>>> from artproject_artwork.models import Test
>>> Test.objects.create()
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/wilfred/work/gap2/django/db/models/manager.py", line 138, in create
return self.get_query_set().create(**kwargs)
File "/home/wilfred/work/gap2/django/db/models/query.py", line 362, in create
obj.save(force_insert=True, using=self.db)
File "/home/wilfred/work/gap2/django/db/models/base.py", line 462, in save
self.save_base(using=using, force_insert=force_insert, force_update=force_update)
File "/home/wilfred/work/gap2/django/db/models/base.py", line 573, in save_base
result = manager._insert(values, return_id=update_pk, using=using)
File "/home/wilfred/work/gap2/django/db/models/manager.py", line 195, in _insert
return insert_query(self.model, values, **kwargs)
File "/home/wilfred/work/gap2/django/db/models/query.py", line 1438, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "/home/wilfred/work/gap2/dbindexer/compiler.py", line 37, in execute_sql
resolver.convert_insert_query(self.query)
File "/home/wilfred/work/gap2/dbindexer/resolver.py", line 39, in convert_insert_query
backend.convert_insert_query(query)
File "/home/wilfred/work/gap2/dbindexer/backends.py", line 53, in convert_insert_query
self._convert_insert_query(query, lookup)
File "/home/wilfred/work/gap2/dbindexer/backends.py", line 64, in _convert_insert_query
value = lookup.convert_value(value)
File "/home/wilfred/work/gap2/dbindexer/lookups.py", line 54, in convert_value
value = self._convert_value(value)
File "/home/wilfred/work/gap2/dbindexer/lookups.py", line 175, in _convert_value
return value.lower()
AttributeError: 'NoneType' object has no attribute 'lower'
When I define an ExtraField lookup on a model, proxy models for that model raise an error when trying to save.
'ProxyModel' object has no attribute 'idxf_team_name_l_icontains'
Patch that allows indexed dates to be None:
diff --git a/dbindexer/lookups.py b/dbindexer/lookups.py
index c72b8a36971d1c9c6911cf418df0b2bf6764ea60..5773fe0b1ee97b70702c9567400369e6d038b786 100644
--- a/dbindexer/lookups.py
+++ b/dbindexer/lookups.py
@@ -87,25 +87,37 @@ class Day(DateLookup):
lookup_types = 'day'
def _convert_value(self, value):
- return value.day
+ if value:
+ return value.day
+ else:
+ return None
class Month(DateLookup):
lookup_types = 'month'
def _convert_value(self, value):
- return value.month
+ if value:
+ return value.month
+ else:
+ return None
class Year(DateLookup):
lookup_types = 'year'
def _convert_value(self, value):
- return value.year
+ if value:
+ return value.year
+ else:
+ return None
class Weekday(DateLookup):
lookup_types = 'week_day'
def _convert_value(self, value):
- return value.isoweekday()
+ if value:
+ return value.isoweekday()
+ else:
+ return None
class Contains(ExtraFieldLookup):
lookup_types = 'contains'
I am getting issues setting up django-nonrel.
I followed steps as http://djangoappengine.readthedocs.org/en/latest/
and then I run this command from django-tesapp folder
$ python manage.py runserver
and I get this :
Traceback (most recent call last):
File "manage.py", line 14, in
execute_manager(settings)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/core/management/init.py", line 438, in execute_manager
utility.execute()
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/core/management/init.py", line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/core/management/init.py", line 261, in fetch_command
klass = load_command_class(app_name, subcommand)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/core/management/init.py", line 67, in load_command_class
module = import_module('%s.management.commands.%s' % (app_name, name))
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/utils/importlib.py", line 35, in import_module
import(name)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/djangoappengine/management/commands/runserver.py", line 5, in
from django.db import connections
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/db/init.py", line 78, in
connection = connections[DEFAULT_DB_ALIAS]
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/db/utils.py", line 94, in getitem
conn = backend.DatabaseWrapper(db, alias)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/dbindexer/base.py", line 54, in DatabaseWrapper
return Wrapper(merged_settings, _args, *_kwargs)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/dbindexer/base.py", line 37, in init
super(BaseDatabaseWrapper, self).init(_args, *_kwargs)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/djangoappengine/db/base.py", line 290, in init
self.ops = DatabaseOperations(self)
TypeError: init() takes exactly 1 argument (2 given)
pradeep@pradeep:~/projects/python/gae_django_nonrel_app/django-testapp$ python manage.py createsuperuser
Traceback (most recent call last):
File "manage.py", line 14, in
execute_manager(settings)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/core/management/init.py", line 438, in execute_manager
utility.execute()
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/core/management/init.py", line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/core/management/init.py", line 261, in fetch_command
klass = load_command_class(app_name, subcommand)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/core/management/init.py", line 67, in load_command_class
module = import_module('%s.management.commands.%s' % (app_name, name))
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/utils/importlib.py", line 35, in import_module
import(name)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/contrib/auth/management/init.py", line 5, in
from django.contrib.auth import models as auth_app
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/contrib/auth/models.py", line 7, in
from django.db import models
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/db/init.py", line 78, in
connection = connections[DEFAULT_DB_ALIAS]
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/django/db/utils.py", line 94, in getitem
conn = backend.DatabaseWrapper(db, alias)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/dbindexer/base.py", line 54, in DatabaseWrapper
return Wrapper(merged_settings, _args, *_kwargs)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/dbindexer/base.py", line 37, in init
super(BaseDatabaseWrapper, self).init(_args, *_kwargs)
File "/home/pradeep/projects/python/gae_django_nonrel_app/django-testapp/djangoappengine/db/base.py", line 290, in init
self.ops = DatabaseOperations(self)
TypeError: init() takes exactly 1 argument (2 given)
HELP
Introduced in #5 . Apologies folks.
The latest version (currently at 041da37) doesn't work with Python 2.5 due to an ImportError trying to do:
from collections import Mapping
Mapping doesn't exist in Python 2.5. Running manage.py will give an error.
Given the models:
from django.db import models
class Foo(models.Model):
name = models.CharField(max_length=100, blank=True, null=True)
class PointsToFoo(models.Model):
foo = models.ForeignKey(Foo, null=True)
And given the following settings (note the FkNullFix):
DATABASES['default'] = {'ENGINE': 'dbindexer', 'TARGET': 'native'}
DBINDEXER_BACKENDS = (
'dbindexer.backends.BaseResolver',
'dbindexer.backends.FKNullFix',
'dbindexer.backends.InMemoryJOINResolver',
)
Querying for null foreign keys generates the following crash:
from django.test import TestCase
from artproject_debug.models import PointsToFoo
class DBIndexerNullableFK(TestCase):
def test_dbindexer(self):
filtered_queryset = PointsToFoo.objects.filter(foo=None)
# force query to occur (yes, twice is necessary)
list(filtered_queryset[:500])
list(filtered_queryset[:500])
Traceback:
======================================================================
ERROR: test_dbindexer (artproject_debug.tests.DBIndexerNullableFK)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/wilfred/work/gap2/artproject_debug/tests.py", line 10, in test_dbindexer
list(filtered_queryset[:500])
File "/home/wilfred/work/gap2/django/db/models/query.py", line 85, in __len__
self._result_cache.extend(self._iter)
File "/home/wilfred/work/gap2/django/db/models/query.py", line 276, in iterator
for row in compiler.results_iter():
File "/home/wilfred/work/gap2/dbindexer/compiler.py", line 27, in results_iter
self.convert_filters()
File "/home/wilfred/work/gap2/dbindexer/compiler.py", line 19, in convert_filters
resolver.convert_filters(self.query)
File "/home/wilfred/work/gap2/dbindexer/resolver.py", line 31, in convert_filters
backend.convert_filters(query)
File "/home/wilfred/work/gap2/dbindexer/backends.py", line 68, in convert_filters
self._convert_filters(query, query.where)
File "/home/wilfred/work/gap2/dbindexer/backends.py", line 75, in _convert_filters
self._convert_filters(query, child)
File "/home/wilfred/work/gap2/dbindexer/backends.py", line 78, in _convert_filters
self.convert_filter(query, filters, child, index)
File "/home/wilfred/work/gap2/dbindexer/backends.py", line 167, in convert_filter
self.fix_fk_null_filter(query, constraint)
File "/home/wilfred/work/gap2/dbindexer/backends.py", line 184, in fix_fk_null_filter
self.unref_alias(query, alias)
File "/home/wilfred/work/gap2/dbindexer/backends.py", line 170, in unref_alias
unref_alias(query, alias)
File "/home/wilfred/work/gap2/dbindexer/backends.py", line 145, in unref_alias
query.table_map[table_name].remove(alias)
ValueError: list.remove(x): x not in list
i've been working on dbindexer to get it ready for django 1.4. i've fixed a couple of issues already, one related to threading changes with connection objects which caused the whole thing to break on appengine. but i've run into a bigger problem with the insert changes in 1.4.
a quick primer on dbindexer for those of you unfamiliar with it. say you want to do a case insensitive query on field 'name', your nonrel db can't do this for you, so dbindexer will secretly add another field to your model called 'name_iexact'. when you insert that model, dbindexer runs before the real insert compiler and copies the value from the 'name' field into the 'name_iexact' field (and converting it to lower case). so now your db row contains an additional field. when you query, it secretly rewrites the query to use this additional field.
in 1.3, insert compilers would get access to the list of already prepared values, so dbindexer could copy and convert those values. with 1.4, the insert query no longer contains prepped values, but instead a list of objects that are meant to be prepared by the compiler itself.
to get around this issue, i have dbindexer replace its hidden fields with proxy objects that look like the original non-dbindexer field but have a different column name. when the real insert compiler runs, it goes through the list of fields for each model instance and uses each field to prepare the value for insertion. the proxied dbindexer field will look up the original value and then store it in the new column.
its a bit of a hack. but you can see it here 657331c
for some reason its flaky, sometimes working, sometimes not. i'm still refining. if anyone has ideas, i'd sure appreciate it.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.