Giter VIP home page Giter VIP logo

mssql-django's Introduction

Microsoft Django backend for SQL Server

Welcome to the MSSQL-Django 3rd party backend project!

mssql-django is a fork of django-mssql-backend. This project provides an enterprise database connectivity option for the Django Web Framework, with support for Microsoft SQL Server and Azure SQL Database.

We'd like to give thanks to the community that made this project possible, with particular recognition of the contributors: OskarPersson, michiya, dlo and the original Google Code django-pyodbc team. Moving forward we encourage partipation in this project from both old and new contributors!

We hope you enjoy using the MSSQL-Django 3rd party backend.

Features

Dependencies

  • pyodbc 3.0 or newer

Installation

  1. Install pyodbc 3.0 (or newer) and Django

  2. Install mssql-django:

    pip install mssql-django
    
  3. Set the ENGINE setting in the settings.py file used by your Django application or project to 'mssql':

    'ENGINE': 'mssql'
    

Configuration

Standard Django settings

The following entries in a database-level settings dictionary in DATABASES control the behavior of the backend:

  • ENGINE

    String. It must be "mssql".

  • NAME

    String. Database name. Required.

  • HOST

    String. SQL Server instance in "server\instance" format.

  • PORT

    String. Server instance port. An empty string means the default port.

  • USER

    String. Database user name in "user" format. If not given then MS Integrated Security will be used.

  • PASSWORD

    String. Database user password.

  • TOKEN

    String. Access token fetched as a user or service principal which has access to the database. E.g. when using azure.identity, the result of DefaultAzureCredential().get_token('https://database.windows.net/.default') can be passed.

  • AUTOCOMMIT

    Boolean. Set this to False if you want to disable Django's transaction management and implement your own.

  • Trusted_Connection

    String. Default is "yes". Can be set to "no" if required.

and the following entries are also available in the TEST dictionary for any given database-level settings dictionary:

  • NAME

    String. The name of database to use when running the test suite. If the default value (None) is used, the test database will use the name "test_" + NAME.

  • COLLATION

    String. The collation order to use when creating the test database. If the default value (None) is used, the test database is assigned the default collation of the instance of SQL Server.

  • DEPENDENCIES

    String. The creation-order dependencies of the database. See the official Django documentation for more details.

  • MIRROR

    String. The alias of the database that this database should mirror during testing. Default value is None. See the official Django documentation for more details.

OPTIONS

Dictionary. Current available keys are:

  • driver

    String. ODBC Driver to use ("ODBC Driver 17 for SQL Server", "SQL Server Native Client 11.0", "FreeTDS" etc). Default is "ODBC Driver 17 for SQL Server".

  • isolation_level

    String. Sets transaction isolation level for each database session. Valid values for this entry are READ UNCOMMITTED, READ COMMITTED, REPEATABLE READ, SNAPSHOT, and SERIALIZABLE. Default is None which means no isolation level is set to a database session and SQL Server default will be used.

  • dsn

    String. A named DSN can be used instead of HOST.

  • host_is_server

    Boolean. Only relevant if using the FreeTDS ODBC driver under Unix/Linux.

    By default, when using the FreeTDS ODBC driver the value specified in the HOST setting is used in a SERVERNAME ODBC connection string component instead of being used in a SERVER component; this means that this value should be the name of a dataserver definition present in the freetds.conf FreeTDS configuration file instead of a hostname or an IP address.

    But if this option is present and its value is True, this special behavior is turned off. Instead, connections to the database server will be established using HOST and PORT options, without requiring freetds.conf to be configured.

    See https://www.freetds.org/userguide/dsnless.html for more information.

  • unicode_results

    Boolean. If it is set to True, pyodbc's unicode_results feature is activated and strings returned from pyodbc are always Unicode. Default value is False.

  • extra_params

    String. Additional parameters for the ODBC connection. The format is "param=value;param=value", Azure AD Authentication (Service Principal, Interactive, Msi) can be added to this field.

  • collation

    String. Name of the collation to use when performing text field lookups against the database. Default is None; this means no collation specifier is added to your lookup SQL (the default collation of your database will be used). For Chinese language you can set it to "Chinese_PRC_CI_AS".

  • connection_timeout

    Integer. Sets the timeout in seconds for the database connection process. Default value is 0 which disables the timeout.

  • connection_retries

    Integer. Sets the times to retry the database connection process. Default value is 5.

  • connection_retry_backoff_time

    Integer. Sets the back off time in seconds for reries of the database connection process. Default value is 5.

  • query_timeout

    Integer. Sets the timeout in seconds for the database query. Default value is 0 which disables the timeout.

  • setencoding and setdecoding

    # Example
    "OPTIONS": {
            "setdecoding": [
                {"sqltype": pyodbc.SQL_CHAR, "encoding": 'utf-8'},
                {"sqltype": pyodbc.SQL_WCHAR, "encoding": 'utf-8'}],
            "setencoding": [
                {"encoding": "utf-8"}],
            ...
            },
  • return_rows_bulk_insert

    Boolean. Sets if backend can return rows from bulk insert. Default value is False which doesn't allows for the backend to return rows from bulk insert. Must be set to False if database has tables with triggers to prevent errors when inserting.

    # Examples
    "OPTIONS": {
        # This database doesn't have any triggers so can use return
        # rows from bulk insert feature
        "return_rows_bulk_insert": True
    }
    
    "OPTIONS": {
        # This database has triggers so leave return_rows_bulk_insert as blank (False)
        # to prevent errors related to inserting and returning rows from bulk insert
    }

Backend-specific settings

The following project-level settings also control the behavior of the backend:

  • DATABASE_CONNECTION_POOLING

    Boolean. If it is set to False, pyodbc's connection pooling feature won't be activated.

Example

Here is an example of the database settings:

    DATABASES = {
        'default': {
            'ENGINE': 'mssql',
            'NAME': 'mydb',
            'USER': 'user@myserver',
            'PASSWORD': 'password',
            'HOST': 'myserver.database.windows.net',
            'PORT': '',

            'OPTIONS': {
                'driver': 'ODBC Driver 17 for SQL Server',
            },
        },
    }

    # set this to False if you want to turn off pyodbc's connection pooling
    DATABASE_CONNECTION_POOLING = False

Limitations

The following features are currently not fully supported:

  • Altering a model field from or to AutoField at migration
  • Django annotate functions have floating point arithmetic problems in some cases
  • Annotate function with exists
  • Exists function in order_by
  • Righthand power and arithmetic with datatimes
  • Timezones, timedeltas not fully supported
  • Rename field/model with foreign key constraint
  • Database level constraints
  • Filtered index
  • Date extract function
  • Bulk insert into a table with a trigger and returning the rows inserted

JSONField lookups have limitations, more details here.

Contributing

More details on contributing can be found here.

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Security Reporting Instructions

For security reporting instructions please refer to the SECURITY.md file in this repository.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

mssql-django's People

Contributors

absci avatar adelkhayata76 avatar avidal avatar beruic avatar chris-beach avatar danifus avatar dauinsight avatar david-engel avatar davidjb avatar daybarr avatar dimbleby avatar federicoemartinez avatar guifran001 avatar hannylicious avatar henrikek avatar jean-frenette-optel avatar jlaundry avatar jmah8 avatar justinsg avatar khanhmaibui avatar kilrogg avatar linkwithkk avatar michiya avatar mshan0 avatar oskarpersson avatar sg3-141-592 avatar sparrowt avatar timnyborg avatar tonybaloney avatar vwarchu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mssql-django's Issues

Error on initial migration: Applying contenttypes.0001_initial...Traceback

Using django 3.2 and mssql-django 1.0b1 I get the following error runing the first migrate

Operations to perform:
dj    |   Apply all migrations: admin, auth, contenttypes, sessions, silk
dj    | Running migrations:
dj    |   Applying contenttypes.0001_initial...Traceback (most recent call last):
dj    |   File "/backend/manage.py", line 21, in <module>
dj    |     main()
dj    |   File "/backend/manage.py", line 17, in main
dj    |     execute_from_command_line(sys.argv)
dj    |   File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
dj    |     utility.execute()
dj    |   File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
dj    |     self.fetch_command(subcommand).run_from_argv(self.argv)
dj    |   File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
dj    |     self.execute(*args, **cmd_options)
dj    |   File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
dj    |     output = self.handle(*args, **options)
dj    |   File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 89, in wrapped
dj    |     res = handle_func(*args, **kwargs)
dj    |   File "/usr/local/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 244, in handle
dj    |     post_migrate_state = executor.migrate(
dj    |   File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 117, in migrate
dj    |     state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
dj    |   File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
dj    |     state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
dj    |   File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
dj    |     state = migration.apply(state, schema_editor)
dj    |   File "/usr/local/lib/python3.9/site-packages/django/db/migrations/migration.py", line 126, in apply
dj    |     operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
dj    |   File "/usr/local/lib/python3.9/site-packages/django/db/migrations/operations/models.py", line 528, in database_forwards
dj    |     alter_together(
dj    |   File "/usr/local/lib/python3.9/site-packages/mssql/schema.py", line 167, in alter_unique_together
dj    |     self.execute(sql)
dj    |   File "/usr/local/lib/python3.9/site-packages/mssql/schema.py", line 856, in execute
dj    |     sql = str(sql)
dj    |   File "/usr/local/lib/python3.9/site-packages/django/db/backends/ddl_references.py", line 201, in __str__
dj    |     return self.template % self.parts
dj    | **KeyError: 'include'**

JSON Field model cause error during migrate

Has anyone come across with an error from models.JSONField() which is caused from models.py. Even the migrations file defined the field as models.TextField(). Because if my understanding is right, there is no JSON column inside MS SQL.

An example of error while migrating
ERRORS: app.SomeModel: (fields.E180) SQL Server does not support JSONFields. script returned exit code 1

The thing is, I don't want to loads and dumps JSON back and forth from the column and JSONField is doing the tedious work for me. Is there an alternative solution for this kind of problem?

Temp Tables for big delete queries don't work for UUIDs

Hello,

we are stuggeling with the following issue:

I'm running a big delete query with ~ 17000 entries:
Model.objects.filter(pk__in=delete_list).delete()

What mssql-django does is to create a temp-table to not exceeed the maximum number of parameters:
CREATE TABLE #Temp_params (params nvarchar(32))
INSERT INTO #Temp_params VALUES ('fe5633e3-5ad9-471a-9046-000364aada58'),

Unfortunately my primary key is a UUID and hence has 36 characters. Wouldn't it make more sense to create a temp table that has the same datatype as the field to be filtered for?

Thanks in adavance for your help.

Nullable OneToOneFields

Due to MS-SQL not properly implementing the ANSI SQL specification on this issue, multiple nulls in a column violate a Unique constraint. This means it is practically impossible to create nullable OneToOneFields in Django. There appears to be a possible fix -- https://stackoverflow.com/a/767702.

I am going to try to implement a fix for my own use case and will share it once I have done so, but I thought I would post an issue in case someone has solved this issue in the past or has any tips.

Domain user auth Failing

Hi

I'm able to use Sql user to communicate with the DB but using domain account is causing error:

"[28000] [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Login failed for user 'domain\user'. (18456) Invalid connection string attribute (0)"

These are the configuration that I'm using:

`DATABASES = {
'default': {
'ENGINE': 'mssql',
'NAME': 'mydb',
'USER': 'domain\user', # I've checked with domain\\user as well user@domain
'PASSWORD': 'pass',
'HOST': 'myserver',
'PORT': '',
'OPTIONS': {
'driver': 'ODBC Driver 11 for SQL Server',
},

},

}`

Environment:
Django 3.1
Windows server 2012 R2
mssql-django 1.0b1
SQL server 2014

Am I missing something obvious ,Can you help me out with this ?

Natural Key not implemented?

Thank you for creating this library so we can support MS SQL using Django.

I am trying to seed my Django app using natural keys instead of hard-coding IDs/PKs like this: https://docs.djangoproject.com/en/3.2/topics/serialization/#natural-keys

But I don't think this library (e.g. mssql-django) recognizes natural keys.

I've tried the same code/YAML that is working with a Django app using PostgreSQL on a Django using MS SQL. But I have been unsuccessful getting the natural keys working on the MS SQL Django app.
Also, when I issue
/manage.py dumpdata --natural-foreign --format yaml
The output YAML file still has normal IDs.

Are natural keys supposed to be working and I did something wrong?
Or are natural keys not implemented and it might or might not be on the roadmap?

Thank you for your time ๐Ÿ™๐Ÿป

SMux Provider: Physical connection is not usable [xFFFFFFFF] & The cursor's connection has been closed

I am desperately trying to migrate our mysql-based Django solution to an Azure SQL database.

Steps I have performed:

  • Run python manage.py dumpdata --format json --indent 2 C:\Temp\dumb.json to backup data from source instance using MySQL backend.

  • Changed settings.py so it points to target Azure SQL database

DATABASES = {
    'default': {
        'ENGINE': 'mssql',
        'NAME': 'sqldb-mydb',
        'USER': 'myuser@sql-myserver',
        'PASSWORD': '<hidden>',
        'HOST': 'sql-myserver.database.windows.net',
        'PORT': '1433',
        'OPTIONS': {
            'driver': 'ODBC Driver 17 for SQL Server',
			
            # Integer. Sets the timeout in seconds for the database connection process. Default value is 0 which disables the timeout.			
            'connection_timeout': '0',
			
            # Integer. Sets the times to retry the database connection process. Default value is 5.
            'connection_retries': '5',
			
            # Integer. Sets the back off time in seconds for reries of the database connection process. Default value is 5.
            'connection_retry_backoff_time': '5',
			
            # Integer. Sets the timeout in seconds for the database query. Default value is 0 which disables the timeout.
            'query_timeout': '0'
        }
    }
}

# Set this to False if you want to turn off pyodbc's connection pooling
DATABASE_CONNECTION_POOLING = True 
  • Run python manage.py migrate to create schema on target database (Azure)

  • Run python manage.py loaddata --format json --verbosity 3 C:\Temp\dumb.json to restore data on target database.

However, it fails with the following output

โฏ python .\manage.py loaddata --format json --verbosity 3 C:\temp\dump.json
Time of launch 2021-10-11 14:30:19.403482

Loading 'C:\temp\dump' fixtures...
Checking 'C:\temp' for fixtures...
Installing json fixture 'dump' from 'C:\temp'.
Processed 58017 object(s).Traceback (most recent call last):
  File "C:\dev\venv\django2-dev\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "C:\dev\venv\django2-dev\lib\site-packages\mssql\base.py", line 567, in execute
    return self.cursor.execute(sql, params)
pyodbc.Error: ('08S02', '[08S02] [Microsoft][ODBC Driver 17 for SQL Server]SMux Provider: Physical connection is not usable [xFFFFFFFF].  (-1) (SQLPrepare); [08S02] [Microsoft][ODBC Driver 17 for SQL Server]Unable to open a logical session (-1)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\dev\venv\django2-dev\lib\site-packages\django\db\models\sql\compiler.py", line 1142, in execute_sql
    cursor.execute(sql, params)
  File "C:\dev\venv\django2-dev\lib\site-packages\django\db\backends\utils.py", line 99, in execute
    return super().execute(sql, params)
  File "C:\dev\venv\django2-dev\lib\site-packages\django\db\backends\utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "C:\dev\venv\django2-dev\lib\site-packages\django\db\backends\utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "C:\dev\venv\django2-dev\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "C:\dev\venv\django2-dev\lib\site-packages\django\db\utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "C:\dev\venv\django2-dev\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "C:\dev\venv\django2-dev\lib\site-packages\mssql\base.py", line 567, in execute
    return self.cursor.execute(sql, params)
django.db.utils.Error: ('08S02', '[08S02] [Microsoft][ODBC Driver 17 for SQL Server]SMux Provider: Physical connection is not usable [xFFFFFFFF].  (-1) (SQLPrepare); [08S02] [Microsoft][ODBC Driver 17 for SQL Server]Unable to open a logical session (-1)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\dev\repos\<my-application>\backend\manage.py", line 21, in <module>
    execute_from_command_line(sys.argv)
  File "C:\dev\venv\django2-dev\lib\site-packages\django\core\management\__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "C:\dev\venv\django2-dev\lib\site-packages\django\core\management\__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "C:\dev\venv\django2-dev\lib\site-packages\django\core\management\base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "C:\dev\venv\django2-dev\lib\site-packages\django\core\management\base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "C:\dev\venv\django2-dev\lib\site-packages\django\core\management\commands\loaddata.py", line 72, in handle
    self.loaddata(fixture_labels)
  File "C:\dev\venv\django2-dev\lib\site-packages\django\core\management\commands\loaddata.py", line 114, in loaddata
    self.load_label(fixture_label)
  File "C:\dev\venv\django2-dev\lib\site-packages\django\core\management\commands\loaddata.py", line 181, in load_label
    obj.save(using=self.using)
  File "C:\dev\venv\django2-dev\lib\site-packages\django\core\serializers\base.py", line 223, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "C:\dev\venv\django2-dev\lib\site-packages\django\db\models\base.py", line 780, in save_base
    updated = self._save_table(
  File "C:\dev\venv\django2-dev\lib\site-packages\django\db\models\base.py", line 853, in _save_table
    updated = self._do_update(base_qs, using, pk_val, values, update_fields,
  File "C:\dev\venv\django2-dev\lib\site-packages\django\db\models\base.py", line 903, in _do_update
    return filtered._update(values) > 0
  File "C:\dev\venv\django2-dev\lib\site-packages\django\db\models\query.py", line 760, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "C:\dev\venv\django2-dev\lib\site-packages\django\db\models\sql\compiler.py", line 1471, in execute_sql
    cursor = super().execute_sql(result_type)
  File "C:\dev\venv\django2-dev\lib\site-packages\django\db\models\sql\compiler.py", line 1145, in execute_sql
    cursor.close()
  File "C:\dev\venv\django2-dev\lib\site-packages\mssql\base.py", line 521, in close
    self.cursor.close()
pyodbc.ProgrammingError: Problem installing fixture 'C:\temp\dump.json': The cursor's connection has been closed.

The processing of these 58'000 items took around 2 hours, why I had to disable Azure's auto-pause feature as it broke my connection before. However, not it failed with the message above.

These are the Azure SQL db service and compute tiers in use:

  • Service tier: General purpose (scalable compute and storage options)
  • Compute tier: Serverless
  • Max vCores: 4
  • Min vCores: 0.5
  • Data max size: 10 GB
  • Database is not zone redundant
  • Backup: Locally-redundant backup storage

Any help on this is greatly appreciated!!!!

Strange isolation level being set to "read committed"

I asked the same question on the old repo, but received no answer, so moved here:

"mssql": {
           "ENGINE": "sql_server.pyodbc",
         # "ENGINE": "mssql",
           "NAME": "...",
           "USER": "...",
           "PASSWORD": "....",
           "OPTIONS": {
                "dsn": "...",
                "initial_state": {
                    "quoted_identifier": "on",
                    "concat_null_yields_null": "on",
                    "ansi_warnings": "on",
                    "ansi_nulls": "on",
                    "ansi_padding": "on",
                    "implicit_transactions": "off"
                },
                "extra_params": "tds_version=7.3",
                "isolation_level": "READ UNCOMMITTED"
           },
          "TEST": {"MIRROR": "default"}
        }

I have the following settings with "isolation_level" set to `READ UNCOMMITTED", but every now and then we have query stuck with isolation level set to "READ COMMITTED", anyone having similar issue?

`nulls_first` or `nulls_last` ordering produces incorrect SQL Server syntax

Using the latest dev (f2d27ee), attempts to use nulls_first or nulls_last ordering produce invalid SQL.

#19 reports a related issue, but the change in #20 that modifies the database backend's features causes Django's other conditional pathway at https://github.com/django/django/blob/main/django/db/models/expressions.py#L1214-L1227 to be followed, resulting in invalid SQL being included in the SQL template, leading to the following syntax error:

Incorrect syntax near the keyword 'IS'

caused by of either of the following SQL statements that get created by sqlserver_orderby are invalid syntax:

  • .desc(nulls_first=True)

    '%(expression)s IS NOT NULL, CASE WHEN %(expression)s IS NULL THEN 0 ELSE 1 END, %(expression)s %(ordering)s'
  • .asc(nulls_last=True)

    '%(expression)s IS NULL, CASE WHEN %(expression)s IS NULL THEN 1 ELSE 0 END, %(expression)s %(ordering)s'

Tested with Django 3.1.8 but appears to still affect 3.2+. I believe https://code.djangoproject.com/ticket/32584 needs to be reopened because the template is still being overwritten, unless there's a better strategy for overriding Django's core behaviour. @timnyborg, do you have any thoughts?

Example Django model illustrating the issue:

from django.db import models

class Event(models.Model):
    # ... fields go here ...

    class Meta(PersonMetadata.Meta):
        ordering = (
            F("end_year").desc(nulls_first=True),
            F("start_year").asc(nulls_last=True),
        )

Event.objects.first()  # Incorrect syntax near the keyword 'IS'

Migration renaming a field fails

Migration renaming a field fails on mssql-django version 1.0 but succeeds on 1.0rc1. Tested on both Django 2.2.17 and 3.2.7. The issue is probably related to commit which removed the index deletion before column rename.

Migration which creates the field:

migrations.AddField(
    model_name='transaction',
    name='conf_first_distribution_date',
    field=models.OneToOneField(blank=True, null=True, default=None, on_delete=django.db.models.deletion.SET_NULL, related_name='first_distribution_date_transaction', to='transaction.DateConfig'),
)

Migration which tries to rename the field but fails:

migrations.RenameField(
    model_name='transaction',
    old_name='conf_first_distribution_date',
    new_name='first_distribution_date',
)

Output:

   Applying transaction.0016_auto_20200424_1103...Traceback (most recent call last):
   File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
     return self.cursor.execute(sql, params)
   File "/usr/local/lib/python3.8/site-packages/mssql/base.py", line 567, in execute
     return self.cursor.execute(sql, params)
 pyodbc.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]The index 'transaction_transaction_conf_first_distribution_date_id_76f3b97f_uniq' is dependent on column 'conf_first_distribution_date_id'. (5074) (SQLExecDirectW)")
 
 The above exception was the direct cause of the following exception:
 
 Traceback (most recent call last):
   File "dev_manage.py", line 15, in <module>
     execute_from_command_line(sys.argv)
   File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
     utility.execute()
   File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
     self.fetch_command(subcommand).run_from_argv(self.argv)
   File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
     self.execute(*args, **cmd_options)
   File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
     output = self.handle(*args, **options)
   File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 89, in wrapped
     res = handle_func(*args, **kwargs)
   File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 244, in handle
     post_migrate_state = executor.migrate(
   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
     state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
     state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
     state = migration.apply(state, schema_editor)
   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/migration.py", line 126, in apply
     operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
   File "/usr/local/lib/python3.8/site-packages/django/db/migrations/operations/fields.py", line 350, in database_forwards
     schema_editor.alter_field(
   File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 608, in alter_field
     self._alter_field(model, old_field, new_field, old_type, new_type,
   File "/usr/local/lib/python3.8/site-packages/mssql/schema.py", line 348, in _alter_field
     self.execute(self._rename_field_sql(model._meta.db_table, old_field, new_field, new_type))
   File "/usr/local/lib/python3.8/site-packages/mssql/schema.py", line 880, in execute
     cursor.execute(sql, params)
   File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
     return super().execute(sql, params)
   File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
   File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
     return executor(sql, params, many, context)
   File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
     return self.cursor.execute(sql, params)
   File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
     raise dj_exc_value.with_traceback(traceback) from exc_value
   File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
     return self.cursor.execute(sql, params)
   File "/usr/local/lib/python3.8/site-packages/mssql/base.py", line 567, in execute
     return self.cursor.execute(sql, params)
 django.db.utils.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]The index 'transaction_transaction_conf_first_distribution_date_id_76f3b97f_uniq' is dependent on column 'conf_first_distribution_date_id'. (5074) (SQLExecDirectW)")

How do I use sqlserver timestamp field?

I did not find a mapping for the TIMESTAMP field.

class DatabaseWrapper(BaseDatabaseWrapper):
    vendor = 'microsoft'
    display_name = 'SQL Server'
    # This dictionary maps Field objects to their associated MS SQL column
    # types, as strings. Column-type strings can contain format strings; they'll
    # be interpolated against the values of Field.__dict__ before being output.
    # If a column type is set to None, it won't be included in the output.
    data_types = {
        'AutoField': 'int',
        'BigAutoField': 'bigint',
        'BigIntegerField': 'bigint',
        'BinaryField': 'varbinary(max)',
        'BooleanField': 'bit',
        'CharField': 'nvarchar(%(max_length)s)',
        'DateField': 'date',
        'DateTimeField': 'datetime2',
        'DecimalField': 'numeric(%(max_digits)s, %(decimal_places)s)',
        'DurationField': 'bigint',
        'FileField': 'nvarchar(%(max_length)s)',
        'FilePathField': 'nvarchar(%(max_length)s)',
        'FloatField': 'double precision',
        'IntegerField': 'int',
        'IPAddressField': 'nvarchar(15)',
        'GenericIPAddressField': 'nvarchar(39)',
        'NullBooleanField': 'bit',
        'OneToOneField': 'int',
        'PositiveIntegerField': 'int',
        'PositiveSmallIntegerField': 'smallint',
        'SlugField': 'nvarchar(%(max_length)s)',
        'SmallAutoField': 'smallint',
        'SmallIntegerField': 'smallint',
        'TextField': 'nvarchar(max)',
        'TimeField': 'time',
        'UUIDField': 'char(32)',
    }

Any ETA on stable release?

Firstly, it is great to see that this package is being continued officially. ๐Ÿ’ฏ

I see currently this package is in beta. Do we have any ETA on stable release?

Support altering from/to (Big)AutoField

Hello.

I have been trying to migrate some of my tables, from <django.db.models.fields.IntegerField: id> to a <django.db.models.fields.BigAutoField: id>.

This change seems not to be supported by this backend since I am getting a NotImplementedError: the backend doesn't support altering from/to AutoField. . I would like to know if there is some way to make this work since I am trying to avoid scaling problems when my DB starts to increase.

I would be glad to work on this change too if necessary, I just need a little more context on why this is not supported yet and the problems I would be probably facing.

Thanks.

Document test processes & dependency installation

At present, there's no information in the README with regards to what tests should be run for this project and how to setup your environment and run them as a contributor.

Someone who's worked with Python/Django before probably knows to start looking at doing pip install -e . and running python manage.py test but setting that out explicitly would help. But that's not the end of the story -- delving into the project's files, there's https://github.com/microsoft/mssql-django/blob/dev/tox.ini but in order to run tox at all requires setting up your environment with a number of various dependencies and files: https://github.com/microsoft/mssql-django/blob/dev/azure-pipelines.yml (with tox itself, a git clone of Django, SQL Server instance running locally etc).

Having the information detailed and documented in the README would be fantastic to help take the guesswork out of ensuring a PR passes tests locally before committing.

Edit: here's my current process for my own or others' reference:

docker pull mcr.microsoft.com/mssql/server:2019-latest
docker run --name sqlserver -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=MyPassword42' -p 1433:1433 -d mcr.microsoft.com/mssql/server:2019-latest

cd path/to/mssql-django
python3 -m venv .
source ./bin/activate
pip install -e
pip install django==3.1.8
# See https://github.com/microsoft/mssql-django/issues/10
sed -i 's/"ODBC Driver 17 for SQL Server"/"FreeTDS", "host_is_server": True/g' testapp/settings.py 
python manage.py test

docker rm -f sqlserver 
# or stop with `docker stop sqlserver` and leave it there for next time

How to specify schema?

Is there a way to specify a schema other than the default? If so, could there be one?

I know there are ways for Django with a Postgres backend, but it doesn't look like it for MSSQL.

Thanks.

KeyError running migrations at Django 3.2

# python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, hunt, sessions
Running migrations:
 Applying contenttypes.0001_initial...Traceback (most recent call last):
  File "manage.py", line 21, in <module>
    main()
  File "manage.py", line 17, in main
    execute_from_command_line(sys.argv)
  File "/tmp/8d8f9b414dbe185/antenv/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/tmp/8d8f9b414dbe185/antenv/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/tmp/8d8f9b414dbe185/antenv/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/tmp/8d8f9b414dbe185/antenv/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/tmp/8d8f9b414dbe185/antenv/lib/python3.8/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/tmp/8d8f9b414dbe185/antenv/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/tmp/8d8f9b414dbe185/antenv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/tmp/8d8f9b414dbe185/antenv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/tmp/8d8f9b414dbe185/antenv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/tmp/8d8f9b414dbe185/antenv/lib/python3.8/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/tmp/8d8f9b414dbe185/antenv/lib/python3.8/site-packages/django/db/migrations/operations/models.py", line 528, in database_forwards
    alter_together(
  File "/tmp/8d8f9b414dbe185/antenv/lib/python3.8/site-packages/mssql/schema.py", line 167, in alter_unique_together
    self.execute(sql)
  File "/tmp/8d8f9b414dbe185/antenv/lib/python3.8/site-packages/mssql/schema.py", line 856, in execute
    sql = str(sql)
  File "/tmp/8d8f9b414dbe185/antenv/lib/python3.8/site-packages/django/db/backends/ddl_references.py", line 201, in __str__
    return self.template % self.parts
KeyError: 'include'

django.db.utils.Error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 17 for SQL Server' : file not found (0) (SQLDriverConnect)")

Hi,

Thank you for providing an officially supported library to connect Django with SQL Server.

I am running into the following error whenever Django tries to connect to the database:
django.db.utils.Error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 17 for SQL Server' : file not found (0) (SQLDriverConnect)")

I've tried to install the ODBC drivers with: yum install unixODBC-devel
These are the versions pip has installed:

mssql-django 1.0b1
pyodbc 4.0.30

These are my Django database settings:

DATABASES = {
    'default': {
        'ENGINE': 'mssql',
        'NAME': 'test',
        'USER': 'test',
        'PASSWORD': 'test',
        'HOST': 'database',
        'PORT': '1433',
        'OPTIONS': {
            'driver': 'ODBC Driver 17 for SQL Server',
        },
    }
}

Please let me know if there's anything else I can provide to help troubleshoot this. Or if there are any troubleshooting steps that you'd recommend I take, that would be extremely helpful.

Thank you for your time ๐Ÿ™๐Ÿป

In case it's useful here are the really long information:

I'm running this in a container based off of registry.access.redhat.com/ubi8/python-38.

All of the libraries and their versions:

$ pip list
Package                       Version
----------------------------- ---------
alabaster                     0.7.12
asgiref                       3.3.4
astroid                       2.5.6
Babel                         2.9.1
certifi                       2020.12.5
chardet                       4.0.0
coverage                      5.3.1
Django                        3.1.10
django-cors-headers           3.7.0
djangorestframework           3.12.4
djangorestframework-jsonapi   4.0.0
docutils                      0.17.1
idna                          2.10
imagesize                     1.2.0
inflection                    0.5.1
isort                         5.8.0
Jinja2                        2.11.3
lazy-object-proxy             1.6.0
MarkupSafe                    1.1.1
mccabe                        0.6.1
mssql-django                  1.0b1
packaging                     20.9
pip                           21.1.1
pycodestyle                   2.6.0
Pygments                      2.9.0
pylint                        2.8.2
pylint-django                 2.4.4
pylint-plugin-utils           0.6
pyodbc                        4.0.30
pyparsing                     2.4.7
pytz                          2021.1
PyYAML                        5.3.1
requests                      2.25.1
setuptools                    41.6.0
snowballstemmer               2.1.0
Sphinx                        3.4.3
sphinxcontrib-applehelp       1.0.2
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        1.0.3
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.4
sqlparse                      0.4.1
toml                          0.10.2
unittest-xml-reporting        3.0.4
urllib3                       1.26.4
wrapt                         1.12.1

Entire error output:

backend_1             | Traceback (most recent call last):
backend_1             |   File "/usr/lib64/python3.8/threading.py", line 932, in _bootstrap_inner
backend_1             |     self.run()
backend_1             |   File "/usr/lib64/python3.8/threading.py", line 870, in run
backend_1             |     self._target(*self._args, **self._kwargs)
backend_1             |   File "/opt/app-root/lib64/python3.8/site-packages/django/utils/autoreload.py", line 53, in wrapper
backend_1             |     fn(*args, **kwargs)
backend_1             |   File "/opt/app-root/lib64/python3.8/site-packages/django/core/management/commands/runserver.py", line 121, in inner_run
backend_1             |     self.check_migrations()
backend_1             |   File "/opt/app-root/lib64/python3.8/site-packages/django/core/management/base.py", line 459, in check_migrations
backend_1             |     executor = MigrationExecutor(connections[DEFAULT_DB_ALIAS])
backend_1             |   File "/opt/app-root/lib64/python3.8/site-packages/django/db/migrations/executor.py", line 18, in __init__
backend_1             |     self.loader = MigrationLoader(self.connection)
backend_1             |   File "/opt/app-root/lib64/python3.8/site-packages/django/db/migrations/loader.py", line 53, in __init__
backend_1             |     self.build_graph()
backend_1             |   File "/opt/app-root/lib64/python3.8/site-packages/django/db/migrations/loader.py", line 216, in build_graph
backend_1             |     self.applied_migrations = recorder.applied_migrations()
backend_1             |   File "/opt/app-root/lib64/python3.8/site-packages/django/db/migrations/recorder.py", line 77, in applied_migrations
backend_1             |     if self.has_table():
backend_1             |   File "/opt/app-root/lib64/python3.8/site-packages/django/db/migrations/recorder.py", line 55, in has_table
backend_1             |     with self.connection.cursor() as cursor:
backend_1             |   File "/opt/app-root/lib64/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
backend_1             |     return func(*args, **kwargs)
backend_1             |   File "/opt/app-root/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 259, in cursor
backend_1             |     return self._cursor()
backend_1             |   File "/opt/app-root/lib64/python3.8/site-packages/mssql/base.py", line 226, in _cursor
backend_1             |     conn = super()._cursor()
backend_1             |   File "/opt/app-root/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 235, in _cursor
backend_1             |     self.ensure_connection()
backend_1             |   File "/opt/app-root/lib64/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
backend_1             |     return func(*args, **kwargs)
backend_1             |   File "/opt/app-root/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
backend_1             |     self.connect()
backend_1             |   File "/opt/app-root/lib64/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
backend_1             |     raise dj_exc_value.with_traceback(traceback) from exc_value
backend_1             |   File "/opt/app-root/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
backend_1             |     self.connect()
backend_1             |   File "/opt/app-root/lib64/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
backend_1             |     return func(*args, **kwargs)
backend_1             |   File "/opt/app-root/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 200, in connect
backend_1             |     self.connection = self.get_new_connection(conn_params)
backend_1             |   File "/opt/app-root/lib64/python3.8/site-packages/mssql/base.py", line 320, in get_new_connection
backend_1             |     conn = Database.connect(connstr,
backend_1             | django.db.utils.Error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 17 for SQL Server' : file not found (0) (SQLDriverConnect)")

Please, is Always encrypted working in this project? Not sure if it is an unsupported feature.

Hello.

I am trying to use this project with the always encrypted feature and Azure sql.

A simple select * query is working, but returning encrypted field illegible so, I tried Pyodbc directly in a small program in python, which is working and returning field as clear text. In Django, however, I am using the equivalent parameters and it is returning the field as encrypted, not as clear text as it should. The OPTIONS parameter I am using is :
...
'OPTIONS': {
'driver': "ODBC Driver 17 for SQL Server",
'extra_params': 'ColumnEncryption=Enabled;KeyStoreAuthentication=KeyVaultClientSecret;KeyStorePrincipalId='+client_id+';KeyStoreSecret='+client_secret+'',
},
And as I said, the equivalent KeyStoreAuthentication, KeyStorePrincipalId, KeyStoreSecret parameters are working in python with pyodbc.

Thank you very much.

string to uniqueidentifier conversion exception

I have a table that uses an uniqueidentifier when adding a new record via admin page a uniqueidentifier conversion exception
is returned. Tried overriding .Save but with no luck, do you have any thoughts?

The table is as follows:
CREATE TABLE [dbo].[DictionaryEntries](
[EntryGUID] [uniqueidentifier] NOT NULL,
[Description] varchar NOT NULL,
CONSTRAINT [pkEntry] PRIMARY KEY CLUSTERED
(
[EntryGUID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, OPTIMIZE_FOR_SEQUENTIAL_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]
GO

ALTER TABLE [dbo].[DictionaryEntries] ADD  DEFAULT (newid()) FOR [EntryGUID]
GO

The Exception Message is as follows:
ProgrammingError at /admin/DictionaryApp/dictionaryentries/add/
('42000', '[42000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Conversion failed when converting from a character string to uniqueidentifier. (8169) (SQLExecDirectW)')
Request Method: POST
Request URL: http://127.0.0.1:8000/admin/DictionaryApp/dictionaryentries/add/
Django Version: 3.1
Exception Type: ProgrammingError
Exception Value:
('42000', '[42000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Conversion failed when converting from a character string to uniqueidentifier. (8169) (SQLExecDirectW)')
Exception Location: D:\python\code\dictionary.env\lib\site-packages\mssql\base.py, line 561, in execute
Python Executable: D:\python\code\dictionary.env\Scripts\python.exe
Python Version: 3.9.1

Model as follows:
from django.db import models
from uuid import UUID, uuid4
import uuid

class DictionaryEntries(models.Model):
EntryGUID = models.UUIDField(primary_key=True, default=uuid4, editable=False)
Description = models.CharField(db_column='Description', unique=True, max_length=50)

class Meta:
    managed = False
    db_table = 'DictionaryEntries'
    ordering = ['Description']
    verbose_name = "Dictionary Entries"
    verbose_name_plural = "Dictionary Entries"

what i tried:
override save in an attempt at manual conversion

def save(self, *args, **kwargs):
    self.EntryGUID = uuid(self.EntryGUID)
    super(DictionaryEntries, self).save(*args, **kwargs)

Default driver is not what the README claims

README says:

  • driver

String. ODBC Driver to use ("ODBC Driver 17 for SQL Server", "SQL Server Native Client 11.0", "FreeTDS" etc). Default is "ODBC Driver 17 for SQL Server".

code says:

driver = options.get('driver', 'ODBC Driver 13 for SQL Server')

(I found this because 17 is the value that one actually needs in an Azure App Service. But since the README told me that was the default, I left it alone; and then it didn't work.)

./manage.py test fails with using Azure SQL and Service Principal authentication

Background:

  • I'm using Django==3.1.13, and mssql-django==1.0
  • I have created Azure SQL Server, using Service Principal authentication, with two databases: [db] and [test_db]
  • the Service Principal has been configured with db_owner on both databases (i.e., CREATE USER [app] FROM EXTERNAL PROVIDER; EXEC sp_addrolemember 'db_owner', [app];)
  • I have updated DATABASES in settings.py with Authentication=ActiveDirectoryServicePrincipal

I can use Django with no issues.

However, when I run ./manage.py test --keepdb, I get the following error:

pyodbc.InterfaceError: ('28000', "[28000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Login failed for user '<token-identified principal>'. (18456) (SQLDriverConnect)")

This is because the test runner first connects to [dbo].[master], verifies [test_db] exists, and then reconnects to [test_db]. However, by default Service Principals do not have permission to access [dbo].[master] (and ideally shouldn't, when using the same server for multiple applications).

This appears to be an intentional design decision to try connecting without the DB name even when using keepdb, as per django/db/backends/base/creation.py:52-57:

        # We could skip this call if keepdb is True, but we instead
        # give it the keepdb param. This is to handle the case
        # where the test DB doesn't exist, in which case we need to
        # create it, then just not destroy it. If we instead skip
        # this, we will get an exception.
        self._create_test_db(verbosity, autoclobber, keepdb)

If I comment out this line, it works without issue.

I've raised this as a Django bug #33009, but this may be rejected as the mssql backend could instead override _create_test_db() or create_test_db() to provide a more specific check (like SQLite does, for example).

Add support for `unique=True` on `BinaryField`, `TextField`, and others

Microsoft SQL server has a set of maximum capacity specifications as documented Maximum capacity specifications for SQL Server

The DatabaseWrapper.data_types dictionary contains a max parameter for some fields in the database that may or may not have a max_length size set in the applications model:

mssql-django/mssql/base.py

Lines 74 to 102 in ff59848

data_types = {
'AutoField': 'int',
'BigAutoField': 'bigint',
'BigIntegerField': 'bigint',
'BinaryField': 'varbinary(max)',
'BooleanField': 'bit',
'CharField': 'nvarchar(%(max_length)s)',
'DateField': 'date',
'DateTimeField': 'datetime2',
'DecimalField': 'numeric(%(max_digits)s, %(decimal_places)s)',
'DurationField': 'bigint',
'FileField': 'nvarchar(%(max_length)s)',
'FilePathField': 'nvarchar(%(max_length)s)',
'FloatField': 'double precision',
'IntegerField': 'int',
'IPAddressField': 'nvarchar(15)',
'GenericIPAddressField': 'nvarchar(39)',
'JSONField': 'nvarchar(max)',
'NullBooleanField': 'bit',
'OneToOneField': 'int',
'PositiveIntegerField': 'int',
'PositiveSmallIntegerField': 'smallint',
'SlugField': 'nvarchar(%(max_length)s)',
'SmallAutoField': 'smallint',
'SmallIntegerField': 'smallint',
'TextField': 'nvarchar(max)',
'TimeField': 'time',
'UUIDField': 'char(32)',
}

Specifically, BinaryField, JSONField, TextField have a corresponding max size set instead of respecting the max_length property that can be set in an application's model.

For example:

class ClientDownload(models.Model):
    access_code = models.BinaryField(max_length=15, default=random_access_code, unique=True)
    # ...

A developer should expect the access_code field to be created in the database with the type varbinary(15) (or binary(15), but that may be out of scope of this issue). However, since the field's type will end up with varbinary(max), the unique=True constraint will throw an exception from SQL server as the field is too large to be indexed.

[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Column 'access_code' in table 'mypp_clientupload' is of a type that is invalid for use as a key column in an index. (1919) (SQLExecDirectW); [42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Could not create constraint or index. See previous errors. (1750)

This is due to the maximum bytes allowed for an indexable field

SQL Server Database Engine object Maximum sizes/numbers SQL Server (64-bit) Additional Information
Bytes per index key 900 bytes for a clustered index. 1,700 for a nonclustered index. The maximum number of bytes in a clustered index key cannot exceed 900 in SQL Server. For a nonclustered index key, the maximum is 1700 bytes. You can define a key using variable-length columns whose maximum sizes add up to more than the limit. However, the combined sizes of the data in those columns can never exceed the limit. In a nonclustered index, you can include extra non-key columns, and they do not count against the size limit of the key. The non-key columns might help some queries perform better.

How to use distinct?

When I use DISTINCT,

Community.objects.using(self._using).distinct("community"))

I got an error,

    raise NotSupportedError('DISTINCT ON fields is not supported by this database backend')
django.db.utils.NotSupportedError: DISTINCT ON fields is not supported by this database backend

Altering Unique, nullable field to Non-nullable

This has been an on-going issue with the Django MSSQL backend. I've experienced it on more than one occasion in my own code and third party modules, the most recent occurring when attempting to integrating djangorestframework-api-key into an existing project.

Running migrations on the newly installed module results in the following failure:

django.db.utils.ProgrammingError: ('42S11', "[42S11] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]The operation failed because an index or statistics with name 'rest_framework_api_key_apikey_prefix_4e0db5f8_uniq' already exists on table 'rest_framework_api_key_apikey'. (1913) (SQLExecDirectW)")

On closer inspection one of the migrations is altering a field from unique=True, null=True to a unique nullable field. Typically, if I run into this issue in my own code, I'll just work around it and make the necessary changes. In this case, I'm concerned about future compatibility issues with the djangorestframework-api-key down the road if I make those changes myself.

Some additional info from the forked project can be found here: ESSolutions/django-mssql-backend#77

'mssql' isn't an available database backend

Hi, I created a new project and installed django, pyodbc and mssql-django and used the example string that was provided. but I am receiving the 'django.core.exceptions.ImproperlyConfigured: 'mssql' isn't an available database backend.

Could not find stored procedure 'sp_msforeachtable'

Hi,

I'm deploying a django application into a Azure App Service for linux, the database is an Azure SQL Database

Context

python 3.8
Django 3.0.13
mssql-django 1.0b1
pyodbc 4.0.30

Description

While loading data from fixtures with the python manage.py loaddata ... command, it fails with the following trace:

 Traceback (most recent call last):
   File "manage.py", line 22, in <module>
     main()
   File "manage.py", line 18, in main
     execute_from_command_line(sys.argv)
   File "/home/site/wwwroot/antenv/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
     utility.execute()
   File "/home/site/wwwroot/antenv/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
     self.fetch_command(subcommand).run_from_argv(self.argv)
   File "/home/site/wwwroot/antenv/lib/python3.8/site-packages/django/core/management/base.py", line 328, in run_from_argv
     self.execute(*args, **cmd_options)
   File "/home/site/wwwroot/antenv/lib/python3.8/site-packages/django/core/management/base.py", line 369, in execute
     output = self.handle(*args, **options)
   File "/home/site/wwwroot/antenv/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 72, in handle
     self.loaddata(fixture_labels)
   File "/home/site/wwwroot/antenv/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 111, in loaddata
     with connection.constraint_checks_disabled():
   File "/opt/python/3.8.6/lib/python3.8/contextlib.py", line 113, in __enter__
     return next(self.gen)
   File "/home/site/wwwroot/antenv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 459, in constraint_checks_disabled
     disabled = self.disable_constraint_checking()
   File "/home/site/wwwroot/antenv/lib/python3.8/site-packages/mssql/base.py", line 490, in disable_constraint_checking
     self.cursor().execute('EXEC sp_msforeachtable "ALTER TABLE ? NOCHECK CONSTRAINT ALL"')
   File "/home/site/wwwroot/antenv/lib/python3.8/site-packages/django/db/backends/utils.py", line 68, in execute
     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
   File "/home/site/wwwroot/antenv/lib/python3.8/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
     return executor(sql, params, many, context)
   File "/home/site/wwwroot/antenv/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
     return self.cursor.execute(sql, params)
   File "/home/site/wwwroot/antenv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
     raise dj_exc_value.with_traceback(traceback) from exc_value
   File "/home/site/wwwroot/antenv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
     return self.cursor.execute(sql)
   File "/home/site/wwwroot/antenv/lib/python3.8/site-packages/mssql/base.py", line 561, in execute
     return self.cursor.execute(sql, params)
 django.db.utils.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Could not find stored procedure 'sp_msforeachtable'. (2812) (SQLExecDirectW)")

This stored procedure seems to be not available in Azure SQL Database.

Adding the missing stored procedure sp_MSforeachtable and its dependency sp_MSforeach_worker does not help because those procedures are called from master database, and I don't have the credentials to add it to master.

I could not find a workaround, even with adding ATOMIC_REQUESTS = True to the Django DATABASES settings, since it is called in methods DatabaseWrapper.disable_constraint_checking and DatabaseWrapper.enable_constraint_checking if property needs_rollback is False in file mssql/base.py

I may have made something wrong since django-mssql documentation tells that it supports Azure SQL Database.

Any help would be much appreciated.

Unable to generate constraints when schema is specified in a model

Hello,

We have an implementation where schemas are specified for most of our models. But when making a constraint to any model where the schema is specified, an error is returned because of a naming issue.
The specific error is this:
django.db.utils.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Incorrect syntax near '.'. (102) (SQLExecDirectW)")
and the cause seems to be that (for db_table = "[vmd].[vmd_virtual_column]") creates an SQL which looks like this:
ALTER TABLE [vmd].[vmd_virtual_column] ADD CONSTRAINT [[vmd].[vmd_virtual_column]virtual_table_id_1400873a_fk[vmd].[vmd_virtual_table]id] FOREIGN KEY ([virtual_table_id]) REFERENCES [vmd].[vmd_virtual_table] ([id]) ()
which is invalid constraint naming in SQL Server. This has also been attempted with table naming db_table = "vmd].[vmd_virtual_column" which results in
ALTER TABLE [vmd].[vmd_virtual_column] ADD CONSTRAINT [vmd].[vmd_virtual_column_virtual_table_id_1400873a_fk
[vmd].[vmd_virtual_table]_id]].[vmd_virtual_column] FOREIGN KEY ([virtual_table_id]) REFERENCES [vmd].[vmd_virtual_table] ([id]) ()
which is also incorrect.

We have implemented a temporary solution, which currently works with db_table = "[...].[...]" way of writing but not with "...].[...", by overriding the quote_name function in schema.py with
def quote_name(self, name):
initial_re = re.sub(r'[[[a-z]+].[', '[',self.connection.ops.quote_name(name).replace(']', ''))
return re.sub(r'[[a-z]+].', '',initial_re)
which simply removes the offending parts of the name string. But a better, and more permanent, fix to this would be preferred.

Regards
Mathias Nielsen

Bad performance on Azure SQL Database

Hi, I tried using this library with a Basic 5 DTU Azure SQL Server and I'm having to wait over 5 seconds for a simple query to go through.

I read somewhere that the horrible performance was caused by the database collation being SQL_Latin1_General_CP1_CI_AS, but I'm not sure if that really is the problem.

I'm not sure if anyone else has this problem, but any help is appreciated.

Subquery with order_by results in `The ORDER BY clause is invalid in views, inline functions, derived tables, subqueries, and common table expressions, unless TOP, OFFSET or FOR XML is also specified

This is a crosspost of ESSolutions/django-mssql-backend#106

I ran into the error:

ProgrammingError: ('42000', '[42000] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]The ORDER BY clause is invalid in views, inline functions, derived tables, subqueries, and common table expressions, unless TOP, OFFSET or FOR XML is also specified. (1033) (SQLExecDirectW)')

when performing a simple API query with Django Rest Framework / Django Rest Framework Filters and I have narrowed it down to an issue when using a Subquery with an ordered queryset.

Here is a reproducible test case using the django-mssql-backend testapp with a freshly created/migrated SQL Server 2019 database:

from django.db.models import Subquery
from testapp.models import Author, Post

authors = [Author.objects.create(name="Author %d" % i for i in range(5)]
posts = [Post.objects.create(title="Post by %s" % author.name, author=author) for author in authors] 

subquery = Subquery(Author.objects.order_by("name").values("pk")) 
print(Post.objects.filter(author__in=subquery))

Which results in:

---------------------------------------------------------------------------
ProgrammingError                          Traceback (most recent call last)
C:\deploy\venvs\qatrack3\lib\site-packages\django\db\backends\utils.py in _execute(self, sql, params, *ignored_wrapper_args)
     83             else:
---> 84                 return self.cursor.execute(sql, params)
     85

E:\django-mssql-backend\sql_server\pyodbc\base.py in execute(self, sql, params)
    554         try:
--> 555             return self.cursor.execute(sql, params)
    556         except Database.Error as e:

ProgrammingError: ('42000', '[42000] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]The ORDER BY clause is invalid in views, inline functions, derived tables, subqueries, and common table expressions, unless TOP, OFFSET or FOR XML is also specified. (1033) (SQLExecDirectW)')

The above exception was the direct cause of the following exception:

ProgrammingError                          Traceback (most recent call last)
<ipython-input-4-ca584aaf3f61> in <module>
----> 1 print(Post.objects.filter(author__in=subquery))

C:\deploy\venvs\qatrack3\lib\site-packages\django\db\models\query.py in __repr__(self)
    248
    249     def __repr__(self):
--> 250         data = list(self[:REPR_OUTPUT_SIZE + 1])
    251         if len(data) > REPR_OUTPUT_SIZE:
    252             data[-1] = "...(remaining elements truncated)..."

C:\deploy\venvs\qatrack3\lib\site-packages\django\db\models\query.py in __iter__(self)
    272                - Responsible for turning the rows into model objects.
    273         """
--> 274         self._fetch_all()
    275         return iter(self._result_cache)
    276

C:\deploy\venvs\qatrack3\lib\site-packages\django\db\models\query.py in _fetch_all(self)
   1240     def _fetch_all(self):
   1241         if self._result_cache is None:
-> 1242             self._result_cache = list(self._iterable_class(self))
   1243         if self._prefetch_related_lookups and not self._prefetch_done:
   1244             self._prefetch_related_objects()

C:\deploy\venvs\qatrack3\lib\site-packages\django\db\models\query.py in __iter__(self)
     53         # Execute the query. This will also fill compiler.select, klass_info,
     54         # and annotations.
---> 55         results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
     56         select, klass_info, annotation_col_map = (compiler.select, compiler.klass_info,
     57                                                   compiler.annotation_col_map)

C:\deploy\venvs\qatrack3\lib\site-packages\django\db\models\sql\compiler.py in execute_sql(self, result_type, chunked_fetch, chunk_size)
   1140             cursor = self.connection.cursor()
   1141         try:
-> 1142             cursor.execute(sql, params)
   1143         except Exception:
   1144             # Might fail for server-side cursors (e.g. connection closed)

C:\deploy\venvs\qatrack3\lib\site-packages\django\db\backends\utils.py in execute(self, sql, params)
     65
     66     def execute(self, sql, params=None):
---> 67         return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
     68
     69     def executemany(self, sql, param_list):

C:\deploy\venvs\qatrack3\lib\site-packages\django\db\backends\utils.py in _execute_with_wrappers(self, sql, params, many, executor)
     74         for wrapper in reversed(self.db.execute_wrappers):
     75             executor = functools.partial(wrapper, executor)
---> 76         return executor(sql, params, many, context)
     77
     78     def _execute(self, sql, params, *ignored_wrapper_args):

C:\deploy\venvs\qatrack3\lib\site-packages\django\db\backends\utils.py in _execute(self, sql, params, *ignored_wrapper_args)
     82                 return self.cursor.execute(sql)
     83             else:
---> 84                 return self.cursor.execute(sql, params)
     85
     86     def _executemany(self, sql, param_list, *ignored_wrapper_args):

C:\deploy\venvs\qatrack3\lib\site-packages\django\db\utils.py in __exit__(self, exc_type, exc_value, traceback)
     87                 if dj_exc_type not in (DataError, IntegrityError):
     88                     self.wrapper.errors_occurred = True
---> 89                 raise dj_exc_value.with_traceback(traceback) from exc_value
     90
     91     def __call__(self, func):

C:\deploy\venvs\qatrack3\lib\site-packages\django\db\backends\utils.py in _execute(self, sql, params, *ignored_wrapper_args)
     82                 return self.cursor.execute(sql)
     83             else:
---> 84                 return self.cursor.execute(sql, params)
     85
     86     def _executemany(self, sql, param_list, *ignored_wrapper_args):

E:\django-mssql-backend\sql_server\pyodbc\base.py in execute(self, sql, params)
    553         self.last_params = params
    554         try:
--> 555             return self.cursor.execute(sql, params)
    556         except Database.Error as e:
    557             self.connection._on_error(e)

ProgrammingError: ('42000', '[42000] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]The ORDER BY clause is invalid in views, inline functions, derived tables, subqueries, and common table expressions, unless TOP, OFFSET or FOR XML is also specified. (1033) (SQLExecDirectW)')

Parameter limitation on prefetch_related

This is an issue that of course also exists in the ESSolution library that this repo is forked from.

When using prefetch_related to optimize queries for many-to-one and many-to-many relationships, Django generates an IN clause containing all of the primary keys that a M:1 model's foreign key should be filtered on, thus solving the N+1 problem.

However, MS SQL Server has a 2100 parameter limit, which causes the rdbms to fail when trying to prefetch objects with more than 2100 possible foreign keys. It would be awesome if the driver could add support for handling this stark limitation.

The official MS SQL Server IN clause documentation recommends creating a temp table and then joining over that. Without being too familiar with Django ORM internals, I don't know how feasible it would be to override the prefetch_related as_sql query to implement this solution.

There is a related issue in the ESSolutions/django-mssql-backend repo, where a suggested workaround is to override Django's In lookup class to basically pass in a single giant parameter string and then have SQL Server split it via {lhs} IN ( SELECT * FROM STRING_SPLIT(CONVERT(nvarchar(max), %s),','))). While this seems to work when filtering a queryset explicitly with __in, the overridden lookup class is not used for prefetch_related calls.

A committed solution would be awesome, but I'd also love any advice you may have as a temporary workaround.

ACTION REQUIRED: Microsoft needs this private repository to complete compliance info

There are open compliance tasks that need to be reviewed for your mssql-django repo.

Action required: 4 compliance tasks

To bring this repository to the standard required for 2021, we require administrators of this and all Microsoft GitHub repositories to complete a small set of tasks within the next 60 days. This is critical work to ensure the compliance and security of your microsoft GitHub organization.

Please take a few minutes to complete the tasks at: https://repos.opensource.microsoft.com/orgs/microsoft/repos/mssql-django/compliance

  • The GitHub AE (GitHub inside Microsoft) migration survey has not been completed for this private repository
  • No Service Tree mapping has been set for this repo. If this team does not use Service Tree, they can also opt-out of providing Service Tree data in the Compliance tab.
  • No repository maintainers are set. The Open Source Maintainers are the decision-makers and actionable owners of the repository, irrespective of administrator permission grants on GitHub.
  • Classification of the repository as production/non-production is missing in the Compliance tab.

You can close this work item once you have completed the compliance tasks, or it will automatically close within a day of taking action.

If you no longer need this repository, it might be quickest to delete the repo, too.

GitHub inside Microsoft program information

More information about GitHub inside Microsoft and the new GitHub AE product can be found at https://aka.ms/gim or by contacting [email protected]

FYI: current admins at Microsoft include @kenvanhyning, @rene-ye, @yitam, @v-makouz, @LuisBosquez, @v-mabarw, @ajlam, @v-chojas, @johnnypham, @cheenamalhotra, @ulvii, @saurabh500, @lilgreenbird

[FA001] [Microsoft][ODBC Driver 17 for SQL Server]Cannot use Authentication option with Integrated Security option

Using:

  • Python 3.8
  • Windows 10
  • mssql-django 1.0b1

In setting up an existing Django application to use SQL server on Azure through AppService, I've got a test databse set up and i'm able to authenticate through Azure when using SQL Management Studio.

Following the example set in https://github.com/microsoft/mssql-django/wiki/Azure-AD-Authentication, for settings.py I have:

 DATABASES = {
        'default': {
            'ENGINE': 'mssql',
            'NAME': 'test',
            'HOST': 'example.database.windows.net',
            'PORT': '',
            'USER': '[email protected]',

            'OPTIONS': {
                'driver': 'ODBC Driver 17 for SQL Server',
                "extra_params": 'Authentication=ActiveDirectoryInteractive',
            },
        },
    }

when running django with python manage.py migrate I get the following:

[FA003] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Password option must not be specified, if Authentication option is 'ActiveDirectoryInteractive'. (0) (SQLDriverConnect); [FA003] [Microsoft][ODBC Driver 17 for SQL Server]Invalid connection string attribute (0)

Okay sure. the recent changes in b41e8b4 haven't been released/published yet.
I thought i'd be clever and set the UID in the extra_params so that a stray PWD doesn't get concatenated.

mssql-django/mssql/base.py

Lines 287 to 289 in 00bbdca

if user:
cstr_parts['UID'] = user
cstr_parts['PWD'] = password

settings.py now has the following:

 DATABASES = {
        'default': {
            'ENGINE': 'mssql',
            'NAME': 'test',
            'HOST': 'example.database.windows.net',
            'PORT': '',

            'OPTIONS': {
                'driver': 'ODBC Driver 17 for SQL Server',
                "extra_params": '[email protected];Authentication=ActiveDirectoryInteractive',
            },
        },
    }

But now I get the following issue.

[FA001] [Microsoft][ODBC Driver 17 for SQL Server]Cannot use Authentication option with Integrated Security option. (0) (SQLDriverConnect); [FA001] [Microsoft][ODBC Driver 17 for SQL Server]Invalid connection string attribute (0)

Is there something else I need to do for Azure authentication to work correctly?

Migration issue - Django 3.2.3

Hi,

Appreciate this is a Beta product; I'm trying to get the SQL script using "sqlmigrate" for a brand new, completely blank Django project with just the basic admin tools.

I'm getting this output, which indicates a failure in the mssql Schema.py file:

python3 manage.py sqlmigrate admin 0001_initial
Traceback (most recent call last):
File "manage.py", line 22, in
main()
File "manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "/home/pmason/.local/lib/python3.8/site-packages/django/core/management/init.py", line 419, in execute_from_command_line
utility.execute()
File "/home/pmason/.local/lib/python3.8/site-packages/django/core/management/init.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/pmason/.local/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/pmason/.local/lib/python3.8/site-packages/django/core/management/commands/sqlmigrate.py", line 29, in execute
return super().execute(*args, **options)
File "/home/pmason/.local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/home/pmason/.local/lib/python3.8/site-packages/django/core/management/commands/sqlmigrate.py", line 65, in handle
sql_statements = loader.collect_sql(plan)
File "/home/pmason/.local/lib/python3.8/site-packages/django/db/migrations/loader.py", line 351, in collect_sql
state = migration.unapply(state, schema_editor, collect_sql=True)
File "/home/pmason/.local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 118, in exit
self.execute(sql)
File "/home/pmason/.local/lib/python3.8/site-packages/mssql/schema.py", line 856, in execute
sql = str(sql)
File "/home/pmason/.local/lib/python3.8/site-packages/django/db/backends/ddl_references.py", line 201, in str
return self.template % self.parts
KeyError: 'include'

When I try to run the migration I get the same sort of error:

python3 manage.py migrate
Operations to perform:
Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
Applying contenttypes.0001_initial...Traceback (most recent call last):
File "manage.py", line 22, in
main()
File "manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "/home/pmason/.local/lib/python3.8/site-packages/django/core/management/init.py", line 419, in execute_from_command_line
utility.execute()
File "/home/pmason/.local/lib/python3.8/site-packages/django/core/management/init.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/pmason/.local/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/pmason/.local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/home/pmason/.local/lib/python3.8/site-packages/django/core/management/base.py", line 89, in wrapped
res = handle_func(*args, **kwargs)
File "/home/pmason/.local/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 244, in handle
post_migrate_state = executor.migrate(
File "/home/pmason/.local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/home/pmason/.local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/home/pmason/.local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
state = migration.apply(state, schema_editor)
File "/home/pmason/.local/lib/python3.8/site-packages/django/db/migrations/migration.py", line 126, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/home/pmason/.local/lib/python3.8/site-packages/django/db/migrations/operations/models.py", line 528, in database_forwards
alter_together(
File "/home/pmason/.local/lib/python3.8/site-packages/mssql/schema.py", line 167, in alter_unique_together
self.execute(sql)
File "/home/pmason/.local/lib/python3.8/site-packages/mssql/schema.py", line 856, in execute
sql = str(sql)
File "/home/pmason/.local/lib/python3.8/site-packages/django/db/backends/ddl_references.py", line 201, in str
return self.template % self.parts
KeyError: 'include'

I don't want to allow Django to control the DB via migration, because it's an existing operational DB with an MS Access front end. But I need the scripts for the admin functionality.

Something is going wrong in the interpretation of the models used in the admin tools.

In the meantime I'll reverse engineer a SQLite DB and see if I can connect with that.

Some indexes dropped in migration are not recreated

Problem

In the following cases (possibly more) any indexes on the column are dropped, but aren't recreated afterwards:
(a) when a column is made null-able (i.e. adding null=True)
(b) when a column is renamed
(c) when a table is renamed

This is quite a major issue because it is silently leaving the database in the wrong state - the migration doesn't fail, but now certain columns with db_index=True won't actually have an index which could cause very slow queries.

Cause

As far as I can see this is a triple-regression introduced (before this fork was created) by 2e60754 (ESSolutions/django-mssql-backend#24).

That commit added new calls to _delete_indexes in 2 places within _alter_field causing the first two cases listed above:

(a) ESSolutions/django-mssql-backend@2e6075456a#diff-e4c1520d8b49ccbf46382bd2eed4e740R400
(b) ESSolutions/django-mssql-backend@2e6075456a#diff-e4c1520d8b49ccbf46382bd2eed4e740R334-R335

and added an explicit index deletion here in alter_db_table before renaming a table causing the third case:

(c) ESSolutions/django-mssql-backend@2e6075456a#diff-e4c1520d8b49ccbf46382bd2eed4e740R222-R225

but in none of those cases is the index re-instated afterwards, even if the field still has db_index=True. Index restoration only happens in certain specific cases (e.g. type change / removal of nullability) which doesn't include the above 3 cases.

Reproduction

I've added 3 tests in #49 which fail due to the bugs described above, but pass if run on older versions like django-mssql-backend v2.4.2 (before ESSolutions/django-mssql-backend#24 was merged).

History

I previously filed this on the project from which this was forked ESSolutions/django-mssql-backend#58 - given there was no feedback to my proposed solution there, I hope it's ok to file an equivalent issue here. Hopefully this more official project might be able to provide some insight. This was my comment (the only one on that issue) re possible fixes:

Considering how to solve this, one approach would be to expand the condition under # Restore an index, or to start explicitly keeping track of which indexes have been dropped and need re-creating.

However before that we should ask: is it actually necessary for all indexes to be dropped in these 3 cases?

If it can be avoided then it will save wasting time during migration of a large table. MS SQL Server seems to allow those 3 operations to occur while the index is there (it didn't explode in those cases using v2.4.2).

@OskarPersson do you know what the reasons were for adding all these "delete index before doing X" in 2e60754?

NULLS LAST added to query in django 3.1

I'm testing upgrading from django 3.0 to 3.1, using the latest backend, version 1.0b1. When using the nulls_last flag in an order_by, it appends a bit of SQL invalid in MSSQL (NULLS LAST)

The following use of nulls_last:

def other_runs(self):
    return (
        Module.objects
        .filter(url=self.url, division=self.division)
        .exclude(id=self.id)
        .order_by(F('start_date').desc(nulls_last=True))
    )

Produces the query:

SELECT TOP 21 [module].[id], ... FROM [module] WHERE ([module].[division] = %s AND [module].[url] = %s 
AND NOT ([module].[id] = %s)) ORDER BY CASE WHEN [module].[start_date] IS NULL THEN 1 ELSE 0 END, 
[module].[start_date] DESC NULLS LAST

So both the MSSQL workaround and the ANSI SQL standard are there.

Naturally, this throws the error [42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Incorrect syntax near 'NULLS'. (102) (SQLExecDirectW)

Package doesn't report error correctly if Email service credentials are wrong

Problem

The gmail user account that I was using to send email was subsequently blocked, which caused confusing error message

On db record creation I trigger a signal to "send.mail()", which adds entry to post_office_email table without problems.
However when I run "python manage.py send_queued_mail", it throws the following error:

pyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]At least one of the result expressions in a CASE specification must be an expression other than the NULL constant. (8133) (SQLExecDirectW)')

which happens at: "/usr/local/lib/python3.9/site-packages/post_office/mail.py", line 308, in _send_bulk

Cause

I tracked down an issue to see what happens. If Email service credentials are not set correctly or account is blocked, package reports error inside failed_emails list as follows:

the print of failed_emails at line 295:
[(<Email: ['[email protected]']>, SMTPAuthenticationError(535, b'5.7.8 Username and Password not accepted. Learn more at\n5.7.8 https://support.google.com/mail/?p=BadCredentials k12sm1104861edq.59 - gsmtp')), (<Email: ['[email protected]']>, SMTPAuthenticationError(535, b'5.7.8 Username and Password not accepted. Learn more at\n5.7.8 https://support.google.com/mail/?p=BadCredentials a15sm1279576edr.2 - gsmtp'))]
preee emails [<Email: ['[email protected]']>, <Email: ['[email protected]']>]

which then subsequently causes above error at line 308:
Email.objects.bulk_update(emails_failed,["status","scheduled_time","number_of_retries"]),

Setup

Django==3.2.4
django-environ-2==2.1.0
djangorestframework==3.12.4
pyodbc==4.0.32
django-mysql==4.0.0
mssql-django==1.0.0
celery==5.1.2
redis==3.5.3
django-post_office==3.5.3

Kind regards

License changed from BSD to MIT without permission

Filing this as a separate issue so it doesn't get lost in a comment thread on the repo from which this was forked:
ESSolutions/django-mssql-backend#91 (comment)

I noticed the license of the fork has been changed from BSD 3-Clause to MIT - may I ask from whom you obtained permission to alter the license?

History of LICENSE through chain of forks (BSD, with varying authors): django-pyodbc, django-pyodbc-azure, django-mssql-backend

Thank you @vwarchu for the response:

@sparrowt - thanks for pointing this out, we'll put this through a legal review process to evaluate reverting to the BSD 3-Clause.

I trust this ticket can serve to track resolving this issue.

As well as re-instating the license it would be appropriate (if not required - I'll let you talk to the lawyers!) to retain the Copyright lines from each fork in the chain, especially the original "django-pyodbc developers" one, to acknowledge their contributions over many years - rather than just replacing them all with "Copyright (c) Microsoft Corporation" on its own which I believe is incorrect.

Azure SQL version/features not checked correctly

Our app uses Azure SQL as database. We encountered a problem while trying to use JSON field and found out that it is supported by Azure SQL but the check is not implemented correctly.

def supports_json_field(self):

It seems that SQL Server and Azure SQL have different version numbers.

SELECT CAST(SERVERPROPERTY('ProductVersion') AS varchar)

reports 12 on Azure but in reality it supports more features.

def sql_server_version(self, _known_versions={}):

There seems to be code that handles version checks for Azure SQL

def to_azure_sql_db(self, _known_azures={}):

but it's not used correctly when declaring features.

model db_table

Hi!

I'm create a model, but it does not respect the name that I assign from the meta method.

models.py

class TeenusProducts(models.Model):
    class Meta:
        db_table = 'teenus_products'

python manage.py sqlmigrate users 0001

--
-- Create model TeenusProducts
--
CREATE TABLE [users_teenusproducts]

I expected table name users_teenus_products

Thanks!

Migrations not working

I was using mariadb connection but had to migrate to MSSQL
So I deleted all migrations folder stuff and remade the migrate, makemigrations...

First problem I found is that some columns that by default allowed Null values on MySQL now it seens the default is to not allow Null values on SQLSERVER...

The second problem is that altering a table is not working on DB level
I have a table created and lately tried to change and migrate a specif column to unique=True, made and applied the migrations but when runing the query directly on a SQL Client the migration did not change the Table structure and I still was able to insert a duplicate value on the column ...
So I had to run directly

ALTER TABLE table ADD UNIQUE (column);

Ability to run flexibly run tests on different database drivers/backends

The original version of django-mssql-backend utilised dj_database_url in order to provide a flexible way of specifying the database configuration when running tests, by way of environment variables (https://github.com/ESSolutions/django-mssql-backend/blob/master/testapp/settings.py#L1-L6). With this fork, that flexibility has been replaced with hard-coded settings in https://github.com/microsoft/mssql-django/blob/dev/testapp/settings.py, forcing one to edit that file to change the user/password/host/port/driver settings.

As a Mac/Linux user I'm using Docker and FreeTDS as the simplest/most convenient way in which to run up and connect to a test SQL Server on my OS (I recall there being issues installing the Microsoft ODBC driver) and do so on a custom port in order to be able to run tests for this project. The original setup in django-mssql-backend means that this was possible:

export DATABASE_URL='mssql://user:password@localhost:9999?engine=FreeTDS&host_is_server=True'
python manage.py test

whereas now in order to submit and test #9 I had to manually edit settings.py and be careful not to commit that change.

Is this or some level of flexibility for test execution able to be reinstated?

bulk_update max batch_size reduced? (1.0b1->1.0) (COUNT field incorrect or syntax error)

After upgrading to Django 3.2.8 and mssql-django ==1.0, existing bulk_update commands started failing with the error below.
Before I could use batch_size of like 5000 with no problem, now I get the below error unless the batch_size is set to like 10.

When reverting back mssql-django==1.0b1, (even with Django 3.2.8) the issue goes away.

Any information is appreciated. Thanks!

Traceback:

File "E:\myproject\test_project\test\management\commands\parse_file.py", line 379, in results_to_sql
    Machine.objects.bulk_update(
  File "C:\Program Files\Python39\lib\site-packages\django\db\models\manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "C:\Program Files\Python39\lib\site-packages\django\db\models\query.py", line 568, in bulk_update
    self.filter(pk__in=pks).update(**update_kwargs)
  File "C:\Program Files\Python39\lib\site-packages\django\db\models\query.py", line 783, in update
    rows = query.get_compiler(self.db).execute_sql(CURSOR)
  File "C:\Program Files\Python39\lib\site-packages\django\db\models\sql\compiler.py", line 1559, in execute_sql
    cursor = super().execute_sql(result_type)
  File "C:\Program Files\Python39\lib\site-packages\django\db\models\sql\compiler.py", line 1175, in execute_sql
    cursor.execute(sql, params)
  File "C:\Program Files\Python39\lib\site-packages\django\db\backends\utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "C:\Program Files\Python39\lib\site-packages\django\db\backends\utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "C:\Program Files\Python39\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "C:\Program Files\Python39\lib\site-packages\django\db\utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "C:\Program Files\Python39\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "C:\Program Files\Python39\lib\site-packages\mssql\base.py", line 567, in execute
    return self.cursor.execute(sql, params)
django.db.utils.Error: ('07002', '[07002] [Microsoft][ODBC Driver 17 for SQL Server]COUNT field incorrect or syntax error (0) (SQLExecDirectW)')

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.