aaugustin / django-sequences Goto Github PK

View Code? Open in Web Editor NEW

222.0 222.0 40.0 127 KB

Generate gapless sequences of integer values.

License: BSD 3-Clause "New" or "Revised" License

Makefile 0.80% Python 99.20%

django-sequences's People

Contributors

Stargazers

Watchers

django-sequences's Issues

Separate sequence property specifiers

When using

get_next_value("orders", initial_value=100, reset_value=500)

I can easily tell what this function does and what values to expect, but if this sequence name was used in multiple files as

get_next_value("orders")

it wouldn't be obvious or it would be hard to figure out what the initial_value or the rest_value is.

My proposal is doing one of:

allowing a declarative class usage:

orders_sequence = DjangoSequence("orders", initial_value=100, reset_value=500)

get_next_value(orders_sequence)

enum-like usage:

class DjangoSequences:
    ORDERS = DjangoSequence("orders", initial_value=100, reset_value=500)

get_next_value(DjangoSequences.ORDERS)

My current usage is like the following:

orders_sequence = {"sequence_name": "orders", "initial_value": 100, "reset_value": 500}

get_next_value(**orders_sequence)

Help elaborate the background of the problem?

Hi, this is a great extension and I love it. Just want to ask though:

If a transaction inserts a row and then is rolled back, the sequence counter isn't rolled back for performance reasons, creating a gap in primary keys.

This can cause compliance issues for some use cases such as accounting.

This risk isn't well known. Since most transactions succeed, values look sequential. Gaps will only be revealed by audits.

Is this only applies to PostgreSQL or it applies to Django with any DB backend in general?

Thanks!

Atomic block behaviour

This would be a question actually, but it will clarify the documentation for me, so I would be grateful if you could elaborate on this. What would be the assumed behaviour in the following scenario:

with transaction.atomic():
    # Code A
    with transaction.atomic():
        x = get_next_value('test')
        # Code B
    # Code C

Thread is invoking Code B instructions - are other threads blocked waiting (on which line?) until it ends to run Code B? What about Code A?
Thread is invoking Code C instructions. Are other threads blocked waiting (on which line?) until it ends to run Code C? Code B? Code A?

I know the library guarantees than get_next_value always return concurrent gapless values. But for example if I assign this value to an Invoice number in Code B and something goes wrong, there may be a chance that the next transaction would get the next number (in case execution of Code B does not block other threads).

Thank in advance for any answer.

Python 2.x support

It is shame that Python 2 support has been dropped. Is there any particular reason for that?

django-sequences version 2.4 is not supporting

django-sequences version 2.4 showing error in installed apps.
get_next_value() giving error
relation "sequences_sequence" does not exist
LINE 2: INSERT INTO sequences_sequence (name, last)

Not support for python 2?

When using it in python2.7 enviroment. met this problem:
File "/usr/local/lib/python2.7/dist-packages/sequences/__init__.py", line 6 *, nowait=False, using=None): ^ SyntaxError: invalid syntax

reset value does not work with PostgresDB

number = get_next_value('transact_number', initial_value=1, reset_value=10)

 [get_next_value('t') for i in range(100)]

The code does not reset the values

Expand get_next_value for multiple sequences

Hi, we are using django-sequences to generate gapless sequence numbers for objects. In some requests we'll create multiple objects in the same sequence, which will all get a sequence value. Currently, we have to call get_next_value() multiple times, as the increment is hard-coded in the upsert query. This leads to an effective N+1 problem, which can harm performance in case of many sequenced objects.

I'd suggest expanding on get_next_value() with a get_next_values() that allows you to retrieve multiple sequences to distribute in a single query. Happy to create a PR for this as well.

Detecting duplicate sequence name

Is there an established way to ensure a "new" sequence isn't created with the same sequence name? I'm mainly thinking about the case of a large code base with a lot of contributors.

Feature request: Get current value

This is a great package and thank you for providing it. However, I have one feature request: what about a function to get the current value of a sequence? Sometimes I want to see where the sequence is at (in my case so I can grab the maximum). Thank you for your consideration.

About daily sequence

Hello, I would like to generate a sequence which renews everyday. Is there any function to do so?
e.g. yyyymmddNN. where NN will start from 01 for each date. Thanks.

PositiveBigIntegerField

Hi, thanks for great package. I would like to ask if it would be possible to change PositiveIntegerField to PositiveBigIntegerField? Because I have sequence where 2147483647 number is too small.

django-sequences/src/sequences/models.py

Line 13 in d58f4e6

last = models.PositiveIntegerField(

New release

Can you please make a new release so we can get rid of the deprecation warning fixed in 631b00d ?

Thanks.

Timeouts on Parallel Requests

Hi,
We're using django 1.10 with Postgresql 9.2 and python 3.5
We have decided to use your package since we need field auto-incremental that is not PK (for some model), and it should be unique per other model.
The sequences table is within our models DB.

It seemed like the solution was good, until we started getting parallel requests. In that case - looks like the DB got locked for too long and we started getting timeouts (We are configured to 10 seconds timeout by our api gateway (Apigee)

Is that behavior expected or the package should handle those cases?
Thanks.

Initial Django Migration Issue

Hi there, I performed a database dump using pg_dump with version 2.7 installed however the new migration file not applied, upon moving to a new instance of my app and restoring the db, the initial migration pops up with:

Running migration against app-debug
Traceback (most recent call last):
  File "/app/wholesale/manage.py", line 23, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 440, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 402, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 448, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 96, in wrapped
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 117, in handle
    executor.loader.check_consistent_history(connection)
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/loader.py", line 327, in check_consistent_history
    raise InconsistentMigrationHistory(
django.db.migrations.exceptions.InconsistentMigrationHistory: Migration table.0025_auto_20190212_1447 is applied before its dependency sequences.0002_alter_sequence_last on database 'default'.

In order to get around this, I reverted back to 2.6 and performed the initial migration, in order to use the new version/migration file is there something I need to do? or is this only if you apply the migrations to a new project? and not one with existing positiveIntegerFields?

Deprecation warning: django.utils.translation.ugettext_lazy() is deprecated

When running tests with deprecation warnings, I see the following warning:

python -Wa ./manage.py test 
site-packages/sequences/models.py:19: RemovedInDjango40Warning: django.utils.translation.ugettext_lazy() is deprecated in favor of django.utils.translation.gettext_lazy().
  verbose_name_plural = _("sequences")

Thanks for creating and maintaining this package.

Unique sequence with Model.objects.bulk_create([...])

Hey @aaugustin @rmoch - I'm working on a project that needs to support some unfortunate functionality in a backwards compatible way in one of our tables. I'm using PostgreSQL 9.5.2, Django 1.9, and Python 2.7.11.

The model in need of the sequencing looks like this:

class Profile(models.Model):
    uid = models.AutoField(primary_key=True)
    unique_archaic_code = models.CharField(max_length=12L, unique=True)

We cannot change the structure of the Profile model - any performance enhancements we make on it must be backwards compatible. Otherwise I would say to heck with the unique_archaic_code system we have set up and make something more modern.

unique_archaic_code follows a specific alpha-numeric pattern. I generate values for it in the Python layer at the moment, but I want to abstract this out into a PostgreSQL SEQUENCE for obvious performance reasons, so we can potentially use a Django bulk_create statement that will make bulk creation of Profile objects with proper unique_archaic_codes possible.

unique_archaic_code must stay, and must continue to be filled with values that follow the old pattern defined in the Python layer - I want to replace that value filling with the sequencing your package appears to support.

Does this package support that functionality currently?

Rename master branch to main

@adamchainz - creating this issue as a heads-up for you.

To update your local copy, GitHub recommends:

git branch -m master main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a

Please close it when you've seen it!

Document usage with different isolation levels

I was getting OperationalError: could not serialize access due to concurrent update when I was testing two concurrent requests with django-sequences. Now I realized, that I get those only if I have set 'isolation_level': ISOLATION_LEVEL_REPEATABLE_READ.

I would appreciate, if usage of django-sequences with different isolation levels was discussed in the documentation.

Calling get_next_value() in multiple paralel requests can cause huge database lags

I am using django-sequences with django-plans to determine invoice sequence numbers for recurring payments.
Recurring payments renewal runs every night and currently we are processing about PayU 20 payments every night.
PayU sends payment confirmation request to the server, in which the get_next_value() is called. The requests for those payments are coming all in one or two minutes.
Sometimes those requests hit 30 second timeout, with most of the time spend in Postgres sequences_sequence insert.

I expect, that the problem might be caused the fact, that the whole Django request runs in a transaction and although I have the get_next_value() call in with transaction.atomic(): block, that it still blocks the whole sequence update until whole request is processed.

I will have to investigate this issue much deeper.

Here are some logs from the time of the long requests:


2022-02-08T01:31:37+00:00 app postgres.2123 - - [AQUA] [45-1] sql_error_code = 00000 LOG: process 2123 acquired ShareLock on transaction 226740799 after 2701.729 ms
2022-02-08T01:31:37+00:00 app postgres.2117 - - [AQUA] [32-1] sql_error_code = 00000 LOG: process 2117 acquired ShareLock on transaction 226740799 after 2701.506 ms
2022-02-08T01:31:37+00:00 app postgres.2117 - - [AQUA] [32-2] sql_error_code = 00000 CONTEXT: while inserting index tuple (1,57) in relation "sequences_sequence"
2022-02-08T01:31:37+00:00 app postgres.2117 - - [AQUA] [32-3] sql_error_code = 00000 STATEMENT:
2022-02-08T01:31:37+00:00 app postgres.2117 - - [AQUA] [32-4] INSERT INTO "sequences_sequence" (name, last)
2022-02-08T01:31:37+00:00 app postgres.2117 - - [AQUA] [32-5] VALUES ('invoice_numbers_1_call_2022_2', 448)
2022-02-08T01:31:37+00:00 app postgres.2117 - - [AQUA] [32-6] ON CONFLICT (name)
2022-02-08T01:31:37+00:00 app postgres.2117 - - [AQUA] [32-7] DO UPDATE SET last = "sequences_sequence".last + 1
2022-02-08T01:31:37+00:00 app postgres.2117 - - [AQUA] [32-8] RETURNING last;
2022-02-08T01:31:37+00:00 app postgres.2117 - - [AQUA] [32-9]
2022-02-08T01:31:37+00:00 app postgres.2116 - - [AQUA] [35-1] sql_error_code = 00000 LOG: process 2116 acquired ShareLock on transaction 226740799 after 2701.447 ms
2022-02-08T01:31:37+00:00 app postgres.2116 - - [AQUA] [35-2] sql_error_code = 00000 CONTEXT: while inserting index tuple (1,57) in relation "sequences_sequence"
2022-02-08T01:31:37+00:00 app postgres.2116 - - [AQUA] [35-3] sql_error_code = 00000 STATEMENT:
2022-02-08T01:31:37+00:00 app postgres.2116 - - [AQUA] [35-4] INSERT INTO "sequences_sequence" (name, last)
2022-02-08T01:31:37+00:00 app postgres.2116 - - [AQUA] [35-5] VALUES ('invoice_numbers_1_call_2022_2', 450)
2022-02-08T01:31:37+00:00 app postgres.2116 - - [AQUA] [35-6] ON CONFLICT (name)
2022-02-08T01:31:37+00:00 app postgres.2116 - - [AQUA] [35-7] DO UPDATE SET last = "sequences_sequence".last + 1
2022-02-08T01:31:37+00:00 app postgres.2116 - - [AQUA] [35-8] RETURNING last;
2022-02-08T01:31:37+00:00 app postgres.2116 - - [AQUA] [35-9]
2022-02-08T01:31:37+00:00 app postgres.2122 - - [AQUA] [42-1] sql_error_code = 00000 LOG: process 2122 acquired ShareLock on transaction 226740799 after 2701.444 ms
2022-02-08T01:31:37+00:00 app postgres.2122 - - [AQUA] [42-2] sql_error_code = 00000 CONTEXT: while inserting index tuple (1,57) in relation "sequences_sequence"
2022-02-08T01:31:37+00:00 app postgres.2122 - - [AQUA] [42-3] sql_error_code = 00000 STATEMENT:
2022-02-08T01:31:37+00:00 app postgres.2122 - - [AQUA] [42-4] INSERT INTO "sequences_sequence" (name, last)
2022-02-08T01:31:37+00:00 app postgres.2122 - - [AQUA] [42-5] VALUES ('invoice_numbers_1_call_2022_2', 450)
2022-02-08T01:31:37+00:00 app postgres.2122 - - [AQUA] [42-6] ON CONFLICT (name)
2022-02-08T01:31:37+00:00 app postgres.2122 - - [AQUA] [42-7] DO UPDATE SET last = "sequences_sequence".last + 1
2022-02-08T01:31:37+00:00 app postgres.2122 - - [AQUA] [42-8] RETURNING last;

and also:

2022-02-08T01:31:38+00:00 app postgres.2228 - - [AQUA] [29-1] sql_error_code = 00000 LOG: process 2228 still waiting for ShareLock on transaction 226740808 after 1001.400 ms
2022-02-08T01:31:38+00:00 app postgres.2235 - - [AQUA] [24-1] sql_error_code = 00000 LOG: process 2235 still waiting for ShareLock on transaction 226740808 after 1000.967 ms
2022-02-08T01:31:38+00:00 app postgres.2234 - - [AQUA] [23-1] sql_error_code = 00000 LOG: process 2234 still waiting for ShareLock on transaction 226740808 after 1001.173 ms
2022-02-08T01:31:38+00:00 app postgres.2236 - - [AQUA] [21-2] sql_error_code = 00000 DETAIL: Process holding the lock: 2116. Wait queue: 2197, 2236, 2228, 2117, 2070, 2122, 2239, 2071, 2123, 2234, 2235.
2022-02-08T01:31:38+00:00 app postgres.2122 - - [AQUA] [43-2] sql_error_code = 00000 DETAIL: Process holding the lock: 2116. Wait queue: 2197, 2236, 2228, 2117, 2070, 2122, 2239, 2071, 2123, 2234, 2235.
2022-02-08T01:31:38+00:00 app postgres.2236 - - [AQUA] [21-3] sql_error_code = 00000 CONTEXT: while inserting index tuple (1,58) in relation "sequences_sequence"
2022-02-08T01:31:38+00:00 app postgres.2234 - - [AQUA] [23-2] sql_error_code = 00000 DETAIL: Process holding the lock: 2116. Wait queue: 2197, 2236, 2228, 2117, 2070, 2122, 2239, 2071, 2123, 2234, 2235.
2022-02-08T01:31:38+00:00 app postgres.2228 - - [AQUA] [29-3] sql_error_code = 00000 CONTEXT: while locking tuple (1,58) in relation "sequences_sequence"
2022-02-08T01:31:38+00:00 app postgres.2122 - - [AQUA] [43-3] sql_error_code = 00000 CONTEXT: while inserting index tuple (1,58) in relation "sequences_sequence"
2022-02-08T01:31:38+00:00 app postgres.2236 - - [AQUA] [21-4] sql_error_code = 00000 STATEMENT:
2022-02-08T01:31:38+00:00 app postgres.2234 - - [AQUA] [23-3] sql_error_code = 00000 CONTEXT: while inserting index tuple (1,58) in relation "sequences_sequence"
2022-02-08T01:31:38+00:00 app postgres.2071 - - [AQUA] [50-2] sql_error_code = 00000 DETAIL: Process holding the lock: 2116. Wait queue: 2197, 2236, 2228, 2117, 2070, 2122, 2239, 2071, 2123, 2234, 2235.
2022-02-08T01:31:38+00:00 app postgres.2117 - - [AQUA] [33-1] sql_error_code = 00000 LOG: process 2117 still waiting for ShareLock on transaction 226740808 after 1000.443 ms

  File "/usr/local/lib/python3.5/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/local/lib/python3.5/site-packages/sequences/__init__.py", line 29
    ):
    ^
SyntaxError: invalid syntax

dropping old sequences

Currently to drop outdated/unwanted sequences from the database, you need to use the private api.
(eg via orm operations on sequences.models.Sequence)

It would be nice if this functionality was exposed in the public api.

aaugustin / django-sequences Goto Github PK

django-sequences's People

Contributors

Stargazers

Watchers

Forkers

django-sequences's Issues

Recommend Projects

Recommend Topics

Recommend Org