rvinzent / django-dynamic-models Goto Github PK

Requesting feature to support type ArrayField to allow the user to store data in an array.

Here is my scenario:

I want to create task table for every user, since each user may add this own dynamic fields.

def create_task_list(request):
    schema = ModelSchema.objects.create(name=f"task_{request.user.id}")
    # create common fields
    FieldSchema.objects.create(model_schema=schema, name="status", data_type="character")
    ...also create common fields like create_datetime, user, update_datetime...
    # create user fields
    for field in request.POST['fields']:
        ...

How can I create an abstract model like

class TaskModel:
    status = models.CharField()
    create_datetime = models.DateTimeField(auto_now_add=True)
    update_datetime = models.DateTimeField(auto_now=True)
    user = models.ForeignKey(User, on_delete=models.CASCADE)

and use this model as a baseclass to create dynamic model like:

def create_task_list(request):
    schema = ModelSchema.objects.create(name=f"task_{request.user.id}", base_model=TaskModel)

    # create user fields
    for field in request.POST['fields']:
        ...

Hi,
I have installed django-dynamic-models and I'm following the documentation.

When I create a new object with the command :
car_schema = ModelSchema.objects.create(name='car')

The command return an error :

no such table: dynamic_models_modelfieldschema

Have you ever encountered this error ?

Hello @rvinzent, I've been trying to work with your module but every time I migrate I get greeted with a wall of errors and a keyerror, I've tried all kinds of things and starting fresh apps but I keep getting the same error.

KeyError: ('dynamic_models', 'modelschema')

Do you have any idea how I can avoid this error?

The relationships between these models should be simplified. Care should be taken to disallow fields with the same name to appear on multiple model schema.

• add python 3.9 to tox and ensure all tests pass

Issue location: /dynamic_models/models.py:254
That is:

class ModelFieldSchema(GenericModel, GenericField):
objects = ModelFieldSchemaManager()
null = models.BooleanField(default=False) # issue1
unique = models.BooleanField(default=False) # issue2
max_length = models.PositiveIntegerField(null=True)

Problem description:

'null' and 'unique' are keywords in mysql grammar.
In our strict DBA system, 'null' and 'unique' are forbidden to be column names.

Suggestion:
code to be modified as follows.

class ModelFieldSchema(GenericModel, GenericField):
objects = ModelFieldSchemaManager()
null = models.BooleanField(default=False, db_column='is_null') # issue1
unique = models.BooleanField(default=False, db_column='is_unique') # issue2
max_length = models.PositiveIntegerField(null=True)

Steps to Reproduce

model_schema = ModelSchema.objects.create(name="SimpleModel")
related_model_schema = ModelSchema.objects.create(name="RelatedModel")

FieldSchema.objects.create(
    name="related",
    model_schema=model_schema,
    class_name="django.db.models.ForeignKey",
        kwargs={
            "to": another_model_schema.model_name,
            "on_delete": models.CASCADE,
            "related_name": "parent_objects",
        },
)
model = model_schema.as_model()
related_model = related_model_schema.as_model()

related_instance = related_model.objects.create()
model_instance = model.objects.create(related=related_instance)
related_instance.parent_objects

AttributeError: "parent_objects" does not exist

Workaround

Call related_model_schema.as_model() before `model_schema.as_model(). This works for some reason

Does this support all the model field available in django itself?

hey!

Cannot we use the AbstractModelSchema anymore? I looked into your codes and commits, also wiki but I did not solve my problem. I cannot extend the base model with version of 0.3.0.

Any help will be appreciated. Thanks.

Hi. I try to execute ModelSchema.objects.get(name=dataset_name).as_model()._meta.db_table, but i have error TypeError: can't compare offset-naive and offset-aware datetimes.
Full traceback:

Traceback (most recent call last):
  File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/django/contrib/staticfiles/handlers.py", line 65, in __call__
    return self.application(environ, start_response)
  File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/django/core/handlers/wsgi.py", line 141, in __call__
    response = self.get_response(request)
  File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/django/core/handlers/base.py", line 75, in get_response
    response = self._middleware_chain(request)
  File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/django/core/handlers/exception.py", line 36, in inner
    response = response_for_exception(request, exc)
  File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/django/core/handlers/exception.py", line 90, in response_for_exception
    response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
  File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/django/core/handlers/exception.py", line 125, in handle_uncaught_exception
    return debug.technical_500_response(request, *exc_info)
  File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/django_extensions/management/technical_response.py", line 37, in null_technical_500_response
    six.reraise(exc_type, exc_value, tb)
  File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/six.py", line 692, in reraise
    raise value.with_traceback(tb)
  File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/rest_framework/viewsets.py", line 114, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/rest_framework/views.py", line 505, in dispatch
    response = self.handle_exception(exc)
  File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/rest_framework/views.py", line 465, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/rest_framework/views.py", line 476, in raise_uncaught_exception
    raise exc
  File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/rest_framework/views.py", line 502, in dispatch
    response = handler(request, *args, **kwargs)
  File "/home/mracobes/univer/visualisator/old_metaphore/views/plot.py", line 16, in get_plot_data
    table_name = ModelSchema.objects.get(name=dataset_name).as_model()._meta.db_table
  File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/dynamic_models/models.py", line 152, in as_model
    return self.factory.get_model()
  File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/dynamic_models/factory.py", line 17, in get_model
    if registered and self.schema.is_current_model(registered):
  File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/dynamic_models/models.py", line 124, in is_current_model
    return model._declared >= self.last_modified
TypeError: can't compare offset-naive and offset-aware datetimes

I have a PostgreSQL database that is not managed by Django. Tables are continuously added to it. You could think of it as a data lake or data warehouse.

I'd like to use the Django ORM to query these tables, but tables are inserted often enough that manage.py inspectdb to generate on-disk code is not practical. Instead, I'm thinking to hook into the internals of the Django inspection code to get the definition of a specific table (providing just the table name), and then use this package to dynamically create a model definition, allowing me to use the ORM on this model.

Has anyone done this before? Do you have any pointers on where to start, or things to note?

Also, if I don't want to store the dynamic models in the database, can I use ModelSchema and FieldSchema without saving the instances to the database?

.

I was wondering.

What would be really awesome for me is to be able to use something like:

schema = ModelSchema.objects.db_manager(DB_NAME).create(name=instance.schema)

Is there anything theoretically stopping me from using this approach?

With the last changes, I'm not able to install the main branch:
Python 3.8.10
Running: pip install -e git+https://github.com/rvinzent/django-dynamic-models.git@main#egg=dynamic_models
raises:

Note: plus with the last changes the lib is no longer compatible with Django 2.2

Obtaining dynamic_models from git+https://github.com/rvinzent/django-dynamic-models.git@main#egg=dynamic_models
  Updating /home/mattia/.virtualenvs/starterkit/src/dynamic-models clone (to revision main)
  Running command git fetch -q --tags
  Running command git reset --hard -q 8a37c0d6d67c65f6de4ee4862c53acaf37071176
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing editable metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [16 lines of output]
      Traceback (most recent call last):
        File "/home/mattia/.virtualenvs/starterkit/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/home/mattia/.virtualenvs/starterkit/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/mattia/.virtualenvs/starterkit/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 191, in prepare_metadata_for_build_editable
          return hook(metadata_directory, config_settings)
        File "/tmp/pip-build-env-brr_5og5/overlay/lib/python3.8/site-packages/poetry/core/masonry/api.py", line 44, in prepare_metadata_for_build_wheel
          builder = WheelBuilder(poetry)
        File "/tmp/pip-build-env-brr_5og5/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/wheel.py", line 58, in __init__
          super(WheelBuilder, self).__init__(poetry, executable=executable)
        File "/tmp/pip-build-env-brr_5og5/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/builder.py", line 85, in __init__
          self._module = Module(
        File "/tmp/pip-build-env-brr_5og5/overlay/lib/python3.8/site-packages/poetry/core/masonry/utils/module.py", line 63, in __init__
          raise ModuleOrPackageNotFound(
      poetry.core.masonry.utils.module.ModuleOrPackageNotFound: No file/folder found for package y
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

cc @rvinzent

Thoughts on supporting Foreign Key field and many to many fields?

File "/home/ubox72/Extras/graph/venv/lib/python3.6/site-packages/django/db/models/query.py", line 250, in __repr__ data = list(self[:REPR_OUTPUT_SIZE + 1]) File "/home/ubox72/Extras/graph/venv/lib/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/home/ubox72/Extras/graph/venv/lib/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/home/ubox72/Extras/graph/venv/lib/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/home/ubox72/Extras/graph/venv/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/home/ubox72/Extras/graph/venv/lib/python3.6/site-packages/django/db/backends/utils.py", line 99, in execute return super().execute(sql, params) File "/home/ubox72/Extras/graph/venv/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/home/ubox72/Extras/graph/venv/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/home/ubox72/Extras/graph/venv/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/home/ubox72/Extras/graph/venv/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/home/ubox72/Extras/graph/venv/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "dynamic_models_modelfieldschema" does not exist LINE 1: ...ynamic_models_modelfieldschema"."max_length" FROM "dynamic_m...
^

I keep getting this error anytime I make a slight change to my code.
I found in the previous version's document that "Be sure to grab the latest version of the dynamic model after changing schema or OutdatedModelError will be raised on save."
I tried to avoid using save() and only use get_or_create() but still got the same error.

@rvinzent it's been a while since the latest upload to PyPI, https://pypi.org/project/django-dynamic-model/#history.

Do you mind pushing the latest release ?

Thanks for this amazing package but I've one questions, can you please explain, How can i use foreignkey and many to many field.

Hi. I've written this code to add verbose_name to a field when model is created:

def create_model(model_name, *args):
    model_schema = models.ModelSchema.objects.create(name=model_name)
    for arg in args:
        field_schema = models.FieldSchema.objects.create(name=arg[0], data_type=arg[1])
        model_schema.add_field(field_schema, unique=False, null=False)
    model = model_schema.as_model()
    for arg in args:
        field = model._meta.get_field(arg[0])
        field.verbose_name = arg[2]

create_model('car', ('max_speed', 'integer', 'MAX SPEED'), ('color', 'character', 'COLOR'))
create_model('cable', ('length', 'integer', 'LENGTH'), ('color', 'character', 'COLOR'))

Which does what I wanted. After first run on clean database no warnings, no errors, but after second run I get this warning every time:

/home/maciej/Pulpit/ekosys2/venv/lib/python3.7/site-packages/django/db/models/base.py:320: RuntimeWarning: Model 'components.cables' was already registered. Reloading models is not advised as it can lead to inconsistencies, most notably with related models.
  new_class._meta.apps.register_model(new_class._meta.app_label, new_class)

Models are implemented as in readme. They are created properly.
Any thoughts on what is going on?

0.2.0 looks like a nice refactor, some things got a lot simpler.

One Q -- How do you propose we migrate an existing app from 0.1.0 to 0.2.0?

The migration that creates the modelschema and fieldschema tables in 0.2.0 is numbered 0001 BUT my app was already using 0.1.0, so it never runs the new 0001 from 0.2.0 since the migration system believes that it has already run the 0001 migration from the dynamic_models app (because it already did, back when it was deployed with 0.1.0 which had a different 0001 migration).

Perhaps a new app name would have been appropriate? ie to avoid potential for this sort of conflict.

hi
I want to call the model of the car made, how to do it

It would be nice if user could specify the db_column option for FieldSchema.

I need uppercase field names and as workaround I have to monkey patch the get_options method to inject the db_column option.

Hi.

Python 3.8.x

django 2.2.x
django_dynamic_models 0.3.0

I set settings.CACHE to use memcached. If the memcached service is offline for any reason, simplest things like creating e FieldSchema fail. As soon as I start the memcached service, things work.

I do not understand exactly how the missing cache backend can affect things. I see this library makes use of cache, but it should be just for performance reason.

from dynamic_models.models import ModelSchema, FieldSchema

car_schema = ModelSchema.objects.create(name='Car')
car_model_field = FieldSchema.objects.create(model_schema=car_schema, name='test', data_type='character')  ### raise error

Traceback

Traceback (most recent call last):
  File ".\test.py", line 31, in <module>
    FieldSchema.objects.create(model_schema = model_schema, name = "test", data_type = 'character')
  File "C:\Users\.....\lib\site-packages\django\db\models\manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "C:\Users\.....\lib\site-packages\django\db\models\query.py", line 422, in create
    obj.save(force_insert=True, using=self.db)
  File "C:\Users\.....\lib\site-packages\dynamic_models\models.py", line 95, in save
    self._schema_editor.update_column(model, field)
  File "C:\Users\.....\lib\site-packages\dynamic_models\schema.py", line 39, in update_column
    self.add_column(model, new_field)
  File "C:\Users\.....\lib\site-packages\dynamic_models\schema.py", line 44, in add_column
    editor.add_field(model, field)
  File "C:\Users\.....\lib\site-packages\sql_server\pyodbc\schema.py", line 635, in add_field
    if field.many_to_many and field.remote_field.through._meta.auto_created:
AttributeError: 'NoneType' object has no attribute 'many_to_many'

Thanks for help

please help me,how to fix this,when i execute this code:
car_model_schema.remove_field(color_field_schema)
thanks for your advice

The wiki is not sufficient for documentation across multiple versions and the docs for the newer API should be published to readthedocs.io

Follow up from #55

A check similar to vanilla Django should be performed to validate the table name constraints of the database platform. Source

cc @mattiagiupponi

Started using poetry at work and it's great. I'd like to use it for dependency management here too.

python manage.py makemigrations
Traceback (most recent call last):
File "manage.py", line 15, in
execute_from_command_line(sys.argv)
File "d:\Anaconda3\envs\python37\lib\site-packages\django\core\management_init_.py", line 381, in execute_from_command_line
utility.execute()
File "d:\Anaconda3\envs\python37\lib\site-packages\django\core\management_init_.py", line 357, in execute
django.setup()
File "d:\Anaconda3\envs\python37\lib\site-packages\django_init_.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "d:\Anaconda3\envs\python37\lib\site-packages\django\apps\registry.py", line 93, in populate
"duplicates: %s" % app_config.label)
django.core.exceptions.ImproperlyConfigured: Application labels aren't unique, duplicates: contenttypes

Once I create a model dynamically how do I migrate it to the database?

And how do I access these models at a later stage?

I am trying to create a model builder class to use it for a specific use case but I am unable to figure out how can I migrate these dynamically created models and once they are created how can use them to fetch data and display to users.

from dynamic_models.models import ModelSchema, FieldSchema


def createCustomModel(modelName):

    model_schema = ModelSchema.objects.create(name=modelName)
    modelName = model_schema.as_model()

    modelName.objects.create()
    assert modelName.objects.count() == 1
    print("0")
    return model_schema


def createField(model_name_schema, fieldName, fieldType):

    field = FieldSchema.objects.create(
        model_schema=model_name_schema, name=fieldName, data_type=fieldType)

    model_name = model_name_schema.as_model()
    print("1")


def populateField(model_name_schema, fieldName, fieldData):
    print("2")
    model_name = model_name_schema.as_model()
    model_name.objects.create(fieldName=fieldData)


def main():
    createField(createCustomModel("Cars"), "model", "character")
    populateField("Car", "model", "Camry")


if __name__ == "__main__":
    main()

Github actions is pretty cool, and obviously integrates well with Github. We should use it instead of Travis CI

In some cases can happen that the table is not created with the dynamic models, but it can be used just to map the existing table coming from an external resource
The idea is to add two new settings into the dynamic models lib:

DYNAMIC_MANAGED_MODELS: default -> True -> if False, skip all the update/create method for models and field schema

USE_APPLABEL_IN_TABLE_NAME_PREFIX: default -> True -> if False, use as table name the name provided in the Model schema creation without appending the app_label prefix

As the title says, how can I make some change and run the test?

Hello @rvinzent , we are using your code to create dynamic models. However, after we shutdown the django server and restart using python3 manage.py runserver 0.0.0.0:8000 , we would no longer be able to write new rows into the dynamic models we created before the restart. More specifically, when we call annotations_m.objects.create(**kwargs) (annotations_m is a model from calling .as_model()), we get a OutdatedModelError from check_model_schema in dynamic_models/factory.py.
Before we restart the server, we do not get such Exception when calling the '.create(**kwargs)` function.
Could you please help us out?