rvinzent / django-dynamic-models Goto Github PK
View Code? Open in Web Editor NEWDynamic Django models allow users to define, edit, and populate their own database schema.
License: MIT License
Dynamic Django models allow users to define, edit, and populate their own database schema.
License: MIT License
Requesting feature to support type ArrayField to allow the user to store data in an array.
Here is my scenario:
I want to create task table for every user, since each user may add this own dynamic fields.
def create_task_list(request):
schema = ModelSchema.objects.create(name=f"task_{request.user.id}")
# create common fields
FieldSchema.objects.create(model_schema=schema, name="status", data_type="character")
...also create common fields like create_datetime, user, update_datetime...
# create user fields
for field in request.POST['fields']:
...
How can I create an abstract model like
class TaskModel:
status = models.CharField()
create_datetime = models.DateTimeField(auto_now_add=True)
update_datetime = models.DateTimeField(auto_now=True)
user = models.ForeignKey(User, on_delete=models.CASCADE)
and use this model as a baseclass to create dynamic model like:
def create_task_list(request):
schema = ModelSchema.objects.create(name=f"task_{request.user.id}", base_model=TaskModel)
# create user fields
for field in request.POST['fields']:
...
Hi,
I have installed django-dynamic-models and I'm following the documentation.
When I create a new object with the command :
car_schema = ModelSchema.objects.create(name='car')
The command return an error :
no such table: dynamic_models_modelfieldschema
Have you ever encountered this error ?
Hello @rvinzent, I've been trying to work with your module but every time I migrate I get greeted with a wall of errors and a keyerror, I've tried all kinds of things and starting fresh apps but I keep getting the same error.
KeyError: ('dynamic_models', 'modelschema')
Do you have any idea how I can avoid this error?
The relationships between these models should be simplified. Care should be taken to disallow fields with the same name to appear on multiple model schema.
Issue location: /dynamic_models/models.py:254
That is:
class ModelFieldSchema(GenericModel, GenericField):
objects = ModelFieldSchemaManager()
null = models.BooleanField(default=False) # issue1
unique = models.BooleanField(default=False) # issue2
max_length = models.PositiveIntegerField(null=True)
Problem description:
'null' and 'unique' are keywords in mysql grammar.
In our strict DBA system, 'null' and 'unique' are forbidden to be column names.
Suggestion:
code to be modified as follows.
class ModelFieldSchema(GenericModel, GenericField):
objects = ModelFieldSchemaManager()
null = models.BooleanField(default=False, db_column='is_null') # issue1
unique = models.BooleanField(default=False, db_column='is_unique') # issue2
max_length = models.PositiveIntegerField(null=True)
model_schema = ModelSchema.objects.create(name="SimpleModel")
related_model_schema = ModelSchema.objects.create(name="RelatedModel")
FieldSchema.objects.create(
name="related",
model_schema=model_schema,
class_name="django.db.models.ForeignKey",
kwargs={
"to": another_model_schema.model_name,
"on_delete": models.CASCADE,
"related_name": "parent_objects",
},
)
model = model_schema.as_model()
related_model = related_model_schema.as_model()
related_instance = related_model.objects.create()
model_instance = model.objects.create(related=related_instance)
related_instance.parent_objects
AttributeError: "parent_objects" does not exist
Call related_model_schema.as_model()
before `model_schema.as_model(). This works for some reason
Does this support all the model field available in django itself?
hey!
Cannot we use the AbstractModelSchema anymore? I looked into your codes and commits, also wiki but I did not solve my problem. I cannot extend the base model with version of 0.3.0.
Any help will be appreciated. Thanks.
Hi. I try to execute ModelSchema.objects.get(name=dataset_name).as_model()._meta.db_table
, but i have error TypeError: can't compare offset-naive and offset-aware datetimes
.
Full traceback:
Traceback (most recent call last):
File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/django/contrib/staticfiles/handlers.py", line 65, in __call__
return self.application(environ, start_response)
File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/django/core/handlers/wsgi.py", line 141, in __call__
response = self.get_response(request)
File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/django/core/handlers/base.py", line 75, in get_response
response = self._middleware_chain(request)
File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/django/core/handlers/exception.py", line 36, in inner
response = response_for_exception(request, exc)
File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/django/core/handlers/exception.py", line 90, in response_for_exception
response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/django/core/handlers/exception.py", line 125, in handle_uncaught_exception
return debug.technical_500_response(request, *exc_info)
File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/django_extensions/management/technical_response.py", line 37, in null_technical_500_response
six.reraise(exc_type, exc_value, tb)
File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/six.py", line 692, in reraise
raise value.with_traceback(tb)
File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/rest_framework/viewsets.py", line 114, in view
return self.dispatch(request, *args, **kwargs)
File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/rest_framework/views.py", line 505, in dispatch
response = self.handle_exception(exc)
File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/rest_framework/views.py", line 465, in handle_exception
self.raise_uncaught_exception(exc)
File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/rest_framework/views.py", line 476, in raise_uncaught_exception
raise exc
File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/rest_framework/views.py", line 502, in dispatch
response = handler(request, *args, **kwargs)
File "/home/mracobes/univer/visualisator/old_metaphore/views/plot.py", line 16, in get_plot_data
table_name = ModelSchema.objects.get(name=dataset_name).as_model()._meta.db_table
File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/dynamic_models/models.py", line 152, in as_model
return self.factory.get_model()
File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/dynamic_models/factory.py", line 17, in get_model
if registered and self.schema.is_current_model(registered):
File "/home/mracobes/univer/visualisator/.venv/lib/python3.7/site-packages/dynamic_models/models.py", line 124, in is_current_model
return model._declared >= self.last_modified
TypeError: can't compare offset-naive and offset-aware datetimes
I have a PostgreSQL database that is not managed by Django. Tables are continuously added to it. You could think of it as a data lake or data warehouse.
I'd like to use the Django ORM to query these tables, but tables are inserted often enough that manage.py inspectdb
to generate on-disk code is not practical. Instead, I'm thinking to hook into the internals of the Django inspection code to get the definition of a specific table (providing just the table name), and then use this package to dynamically create a model definition, allowing me to use the ORM on this model.
Has anyone done this before? Do you have any pointers on where to start, or things to note?
Also, if I don't want to store the dynamic models in the database, can I use ModelSchema
and FieldSchema
without saving the instances to the database?
.
I was wondering.
What would be really awesome for me is to be able to use something like:
schema = ModelSchema.objects.db_manager(DB_NAME).create(name=instance.schema)
Is there anything theoretically stopping me from using this approach?
With the last changes, I'm not able to install the main branch:
Python 3.8.10
Running: pip install -e git+https://github.com/rvinzent/django-dynamic-models.git@main#egg=dynamic_models
raises:
Note: plus with the last changes the lib is no longer compatible with Django 2.2
Obtaining dynamic_models from git+https://github.com/rvinzent/django-dynamic-models.git@main#egg=dynamic_models
Updating /home/mattia/.virtualenvs/starterkit/src/dynamic-models clone (to revision main)
Running command git fetch -q --tags
Running command git reset --hard -q 8a37c0d6d67c65f6de4ee4862c53acaf37071176
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... done
Preparing editable metadata (pyproject.toml) ... error
error: subprocess-exited-with-error
× Preparing editable metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [16 lines of output]
Traceback (most recent call last):
File "/home/mattia/.virtualenvs/starterkit/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
main()
File "/home/mattia/.virtualenvs/starterkit/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/mattia/.virtualenvs/starterkit/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 191, in prepare_metadata_for_build_editable
return hook(metadata_directory, config_settings)
File "/tmp/pip-build-env-brr_5og5/overlay/lib/python3.8/site-packages/poetry/core/masonry/api.py", line 44, in prepare_metadata_for_build_wheel
builder = WheelBuilder(poetry)
File "/tmp/pip-build-env-brr_5og5/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/wheel.py", line 58, in __init__
super(WheelBuilder, self).__init__(poetry, executable=executable)
File "/tmp/pip-build-env-brr_5og5/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/builder.py", line 85, in __init__
self._module = Module(
File "/tmp/pip-build-env-brr_5og5/overlay/lib/python3.8/site-packages/poetry/core/masonry/utils/module.py", line 63, in __init__
raise ModuleOrPackageNotFound(
poetry.core.masonry.utils.module.ModuleOrPackageNotFound: No file/folder found for package y
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
cc @rvinzent
Thoughts on supporting Foreign Key field and many to many fields?
File "/home/ubox72/Extras/graph/venv/lib/python3.6/site-packages/django/db/models/query.py", line 250, in __repr__ data = list(self[:REPR_OUTPUT_SIZE + 1]) File "/home/ubox72/Extras/graph/venv/lib/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__ self._fetch_all() File "/home/ubox72/Extras/graph/venv/lib/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/home/ubox72/Extras/graph/venv/lib/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/home/ubox72/Extras/graph/venv/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql cursor.execute(sql, params) File "/home/ubox72/Extras/graph/venv/lib/python3.6/site-packages/django/db/backends/utils.py", line 99, in execute return super().execute(sql, params) File "/home/ubox72/Extras/graph/venv/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/home/ubox72/Extras/graph/venv/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers return executor(sql, params, many, context) File "/home/ubox72/Extras/graph/venv/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/home/ubox72/Extras/graph/venv/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/home/ubox72/Extras/graph/venv/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: relation "dynamic_models_modelfieldschema" does not exist LINE 1: ...ynamic_models_modelfieldschema"."max_length" FROM "dynamic_m...
^
I keep getting this error anytime I make a slight change to my code.
I found in the previous version's document that "Be sure to grab the latest version of the dynamic model after changing schema or OutdatedModelError will be raised on save."
I tried to avoid using save()
and only use get_or_create()
but still got the same error.
@rvinzent it's been a while since the latest upload to PyPI, https://pypi.org/project/django-dynamic-model/#history.
Do you mind pushing the latest release ?
Thanks for this amazing package but I've one questions, can you please explain, How can i use foreignkey and many to many field.
Hi. I've written this code to add verbose_name to a field when model is created:
def create_model(model_name, *args):
model_schema = models.ModelSchema.objects.create(name=model_name)
for arg in args:
field_schema = models.FieldSchema.objects.create(name=arg[0], data_type=arg[1])
model_schema.add_field(field_schema, unique=False, null=False)
model = model_schema.as_model()
for arg in args:
field = model._meta.get_field(arg[0])
field.verbose_name = arg[2]
create_model('car', ('max_speed', 'integer', 'MAX SPEED'), ('color', 'character', 'COLOR'))
create_model('cable', ('length', 'integer', 'LENGTH'), ('color', 'character', 'COLOR'))
Which does what I wanted. After first run on clean database no warnings, no errors, but after second run I get this warning every time:
/home/maciej/Pulpit/ekosys2/venv/lib/python3.7/site-packages/django/db/models/base.py:320: RuntimeWarning: Model 'components.cables' was already registered. Reloading models is not advised as it can lead to inconsistencies, most notably with related models.
new_class._meta.apps.register_model(new_class._meta.app_label, new_class)
Models are implemented as in readme. They are created properly.
Any thoughts on what is going on?
0.2.0 looks like a nice refactor, some things got a lot simpler.
One Q -- How do you propose we migrate an existing app from 0.1.0 to 0.2.0?
The migration that creates the modelschema and fieldschema tables in 0.2.0 is numbered 0001 BUT my app was already using 0.1.0, so it never runs the new 0001 from 0.2.0 since the migration system believes that it has already run the 0001 migration from the dynamic_models app (because it already did, back when it was deployed with 0.1.0 which had a different 0001 migration).
Perhaps a new app name would have been appropriate? ie to avoid potential for this sort of conflict.
hi
I want to call the model of the car made, how to do it
It would be nice if user could specify the db_column
option for FieldSchema.
I need uppercase field names and as workaround I have to monkey patch the get_options
method to inject the db_column
option.
Hi.
Python 3.8.x
django 2.2.x
django_dynamic_models 0.3.0
I set settings.CACHE to use memcached. If the memcached service is offline for any reason, simplest things like creating e FieldSchema fail. As soon as I start the memcached service, things work.
I do not understand exactly how the missing cache backend can affect things. I see this library makes use of cache, but it should be just for performance reason.
from dynamic_models.models import ModelSchema, FieldSchema
car_schema = ModelSchema.objects.create(name='Car')
car_model_field = FieldSchema.objects.create(model_schema=car_schema, name='test', data_type='character') ### raise error
Traceback
Traceback (most recent call last):
File ".\test.py", line 31, in <module>
FieldSchema.objects.create(model_schema = model_schema, name = "test", data_type = 'character')
File "C:\Users\.....\lib\site-packages\django\db\models\manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "C:\Users\.....\lib\site-packages\django\db\models\query.py", line 422, in create
obj.save(force_insert=True, using=self.db)
File "C:\Users\.....\lib\site-packages\dynamic_models\models.py", line 95, in save
self._schema_editor.update_column(model, field)
File "C:\Users\.....\lib\site-packages\dynamic_models\schema.py", line 39, in update_column
self.add_column(model, new_field)
File "C:\Users\.....\lib\site-packages\dynamic_models\schema.py", line 44, in add_column
editor.add_field(model, field)
File "C:\Users\.....\lib\site-packages\sql_server\pyodbc\schema.py", line 635, in add_field
if field.many_to_many and field.remote_field.through._meta.auto_created:
AttributeError: 'NoneType' object has no attribute 'many_to_many'
Thanks for help
please help me,how to fix this,when i execute this code:
car_model_schema.remove_field(color_field_schema)
thanks for your advice
The wiki is not sufficient for documentation across multiple versions and the docs for the newer API should be published to readthedocs.io
Started using poetry
at work and it's great. I'd like to use it for dependency management here too.
python manage.py makemigrations
Traceback (most recent call last):
File "manage.py", line 15, in
execute_from_command_line(sys.argv)
File "d:\Anaconda3\envs\python37\lib\site-packages\django\core\management_init_.py", line 381, in execute_from_command_line
utility.execute()
File "d:\Anaconda3\envs\python37\lib\site-packages\django\core\management_init_.py", line 357, in execute
django.setup()
File "d:\Anaconda3\envs\python37\lib\site-packages\django_init_.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "d:\Anaconda3\envs\python37\lib\site-packages\django\apps\registry.py", line 93, in populate
"duplicates: %s" % app_config.label)
django.core.exceptions.ImproperlyConfigured: Application labels aren't unique, duplicates: contenttypes
Once I create a model dynamically how do I migrate it to the database?
And how do I access these models at a later stage?
I am trying to create a model builder class to use it for a specific use case but I am unable to figure out how can I migrate these dynamically created models and once they are created how can use them to fetch data and display to users.
from dynamic_models.models import ModelSchema, FieldSchema
def createCustomModel(modelName):
model_schema = ModelSchema.objects.create(name=modelName)
modelName = model_schema.as_model()
modelName.objects.create()
assert modelName.objects.count() == 1
print("0")
return model_schema
def createField(model_name_schema, fieldName, fieldType):
field = FieldSchema.objects.create(
model_schema=model_name_schema, name=fieldName, data_type=fieldType)
model_name = model_name_schema.as_model()
print("1")
def populateField(model_name_schema, fieldName, fieldData):
print("2")
model_name = model_name_schema.as_model()
model_name.objects.create(fieldName=fieldData)
def main():
createField(createCustomModel("Cars"), "model", "character")
populateField("Car", "model", "Camry")
if __name__ == "__main__":
main()
Github actions is pretty cool, and obviously integrates well with Github. We should use it instead of Travis CI
In some cases can happen that the table is not created with the dynamic models, but it can be used just to map the existing table coming from an external resource
The idea is to add two new settings into the dynamic models lib:
DYNAMIC_MANAGED_MODELS
: default -> True -> if False, skip all the update/create method for models and field schema
USE_APPLABEL_IN_TABLE_NAME_PREFIX
: default -> True -> if False, use as table name the name provided in the Model schema creation without appending the app_label prefix
As the title says, how can I make some change and run the test?
Hello @rvinzent , we are using your code to create dynamic models. However, after we shutdown the django server and restart using python3 manage.py runserver 0.0.0.0:8000
, we would no longer be able to write new rows into the dynamic models we created before the restart. More specifically, when we call annotations_m.objects.create(**kwargs)
(annotations_m
is a model from calling .as_model()
), we get a OutdatedModelError
from check_model_schema
in dynamic_models/factory.py
.
Before we restart the server, we do not get such Exception when calling the '.create(**kwargs)` function.
Could you please help us out?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.