Giter VIP home page Giter VIP logo

django-cleanup's Introduction

Django Cleanup

PyPI Package Build Status MIT License

Features

The django-cleanup app automatically deletes files for FileField, ImageField and subclasses. When a FileField's value is changed and the model is saved, the old file is deleted. When a model that has a FileField is deleted, the file is also deleted. A file that is set as the FileField's default value will not be deleted.

Compatibility

How does it work?

In order to track changes of a FileField and facilitate file deletions, django-cleanup connects post_init, pre_save, post_save and post_delete signals to signal handlers for each INSTALLED_APPS model that has a FileField. In order to tell whether or not a FileField's value has changed a local cache of original values is kept on the model instance. If a condition is detected that should result in a file deletion, a function to delete the file is setup and inserted into the commit phase of the current transaction.

Warning! Please be aware of the known limitations documented below!

Installation

pip install django-cleanup

Configuration

Add django_cleanup to the bottom of INSTALLED_APPS in settings.py

INSTALLED_APPS = (
    ...,
    'django_cleanup.apps.CleanupConfig',
)

That is all, no other configuration is necessary.

Note: Order of INSTALLED_APPS is important. To ensure that exceptions inside other apps' signal handlers do not affect the integrity of file deletions within transactions, django_cleanup should be placed last in INSTALLED_APPS.

Troubleshooting

If you notice that django-cleanup is not removing files when expected, check that your models are being properly loaded:

You must define or import all models in your application's models.py or models/__init__.py. Otherwise, the application registry may not be fully populated at this point, which could cause the ORM to malfunction.

If your models are not loaded, django-cleanup will not be able to discover their FileField's.

You can check if your Model is loaded by using

from django.apps import apps
apps.get_models()

Known limitations

Database should support transactions

If you are using a database that does not support transactions you may lose files if a transaction will rollback at the right instance. This outcome is mitigated by our use of post_save and post_delete signals, and by following the recommended configuration in this README. This outcome will still occur if there are signals registered after app initialization and there are exceptions when those signals are handled. In this case, the old file will be lost and the new file will not be referenced in a model, though the new file will likely still exist on disk. If you are concerned about this behavior you will need another solution for old file deletion in your project.

File referenced by multiple model instances

This app is designed with the assumption that each file is referenced only once. If you are sharing a file over two or more model instances you will not have the desired functionality. Be cautious of copying model instances, as this will cause a file to be shared by more than one instance. If you want to reference a file from multiple models add a level of indirection. That is, use a separate file model that is referenced from other models through a foreign key. There are many file management apps already available in the django ecosystem that fulfill this behavior.

Advanced

This section contains additional functionality that can be used to interact with django-cleanup for special cases.

Signals

To facilitate interactions with other django apps django-cleanup sends the following signals which can be imported from django_cleanup.signals:

  • cleanup_pre_delete: just before a file is deleted. Passes a file keyword argument.
  • cleanup_post_delete: just after a file is deleted. Passes a file keyword argument.

Signals example for sorl.thumbnail:

from django_cleanup.signals import cleanup_pre_delete
from sorl.thumbnail import delete

def sorl_delete(**kwargs):
    delete(kwargs['file'])

cleanup_pre_delete.connect(sorl_delete)

Refresh the cache

There have been rare cases where the cache would need to be refreshed. To do so the django_cleanup.cleanup.refresh method can be used:

from django_cleanup import cleanup

cleanup.refresh(model_instance)

Ignore cleanup for a specific model

To ignore a model and not have cleanup performed when the model is deleted or its files change, use the ignore decorator to mark that model:

from django_cleanup import cleanup

@cleanup.ignore
class MyModel(models.Model):
    image = models.FileField()

Only cleanup selected models

If you have many models to ignore, or if you prefer to be explicit about what models are selected, you can change the mode of django-cleanup to "select mode" by using the select mode app config. In your INSTALLED_APPS setting you will replace 'django_cleanup.apps.CleanupConfig' with 'django_cleanup.apps.CleanupSelectedConfig'. Then use the select decorator to mark a model for cleanup:

from django_cleanup import cleanup

@cleanup.select
class MyModel(models.Model):
    image = models.FileField()

How to run tests

Install, setup and use pyenv to install all the required versions of cPython (see the tox.ini).

Setup pyenv to have all versions of python activated within your local django-cleanup repository. Ensuring that the python 3.12 that was installed is first priority.

Install tox on python 3.12 and run the tox command from your local django-cleanup repository.

How to write tests

This app requires the use of django.test.TransactionTestCase when writing tests.

For details on why this is required see here:

Django's TestCase class wraps each test in a transaction and rolls back that transaction after each test, in order to provide test isolation. This means that no transaction is ever actually committed, thus your on_commit() callbacks will never be run. If you need to test the results of an on_commit() callback, use a TransactionTestCase instead.

License

django-cleanup is free software under terms of the:

MIT License

Copyright (C) 2012 by Ilya Shalyapin, [email protected]

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

django-cleanup's People

Contributors

alosultan avatar avallbona avatar beruic avatar coredumperror avatar d3x avatar gak avatar johnthagen avatar jpotterm avatar ldoubleuz avatar nikolaik avatar nisafknisthar avatar rajivm1991 avatar suvit avatar un1t avatar vinnyrose avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

django-cleanup's Issues

Deletes Files shared between instances

I've discovered that if two models instances share a single file, and one of the instances is deleted, the file is also deleted. Obviously this creates an issue when trying to parse the model as the reference to the file remains, while the file does not.

No CHANGELOG

Hi there,

I think we should have some of CHANGELOG file or at least changes in release at GitHub.

cleanup_pre_delete is not called in production environment?

Production environment : Django1.9.7 / AWS S3, AWS Cloudfront (for managing static and media files)

cleanup_pre_delete is not called in production environment.

But it called very well in local development environment.

I have no idea why this happens.

Add tagged releases.

Please add tags to each version so we can better track them. (Only a repo committer can run these)

git tag -a 0.1.0 0fada14463 -m ''
git tag -a 0.1.4 6440f588d4 -m ''
git tag -a 0.1.5 57ec98ae31 -m ''
git tag -a 0.1.6 69a5840949 -m ''
git tag -a 0.1.7 c8f6484153 -m ''
git tag -a 0.1.8 031262d702 -m ''
git tag -a 0.1.9 6adeedba94 -m ''
git tag -a 0.1.10 715d0fcdfb -m ''
git tag -a 0.1.11 40b2983f53 -m ''
git tag -a 0.1.12 d9989d3f30 -m ''
git tag -a 0.1.13 a0901e54b1 -m ''
git tag -a 0.2.0 1902f18ca9 -m ''
git tag -a 0.2.1 bc4925d487 -m ''
git tag -a 0.3.0 6fb0e518fe -m ''
git tag -a 0.3.1 9c4ed988a4 -m ''
git tag -a 0.4.0 2bb0ffe48a -m ''
git push --tags

If you want to check, these should link to their commits:
0.1.0: 0fada14
0.1.4: 6440f58
0.1.5: 57ec98a
0.1.6: 69a5840
0.1.7: c8f6484
0.1.8: 031262d
0.1.9: 6adeedb
0.1.10: 715d0fc
0.1.11: 40b2983
0.1.12: d9989d3
0.1.13: a0901e5
0.2.0: 1902f18
0.2.1: bc4925d
0.3.0: 6fb0e51
0.3.1: 9c4ed98
0.4.0: 2bb0ffe

FileField ID in pre-delete

To delete a file in a non-typical storage location, I required the file ID.
The problem is that cleanup_pre_delete() was being called AFTER the FakeInstance(), which effectively wipes the id from the instance.
My solution is to move cleanup_pre_delete() before calling FakeInstance().
Not sure what other impact this has...

files are deleted when model is invalid

When I use django admin I validate some models because of special requirements. But problem si that django_cleanup deletes files before model is validated. So valid files are removed but model is not saved.
I use clean() method in model to do my validation.

Documentation needs improvement.

Hello Ilya!

For example if I have a model that look like this:

class Profile(models.Model):
    name = models.CharField(max_length=40)
    doc = models.FieldField(upload_to='folder')

The documentation doesn't say how one should use the signals an not every projects need sorl-thumbnail

Exclude a particular file/files?

I have a couple of default image files in my media folder, 'default_m.png' and 'default_f.png' which are assigned to the user as soon as the user is created.

When the user edits his/her profile and uploads a new profile image, the uploaded image is stored in media/profile/ and the default images, (media/default_m.png and media/default_f.png) are deleted.

I would like to exclude these two files from being deleted.

transaction rollback handling

I'm not sure if it's possible to fix this, but I found an issue with a transaction rollback. Basically, when a a model with an image is created and saved inside a transaction, and the transaction then gets rollbacked, the data is not in the db table, but the image is still stored on disk.

I found this issue while writing tests. When I use django.test.TestCase, and work with models, at the end of the test django rollbacks the changes, but the images created during instance.save() calls are still on the disk, which prevents me from running multiple tests in parallel, because I would have to manually delete the files at the end of each test, which would be concurrent. Also, It's not good in an app if you're using db transactions.

Demo code:

try:
    with transaction.atomic():
        self.create_model_with_image()
        raise ValidationError('INVALID')
except ValidationError as e:
    print(e)

Ignoring certain files

Hello!

It is possible to mark certain files to be ignored by this app? I am currently using it to cleanup an avatar gallery, but I wish to have some default avatars as well, that should not be deleted. However, when the User changes their avatar or the User is deleted, the default avatar is deleted as well.

Cleanup empty directories

I switched my project to use django-cleanup and it does things much better than my solution before, but one feature I don't have anymore is cleaning up empty directories after deleting a file.

This was my function for deleting File/Image:

def perform_media_delete(media_field):
    if os.path.isfile(media_field.path):
        os.remove(media_field.path)
        walk = list(os.walk(settings.MEDIA_ROOT))
        for path, _, _ in walk[::-1]:
            if len(os.listdir(path)) == 0:
                os.rmdir(path)
    else:
        return False

It removes all empty sub-directories relative to the MEDIA_ROOT location. This is useful when your upload_to path is something that follows the standards of using the first and second letter of the file name to build the directory:

/path/to/file/a/p/ap45bgfh3.png
--------------^ ^-^^

This feature may be undesirable for most, but is it something that can optional and disabled by default?

Have option to use django-cleanup explicitly on models

By default, django-cleanup works by cleaning all models unless you explicitly ignore them with @cleanup.ignore. I'm wondering if django-cleanup could work in such a way that it doesn't cleanup any models unless you explicitly ask it to?

For example:

@cleanup.cleanup
class UserImage(models.Model):
    image = models.FileField()

Renaming file, causes deletion

Hi, I have a model with a file field and a char field representing the name of the file.
on model.save() im renaming the file.name attribute to match the name attribute of the instance.
This causes file deletion when using django-cleanup.

Any way to prevent this?

Django-cleanup seems not working anymore on Django 1.8

Django 1.8.5
Django-cleanup 0.4.0

File "/home/khamaileon/workspace/khamaileon/project/api/feed/models.py", line 73, in fetch
    self.save()
  File "/home/khamaileon/.virtualenvs/project-api/lib/python3.4/site-packages/django/db/models/base.py", line 734, in save
    force_update=force_update, update_fields=update_fields)
  File "/home/khamaileon/.virtualenvs/project-api/lib/python3.4/site-packages/django/db/models/base.py", line 771, in save_base
    update_fields=update_fields, raw=raw, using=using)
  File "/home/khamaileon/.virtualenvs/project-api/lib/python3.4/site-packages/django/dispatch/dispatcher.py", line 201, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/home/khamaileon/.virtualenvs/project-api/lib/python3.4/site-packages/django_cleanup/models.py", line 65, in delete_old_post_save
    delete_file(old_file, using)
  File "/home/khamaileon/.virtualenvs/project-api/lib/python3.4/site-packages/django_cleanup/models.py", line 87, in delete_file
    on_commit(run_on_commit, using)
  File "/home/khamaileon/.virtualenvs/project-api/lib/python3.4/site-packages/django_cleanup/models.py", line 17, in on_commit
    func()
  File "/home/khamaileon/.virtualenvs/project-api/lib/python3.4/site-packages/django_cleanup/models.py", line 84, in run_on_commit
    file_.delete(save=False)
  File "/home/khamaileon/.virtualenvs/project-api/lib/python3.4/site-packages/django/db/models/fields/files.py", line 124, in delete
    self.storage.delete(self.name)
AttributeError: 'FieldFile' object has no attribute 'storage'

When copying Model-Instances as per django documentation, adjusting either deletes the File for the other

Hi,
The Django documentation mentions this method for copying Model-Instance: https://docs.djangoproject.com/en/4.0/topics/db/queries/#copying-model-instances as per default Django behaviour this results in a non-unique filename. If one of them is afterwards adjusted, #51 takes place and The file is deleted from the other copy. Could you please mention in your documentation/enforce in some way Uniqueness in FileFields so that Files with the same name are not deleted as easily? :)

how it works with aws S3

Hi,
I am having trouble to understand the thumbnail example.
My case is that I have a class CV which has a file attribute - type : FileField.this file is stored in S3

Now I would like to send signal pre_delete to the model before its instance is going to be deleted.
My issue is that I don't know how to import the CV model into the function.
What i have tried.
from django_cleanup.signals import cleanup_pre_delete, cleanup_post_delete

def CV(models.Model):
...
file = models.FileFiles(path)

def cv_file_delete(**kwargs):
from CV import delete // this is raising error
delete(CV.file)
cleanup_pre_delete.connect(cv_file_delete)

Migrate to new CI platform

If you navigate here: https://travis-ci.org/github/un1t/django-cleanup

Since June 15th, 2021, the building on travis-ci.org is ceased. Please use travis-ci.com from now on.

Screenshot 2021-10-24 073717

This project will need to migrate to either travis-ci.com or GitHub Actions/Circle CI/etc.

Travis CI (.com) is no longer free for FOSS projects

I would personally recommend GitHub Actions. An example of how to set up GitHub Actions with tox:

Breaks django-admin-sortable2

Hey. Django-cleanup seems to break django-admin-sortable2 (or vice versa?)

When reordering some models in the Admin, the AJAX request sent will fail with HTTP 500. I am not quite sure what is happening, but fallback_pre_save (part of django-cleanup) is called at some point, resulting in the following error:

TypeError at /admin/main/trailers/adminsortable2_update/
fallback_pre_save() missing 2 required positional arguments: 'raw' and 'using'

Not sure what app actually needs its behaviour fixed, maybe @jrief can jump on the issue too.

Does this package support djangorestframework (drf)?

I am using DRF to provide a REST API on top of Django. I recently noticed that Django does not automatically delete files on disk when the FileField is deleted, so I was hoping django-cleanup could address this.

setup.py:

INSTALLED_APPS = [
    'grappelli',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'rest_framework',
    'myapp.apps.MyAppConfig',
    'debug_toolbar',
    'guardian',
    'django_cleanup.apps.CleanupConfig',
]

models.py:

class Project(models.Model):
    pass

class MyModel(models.Model):
    project = models.ForeignKey(Project, related_name='my_models', on_delete=models.CASCADE)
    image = models.ImageField(upload_to='my_images')

But when I do a RESTful DELETE of an instance of Project, the files still don't delete?

The DRF ModelSerializer for Project deletes all of the Project and MyModel instances correctly (the database is empty), but the files remain.

Is there any configuration needed in MyAppConfig to associate the model? Is there a way to debug why django-cleanup doesn't seem to be getting called?

Wanted to check that this wasn't some kind of known limitation.

Check uniqueness before delete?

I'm using django 3.2.
Assume we have a Image model, which includes a ImageField to save image.
On my local computer, if I create two Image objects and save image.jpg twice, it will produce two files.

  • image.jpg
  • image_ABCDE.jpg

However, on google cloud storage. https://django-storages.readthedocs.io/en/latest/backends/gcloud.html
Storing image.jpg twice will only produce one file.

  • image.jpg

As a result, if I delete one Image object, another Image object cannot find the image.jpg file.
It would be nice if django-cleanup can provide an option to explicitly scan the uniqueness of the file.

Infinite recursion between django-modeltranslations and django-cleanup

Not sure if it's related to a recent upgrade to django 1.10 but there is an infinite recursion between django-cleanup and django modeltranslations.

django-cleanup==0.4.2
django-modeltranslation==0.12
Django==1.10

File "/site/env/python/local/lib/python2.7/site-packages/django/db/models/base.py", line 557, in init
signals.post_init.send(sender=self.class, instance=self)
File "/site/env/python/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 191, in send
response = receiver(signal=self, sender=sender, **named)
File "/site/env/python/local/lib/python2.7/site-packages/django_cleanup/handlers.py", line 44, in cache_original_post_init
cache.make_cleanup_cache(instance)
File "/site/env/python/local/lib/python2.7/site-packages/django_cleanup/cache.py", line 165, in make_cleanup_cache
fields_for_model_instance(source, using=instance)))
File "/site/env/python/local/lib/python2.7/site-packages/django_cleanup/cache.py", line 108, in fields_for_model_instance
field = get_field_instance(instance, field_name, using=using)
File "/site/env/python/local/lib/python2.7/site-packages/django_cleanup/cache.py", line 82, in get_field_instance
field = getattr(instance, field_name, None)
File "/site/env/python/local/lib/python2.7/site-packages/django/db/models/fields/files.py", line 173, in get
instance.refresh_from_db(fields=[self.field.name])
File "/site/env/python/local/lib/python2.7/site-packages/modeltranslation/translator.py", line 304, in new_refresh_from_db
return old_refresh_from_db(self, using, fields)
File "/site/env/python/local/lib/python2.7/site-packages/django/db/models/base.py", line 685, in refresh_from_db
db_instance = db_instance_qs.get()
File "/site/env/python/local/lib/python2.7/site-packages/django/db/models/query.py", line 379, in get
num = len(clone)
File "/site/env/python/local/lib/python2.7/site-packages/django/db/models/query.py", line 238, in len
self._fetch_all()
File "/site/env/python/local/lib/python2.7/site-packages/django/db/models/query.py", line 1085, in _fetch_all
self._result_cache = list(self.iterator())
File "/site/env/python/local/lib/python2.7/site-packages/django/db/models/query.py", line 66, in iter
obj = model_cls.from_db(db, init_list, row[model_fields_start:model_fields_end])
File "/site/env/python/local/lib/python2.7/site-packages/django/db/models/base.py", line 565, in from_db
new = cls(*values)
File "/site/env/python/local/lib/python2.7/site-packages/modeltranslation/translator.py", line 245, in new_init
old_init(self, *args, **kwargs)
File "/site/env/python/local/lib/python2.7/site-packages/django/db/models/base.py", line 557, in init
signals.post_init.send(sender=self.class, instance=self)

ImportError using django 1.6.5

[...]/lib/python2.7/site-packages/django_cleanup/models.py", line 3, in
from django.apps import apps
ImportError: No module named apps

PermissionError: There was an exception deleting the file because it is being used by another process

Strange kind of bug i came across.

When initiating a model instance, and then opening it in the admin page as if i were to modify it, simpli saving without making any changes prompts me with the following traceback:

There was an exception deleting the file `logo/6000_4000.webp` on field `accounts.restaurant.logo`
Traceback (most recent call last):

  File "C:\Users\sherzog\web_app\env\lib\site-packages\django_cleanup\handlers.py", line 96, in run_on_commit
    file_.delete(save=False)

  File "C:\Users\sherzog\web_app\env\lib\site-packages\django\db\models\fields\files.py", line 373, in delete
    super().delete(save)

  File "C:\Users\sherzog\web_app\env\lib\site-packages\django\db\models\fields\files.py", line 105, in delete
    self.storage.delete(self.name)

  File "C:\Users\sherzog\web_app\env\lib\site-packages\django\core\files\storage.py", line 304, in delete
    os.remove(name)

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\sherzog\\web_app\\backend\\media\\logo\\6000_4000.webp'

It seems it is interpreting this as a replacement of the original logo with the same one, or something like this. Should i be worried about this?

Not working with Django 4.1+

When using with a FileField in Django 4.1 neither automatic nor manual cleanup works.

Files still remaining after clearing the file or updating with a new file.

Calling cleanup.refresh(model_instance) does not work either.

Support integration with django-reversion

Are there any plans to or an opinion on supporting a deeper integration with django-reversion?
E.g. Do not delete files of objects which are under version control but delete them in case the last version referencing this particular file is deleted.

We do need this in our project and I would be happy to contribute any work as a PR.

Make a Django 3.0 release

Hello,

It seems the most recent release (4.0.0) is not yet compatible with Django 3.0 while Master is.

Would it be possible to create a stable release which can be installed from Pypi?

Совместимость с django 1.7

Здравствуйте!
Успешно использую Ваше приложение.
Но при обновлении до django-1.7b2 приложение выдает ошибку.

raise RuntimeError("App registry isn't ready yet.")
RuntimeError: App registry isn't ready yet.

Exception raised in delete_file() method when previously no file exists

Full traceback:

ERROR Unexpected exception while attempting to delete file ''
Traceback (most recent call last):
  File "/home/ubuntu/.virtualenvs/home/local/lib/python2.7/site-packages/django_cleanup/models.py", line 62, in delete_file
    storage.delete(file_.name)
  File "/home/ubuntu/.virtualenvs/home/local/lib/python2.7/site-    packages/django/core/files/storage.py", line 231, in delete
    assert name, "The name argument is not allowed to be empty."
AssertionError: The name argument is not allowed to be empty.

If previously a file does not exist in the model instance, and a new one is being uploaded, the old_file gets the value None, and it is sent for deletion without checking whether that file actually exists or not.

if old_file != new_file:
        delete_file(old_file)       # in L36 in models.py `

And storage.exists(file_.name) comes out to be true for empty name.

Suggestion: Add Disclaimer that django-cleanup doesn't work with python manage.py flush

So I needed a solution to periodically clear out old database entries from my database, and that database also has attached files -- something django-cleanup is well suited for. The obvious practice is to write a script using delete() and attach a crontab job to it, but I initially tried python manage.py flush (which clears out the entire database anyways) figuring that would be a quicker solution. Interestingly django-cleanup didn't catch the database records being deleted and as a result didn't delete the old files. manage.py flush isn't likely to be part of a standard production routine (which is probably why it was overlooked) but it might be a good idea to add a small disclaimer to README.md warning that manage.py flush won't work with django-cleanup. Thanks for making django-cleanup!

Data loss if same file is saved in multiple fields

If a file is saved in multiple instances, then removing a single instance removes the file and cause 'file not found' error for every other instance.

We need to check if the file is referenced in any other object before deleting.

Add License To Project

Would you be willing to add a license to this project so users of this project know how they are allowed to us it?

Test with Django 2.2

Saw indications that it works with Django 2.2? Is it approved for 2.2 and maintained?

It's a very useful tool, but it's dangerous (can be data-destructive), so I would only use it if it's maintained.

Doesn't seem to work with replaced files under Django 2.0.3

I'm trying it out for the first time, and noticed that while it does correctly delete the files when the model instance is deleted, it fails to delete files when they are replaced.

Hooking it up to a debugger and stepping through the code, it appears that in 'django_cleanup.handlers.delete_old_post_save, fetching the cached FileFieldreturns *the same object* as the "new"FileField. My suspicion is that in the past Django would create a new File` object when a file is replaced, but the current version instead updates the existing object. I suspect that the solution is to have the cache system store a copy of the field.

I'm going to run some experiments and report back.

Deletes unwanted files

I have a model having ImageField (default to some image). When the ImageField is updated by any instance of the model, the default image is deleted. This causes problems when many instances of same model are linked with default image, and one of them updates.

Reference:
my project Develop branch, Profile model.

save() raises FileNotFoundError

Hello, Ilya!

We are using this lib in a while. In most cases it works perfect! Saves lots of space..

Resently my colleague found small issue with manual file removal.. If somewhere in code I already delete a file (or it is disappeared after django created an object) then .save() will raise FileNotFoundError

>>> obj = MyModel.object.all().first()
>>> obj.data.delete()
>>> obj.data = None
>>> obj.save()  # raises FileNotFoundError

May be it will be better to catch this exception inside this application? In case of, for example, using another library, which is know nothing about django_cleanup?

I'm ready to create pull request if you don't mind

Files not deleted on database rollbacks

I found that my device still would contain files which are created in a atomic nested post request (using drf with multiple nested models), WHEN an error occurs during the request after the files have been written and a database rollback is triggered because after all I don't want to have a partial instance created.

My current workaround is to overwrite the save method in my FileSerializer and store the information about the created files so I can manually delete them in a try+except wrapper later:

class Singleton(type):
	_instances = {}

	def __call__(cls, *args, **kwargs):
		if cls not in cls._instances:
			cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
		return cls._instances[cls]


class FileCreationHistory(metaclass=Singleton):
	createdFiles = []

	@staticmethod
	def reset():
		FileCreationHistory.createdFiles = []


class FileSerializer(ModelSerializer):
	scenery = PrimaryKeyRelatedField(allow_null=True, required=False, queryset=Scenery.objects.all())

	class Meta:
		model = File
		fields = ['file', 'scenery']

	def save(self, **kwargs):
		result = super(FileSerializer, self).save(**kwargs)
		FileCreationHistory.createdFiles.append(result.file)
		return result

# WritableNestedModelSerializer is from the drf_writable_nested package
class ScenerySerializer(WritableNestedModelSerializer):

	# ... more child fields and serializers here...

	# Reverse foreign key
	files = FileSerializer(many=True)

	class Meta:
		model = Scenery
		fields = [
			'title',
			'description',
			'files'
			# ... more fields here...
		]

	def save(self, **kwargs):
		FileCreationHistory.reset()
		try:
			with transaction.atomic():
				self._save_kwargs = defaultdict(dict, kwargs)
				return super(BaseNestedModelSerializer, self).save(**kwargs)
		except Exception as e:
			"""
				Whatever errors occurred during the creation of the Scenery and its children:
				We have to properly delete all files which have been placed at the local storage device, because
				at this point they are deleted from the db, but are still on the hard drive.
				django-cleanup does not work with db rollbacks as it seems. So we use a singleton file creation history here.
			"""
			for file in FileCreationHistory.createdFiles:
				file.delete(save=False)  # save=False because the instance probably does not exist anymore at this point.
			FileCreationHistory.reset()
			raise e

This implementation is not thread-safe (yet), just wanted to share if someone runs into the same issue or if the dev could find a solution to this problem.

cleanup command

It would be very nice to have a command that iterate over all file fields an delete all media-files (under a path specified by a setting) that are not used by any model-field anymore.
This would be very useful for existing projects where django-cleanup was not installed since the beginning.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.