un1t / django-cleanup Goto Github PK

Hi there,

I think we should have some of CHANGELOG file or at least changes in release at GitHub.

It would be very nice to have a command that iterate over all file fields an delete all media-files (under a path specified by a setting) that are not used by any model-field anymore.
This would be very useful for existing projects where django-cleanup was not installed since the beginning.

To delete a file in a non-typical storage location, I required the file ID.
The problem is that cleanup_pre_delete() was being called AFTER the FakeInstance(), which effectively wipes the id from the instance.
My solution is to move cleanup_pre_delete() before calling FakeInstance().
Not sure what other impact this has...

Travis CI could be added so that each commit (and every PR) is tested against all configurations.

DRF has a pretty good example Travis config you could use as a reference: https://github.com/encode/django-rest-framework/blob/master/.travis.yml

Здравствуйте!
Успешно использую Ваше приложение.
Но при обновлении до django-1.7b2 приложение выдает ошибку.

raise RuntimeError("App registry isn't ready yet.")
RuntimeError: App registry isn't ready yet.

Are there any plans to or an opinion on supporting a deeper integration with django-reversion?
E.g. Do not delete files of objects which are under version control but delete them in case the last version referencing this particular file is deleted.

We do need this in our project and I would be happy to contribute any work as a PR.

Django 1.8.5
Django-cleanup 0.4.0

File "/home/khamaileon/workspace/khamaileon/project/api/feed/models.py", line 73, in fetch
    self.save()
  File "/home/khamaileon/.virtualenvs/project-api/lib/python3.4/site-packages/django/db/models/base.py", line 734, in save
    force_update=force_update, update_fields=update_fields)
  File "/home/khamaileon/.virtualenvs/project-api/lib/python3.4/site-packages/django/db/models/base.py", line 771, in save_base
    update_fields=update_fields, raw=raw, using=using)
  File "/home/khamaileon/.virtualenvs/project-api/lib/python3.4/site-packages/django/dispatch/dispatcher.py", line 201, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/home/khamaileon/.virtualenvs/project-api/lib/python3.4/site-packages/django_cleanup/models.py", line 65, in delete_old_post_save
    delete_file(old_file, using)
  File "/home/khamaileon/.virtualenvs/project-api/lib/python3.4/site-packages/django_cleanup/models.py", line 87, in delete_file
    on_commit(run_on_commit, using)
  File "/home/khamaileon/.virtualenvs/project-api/lib/python3.4/site-packages/django_cleanup/models.py", line 17, in on_commit
    func()
  File "/home/khamaileon/.virtualenvs/project-api/lib/python3.4/site-packages/django_cleanup/models.py", line 84, in run_on_commit
    file_.delete(save=False)
  File "/home/khamaileon/.virtualenvs/project-api/lib/python3.4/site-packages/django/db/models/fields/files.py", line 124, in delete
    self.storage.delete(self.name)
AttributeError: 'FieldFile' object has no attribute 'storage'

Instead of the generic message the exception should be passed so the user knows what exact error was thrown.

Hello,

It seems the most recent release (4.0.0) is not yet compatible with Django 3.0 while Master is.

Would it be possible to create a stable release which can be installed from Pypi?

Hello Ilya!

For example if I have a model that look like this:

class Profile(models.Model):
    name = models.CharField(max_length=40)
    doc = models.FieldField(upload_to='folder')

The documentation doesn't say how one should use the signals an not every projects need sorl-thumbnail

Please add tags to each version so we can better track them. (Only a repo committer can run these)

git tag -a 0.1.0 0fada14463 -m ''
git tag -a 0.1.4 6440f588d4 -m ''
git tag -a 0.1.5 57ec98ae31 -m ''
git tag -a 0.1.6 69a5840949 -m ''
git tag -a 0.1.7 c8f6484153 -m ''
git tag -a 0.1.8 031262d702 -m ''
git tag -a 0.1.9 6adeedba94 -m ''
git tag -a 0.1.10 715d0fcdfb -m ''
git tag -a 0.1.11 40b2983f53 -m ''
git tag -a 0.1.12 d9989d3f30 -m ''
git tag -a 0.1.13 a0901e54b1 -m ''
git tag -a 0.2.0 1902f18ca9 -m ''
git tag -a 0.2.1 bc4925d487 -m ''
git tag -a 0.3.0 6fb0e518fe -m ''
git tag -a 0.3.1 9c4ed988a4 -m ''
git tag -a 0.4.0 2bb0ffe48a -m ''
git push --tags

If you want to check, these should link to their commits:
0.1.0: 0fada14
0.1.4: 6440f58
0.1.5: 57ec98a
0.1.6: 69a5840
0.1.7: c8f6484
0.1.8: 031262d
0.1.9: 6adeedb
0.1.10: 715d0fc
0.1.11: 40b2983
0.1.12: d9989d3
0.1.13: a0901e5
0.2.0: 1902f18
0.2.1: bc4925d
0.3.0: 6fb0e51
0.3.1: 9c4ed98
0.4.0: 2bb0ffe

clean empty parent folders if not equal media, maybe ?

Hi, I have a model with a file field and a char field representing the name of the file.
on model.save() im renaming the file.name attribute to match the name attribute of the instance.
This causes file deletion when using django-cleanup.

Any way to prevent this?

Hello, Ilya!

We are using this lib in a while. In most cases it works perfect! Saves lots of space..

Resently my colleague found small issue with manual file removal.. If somewhere in code I already delete a file (or it is disappeared after django created an object) then .save() will raise FileNotFoundError

>>> obj = MyModel.object.all().first()
>>> obj.data.delete()
>>> obj.data = None
>>> obj.save()  # raises FileNotFoundError

May be it will be better to catch this exception inside this application? In case of, for example, using another library, which is know nothing about django_cleanup?

I'm ready to create pull request if you don't mind

I am using DRF to provide a REST API on top of Django. I recently noticed that Django does not automatically delete files on disk when the FileField is deleted, so I was hoping django-cleanup could address this.

setup.py:

INSTALLED_APPS = [
    'grappelli',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'rest_framework',
    'myapp.apps.MyAppConfig',
    'debug_toolbar',
    'guardian',
    'django_cleanup.apps.CleanupConfig',
]

models.py:

class Project(models.Model):
    pass

class MyModel(models.Model):
    project = models.ForeignKey(Project, related_name='my_models', on_delete=models.CASCADE)
    image = models.ImageField(upload_to='my_images')

But when I do a RESTful DELETE of an instance of Project, the files still don't delete?

The DRF ModelSerializer for Project deletes all of the Project and MyModel instances correctly (the database is empty), but the files remain.

Is there any configuration needed in MyAppConfig to associate the model? Is there a way to debug why django-cleanup doesn't seem to be getting called?

Wanted to check that this wasn't some kind of known limitation.

Discovered that TestCases must use TransactionalTestCase, in order for the library to work. Might be worth explicitly mentioning this.

I switched my project to use django-cleanup and it does things much better than my solution before, but one feature I don't have anymore is cleaning up empty directories after deleting a file.

This was my function for deleting File/Image:

def perform_media_delete(media_field):
    if os.path.isfile(media_field.path):
        os.remove(media_field.path)
        walk = list(os.walk(settings.MEDIA_ROOT))
        for path, _, _ in walk[::-1]:
            if len(os.listdir(path)) == 0:
                os.rmdir(path)
    else:
        return False

It removes all empty sub-directories relative to the MEDIA_ROOT location. This is useful when your upload_to path is something that follows the standards of using the first and second letter of the file name to build the directory:

/path/to/file/a/p/ap45bgfh3.png
--------------^ ^-^^

This feature may be undesirable for most, but is it something that can optional and disabled by default?

Strange kind of bug i came across.

When initiating a model instance, and then opening it in the admin page as if i were to modify it, simpli saving without making any changes prompts me with the following traceback:

There was an exception deleting the file `logo/6000_4000.webp` on field `accounts.restaurant.logo`
Traceback (most recent call last):

  File "C:\Users\sherzog\web_app\env\lib\site-packages\django_cleanup\handlers.py", line 96, in run_on_commit
    file_.delete(save=False)

  File "C:\Users\sherzog\web_app\env\lib\site-packages\django\db\models\fields\files.py", line 373, in delete
    super().delete(save)

  File "C:\Users\sherzog\web_app\env\lib\site-packages\django\db\models\fields\files.py", line 105, in delete
    self.storage.delete(self.name)

  File "C:\Users\sherzog\web_app\env\lib\site-packages\django\core\files\storage.py", line 304, in delete
    os.remove(name)

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\sherzog\\web_app\\backend\\media\\logo\\6000_4000.webp'

It seems it is interpreting this as a replacement of the original logo with the same one, or something like this. Should i be worried about this?

Saw indications that it works with Django 2.2? Is it approved for 2.2 and maintained?

It's a very useful tool, but it's dangerous (can be data-destructive), so I would only use it if it's maintained.

I have a model having ImageField (default to some image). When the ImageField is updated by any instance of the model, the default image is deleted. This causes problems when many instances of same model are linked with default image, and one of them updates.

Reference:
my project Develop branch, Profile model.

i use python 3.6 and dango 2.0 , i got some default image i would like to ignore in my models, is there a way please?

So I needed a solution to periodically clear out old database entries from my database, and that database also has attached files -- something django-cleanup is well suited for. The obvious practice is to write a script using delete() and attach a crontab job to it, but I initially tried python manage.py flush (which clears out the entire database anyways) figuring that would be a quicker solution. Interestingly django-cleanup didn't catch the database records being deleted and as a result didn't delete the old files. manage.py flush isn't likely to be part of a standard production routine (which is probably why it was overlooked) but it might be a good idea to add a small disclaimer to README.md warning that manage.py flush won't work with django-cleanup. Thanks for making django-cleanup!

I've discovered that if two models instances share a single file, and one of the instances is deleted, the file is also deleted. Obviously this creates an issue when trying to parse the model as the reference to the file remains, while the file does not.

I'm getting this issue when using django cassandra engine and run sync_cassandra command.

It traces back to this line, https://github.com/r4fek/django-cassandra-engine/blob/b43f8fddd4bba143f03f73f8bbfc09e6b32c699b/django_cassandra_engine/management/commands/sync_cassandra.py#L33

Would you be willing to add a license to this project so users of this project know how they are allowed to us it?

I make some test with s3 storage but not working
is there some code to add to make it work with s3 storage ?

I have a couple of default image files in my media folder, 'default_m.png' and 'default_f.png' which are assigned to the user as soon as the user is created.

When the user edits his/her profile and uploads a new profile image, the uploaded image is stored in media/profile/ and the default images, (media/default_m.png and media/default_f.png) are deleted.

I would like to exclude these two files from being deleted.

Curious as to the drop in support for 1.10

If you navigate here: https://travis-ci.org/github/un1t/django-cleanup

Since June 15th, 2021, the building on travis-ci.org is ceased. Please use travis-ci.com from now on.

This project will need to migrate to either travis-ci.com or GitHub Actions/Circle CI/etc.

Travis CI (.com) is no longer free for FOSS projects

• johnthagen/python-blueprint#36

I would personally recommend GitHub Actions. An example of how to set up GitHub Actions with tox:

• https://github.com/johnthagen/python-blueprint/blob/master/.github/workflows/python.yml
• https://github.com/johnthagen/python-blueprint/blob/6f0e28e740005a97c780507786acc4b02a94e6c2/tox.ini#L12

I'm trying it out for the first time, and noticed that while it does correctly delete the files when the model instance is deleted, it fails to delete files when they are replaced.

Hooking it up to a debugger and stepping through the code, it appears that in 'django_cleanup.handlers.delete_old_post_save, fetching the cached FileFieldreturns *the same object* as the "new"FileField. My suspicion is that in the past Django would create a new File` object when a file is replaced, but the current version instead updates the existing object. I suspect that the solution is to have the cache system store a copy of the field.

I'm going to run some experiments and report back.

I'm using django 3.2.
Assume we have a Image model, which includes a ImageField to save image.
On my local computer, if I create two Image objects and save image.jpg twice, it will produce two files.

• image.jpg
• image_ABCDE.jpg

However, on google cloud storage. https://django-storages.readthedocs.io/en/latest/backends/gcloud.html
Storing image.jpg twice will only produce one file.

• image.jpg

As a result, if I delete one Image object, another Image object cannot find the image.jpg file.
It would be nice if django-cleanup can provide an option to explicitly scan the uniqueness of the file.

When using with a FileField in Django 4.1 neither automatic nor manual cleanup works.

Files still remaining after clearing the file or updating with a new file.

Calling cleanup.refresh(model_instance) does not work either.

When I use django admin I validate some models because of special requirements. But problem si that django_cleanup deletes files before model is validated. So valid files are removed but model is not saved.
I use clean() method in model to do my validation.

The post_save signal mess with instance.field_name value when deleting the old file:

>>> product = Product.objects.create(image=image1)
>>> product.image = 'new.jpg'
>>> product.save()
>>> product.image
None

since 0.4.0 release

Not sure if it's related to a recent upgrade to django 1.10 but there is an infinite recursion between django-cleanup and django modeltranslations.

django-cleanup==0.4.2
django-modeltranslation==0.12
Django==1.10

File "/site/env/python/local/lib/python2.7/site-packages/django/db/models/base.py", line 557, in init
signals.post_init.send(sender=self.class, instance=self)
File "/site/env/python/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 191, in send
response = receiver(signal=self, sender=sender, **named)
File "/site/env/python/local/lib/python2.7/site-packages/django_cleanup/handlers.py", line 44, in cache_original_post_init
cache.make_cleanup_cache(instance)
File "/site/env/python/local/lib/python2.7/site-packages/django_cleanup/cache.py", line 165, in make_cleanup_cache
fields_for_model_instance(source, using=instance)))
File "/site/env/python/local/lib/python2.7/site-packages/django_cleanup/cache.py", line 108, in fields_for_model_instance
field = get_field_instance(instance, field_name, using=using)
File "/site/env/python/local/lib/python2.7/site-packages/django_cleanup/cache.py", line 82, in get_field_instance
field = getattr(instance, field_name, None)
File "/site/env/python/local/lib/python2.7/site-packages/django/db/models/fields/files.py", line 173, in get
instance.refresh_from_db(fields=[self.field.name])
File "/site/env/python/local/lib/python2.7/site-packages/modeltranslation/translator.py", line 304, in new_refresh_from_db
return old_refresh_from_db(self, using, fields)
File "/site/env/python/local/lib/python2.7/site-packages/django/db/models/base.py", line 685, in refresh_from_db
db_instance = db_instance_qs.get()
File "/site/env/python/local/lib/python2.7/site-packages/django/db/models/query.py", line 379, in get
num = len(clone)
File "/site/env/python/local/lib/python2.7/site-packages/django/db/models/query.py", line 238, in len
self._fetch_all()
File "/site/env/python/local/lib/python2.7/site-packages/django/db/models/query.py", line 1085, in _fetch_all
self._result_cache = list(self.iterator())
File "/site/env/python/local/lib/python2.7/site-packages/django/db/models/query.py", line 66, in iter
obj = model_cls.from_db(db, init_list, row[model_fields_start:model_fields_end])
File "/site/env/python/local/lib/python2.7/site-packages/django/db/models/base.py", line 565, in from_db
new = cls(*values)
File "/site/env/python/local/lib/python2.7/site-packages/modeltranslation/translator.py", line 245, in new_init
old_init(self, *args, **kwargs)
File "/site/env/python/local/lib/python2.7/site-packages/django/db/models/base.py", line 557, in init
signals.post_init.send(sender=self.class, instance=self)

If two instances have the same ImageField, deleting on of them causing file deletion.

Hello!

It is possible to mark certain files to be ignored by this app? I am currently using it to cleanup an avatar gallery, but I wish to have some default avatars as well, that should not be deleted. However, when the User changes their avatar or the User is deleted, the default avatar is deleted as well.

By default, django-cleanup works by cleaning all models unless you explicitly ignore them with @cleanup.ignore. I'm wondering if django-cleanup could work in such a way that it doesn't cleanup any models unless you explicitly ask it to?

For example:

@cleanup.cleanup
class UserImage(models.Model):
    image = models.FileField()

Not sure if this should be in scope, but what would be the best approach to order generated thumbnails?
I am using https://github.com/SmileyChris/easy-thumbnails

Thanks for a nice library!

Production environment : Django1.9.7 / AWS S3, AWS Cloudfront (for managing static and media files)

cleanup_pre_delete is not called in production environment.

But it called very well in local development environment.

I have no idea why this happens.

Hi,
The Django documentation mentions this method for copying Model-Instance: https://docs.djangoproject.com/en/4.0/topics/db/queries/#copying-model-instances as per default Django behaviour this results in a non-unique filename. If one of them is afterwards adjusted, #51 takes place and The file is deleted from the other copy. Could you please mention in your documentation/enforce in some way Uniqueness in FileFields so that Files with the same name are not deleted as easily? :)

Hi,
I am having trouble to understand the thumbnail example.
My case is that I have a class CV which has a file attribute - type : FileField.this file is stored in S3

Now I would like to send signal pre_delete to the model before its instance is going to be deleted.
My issue is that I don't know how to import the CV model into the function.
What i have tried.
from django_cleanup.signals import cleanup_pre_delete, cleanup_post_delete

def CV(models.Model):
...
file = models.FileFiles(path)

def cv_file_delete(**kwargs):
from CV import delete // this is raising error
delete(CV.file)
cleanup_pre_delete.connect(cv_file_delete)

[...]/lib/python2.7/site-packages/django_cleanup/models.py", line 3, in
from django.apps import apps
ImportError: No module named apps

I found that my device still would contain files which are created in a atomic nested post request (using drf with multiple nested models), WHEN an error occurs during the request after the files have been written and a database rollback is triggered because after all I don't want to have a partial instance created.

My current workaround is to overwrite the save method in my FileSerializer and store the information about the created files so I can manually delete them in a try+except wrapper later:

class Singleton(type):
	_instances = {}

	def __call__(cls, *args, **kwargs):
		if cls not in cls._instances:
			cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
		return cls._instances[cls]


class FileCreationHistory(metaclass=Singleton):
	createdFiles = []

	@staticmethod
	def reset():
		FileCreationHistory.createdFiles = []


class FileSerializer(ModelSerializer):
	scenery = PrimaryKeyRelatedField(allow_null=True, required=False, queryset=Scenery.objects.all())

	class Meta:
		model = File
		fields = ['file', 'scenery']

	def save(self, **kwargs):
		result = super(FileSerializer, self).save(**kwargs)
		FileCreationHistory.createdFiles.append(result.file)
		return result

# WritableNestedModelSerializer is from the drf_writable_nested package
class ScenerySerializer(WritableNestedModelSerializer):

	# ... more child fields and serializers here...

	# Reverse foreign key
	files = FileSerializer(many=True)

	class Meta:
		model = Scenery
		fields = [
			'title',
			'description',
			'files'
			# ... more fields here...
		]

	def save(self, **kwargs):
		FileCreationHistory.reset()
		try:
			with transaction.atomic():
				self._save_kwargs = defaultdict(dict, kwargs)
				return super(BaseNestedModelSerializer, self).save(**kwargs)
		except Exception as e:
			"""
				Whatever errors occurred during the creation of the Scenery and its children:
				We have to properly delete all files which have been placed at the local storage device, because
				at this point they are deleted from the db, but are still on the hard drive.
				django-cleanup does not work with db rollbacks as it seems. So we use a singleton file creation history here.
			"""
			for file in FileCreationHistory.createdFiles:
				file.delete(save=False)  # save=False because the instance probably does not exist anymore at this point.
			FileCreationHistory.reset()
			raise e

This implementation is not thread-safe (yet), just wanted to share if someone runs into the same issue or if the dev could find a solution to this problem.

Hey. Django-cleanup seems to break django-admin-sortable2 (or vice versa?)

When reordering some models in the Admin, the AJAX request sent will fail with HTTP 500. I am not quite sure what is happening, but fallback_pre_save (part of django-cleanup) is called at some point, resulting in the following error:

TypeError at /admin/main/trailers/adminsortable2_update/
fallback_pre_save() missing 2 required positional arguments: 'raw' and 'using'

Not sure what app actually needs its behaviour fixed, maybe @jrief can jump on the issue too.

Full traceback:

ERROR Unexpected exception while attempting to delete file ''
Traceback (most recent call last):
  File "/home/ubuntu/.virtualenvs/home/local/lib/python2.7/site-packages/django_cleanup/models.py", line 62, in delete_file
    storage.delete(file_.name)
  File "/home/ubuntu/.virtualenvs/home/local/lib/python2.7/site-    packages/django/core/files/storage.py", line 231, in delete
    assert name, "The name argument is not allowed to be empty."
AssertionError: The name argument is not allowed to be empty.

If previously a file does not exist in the model instance, and a new one is being uploaded, the old_file gets the value None, and it is sent for deletion without checking whether that file actually exists or not.

if old_file != new_file:
        delete_file(old_file)       # in L36 in models.py `

And storage.exists(file_.name) comes out to be true for empty name.

I'm not sure if it's possible to fix this, but I found an issue with a transaction rollback. Basically, when a a model with an image is created and saved inside a transaction, and the transaction then gets rollbacked, the data is not in the db table, but the image is still stored on disk.

I found this issue while writing tests. When I use django.test.TestCase, and work with models, at the end of the test django rollbacks the changes, but the images created during instance.save() calls are still on the disk, which prevents me from running multiple tests in parallel, because I would have to manually delete the files at the end of each test, which would be concurrent. Also, It's not good in an app if you're using db transactions.

Demo code:

try:
    with transaction.atomic():
        self.create_model_with_image()
        raise ValidationError('INVALID')
except ValidationError as e:
    print(e)

If a file is saved in multiple instances, then removing a single instance removes the file and cause 'file not found' error for every other instance.

We need to check if the file is referenced in any other object before deleting.