antonagestam / collectfast Goto Github PK

I have include license file in the manifest.in file. Could you please make sure it gets into effect in future releases, otherwise we need to add license manually while building it.

I use S3 for production, but on my dev machine I use STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.StaticFilesStorage'.

If I have Collectfast installed on my dev machine I get this traceback:

./manage.py collectstatic --noinput
Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/core/management/base.py", line 415, in handle
    return self.handle_noargs(**options)
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/collectfast/management/commands/collectstatic.py", line 147, in handle_noargs
    collected = self.collect()
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/collectfast/management/commands/collectstatic.py", line 39, in collect
    ret = super(Command, self).collect(*args, **kwargs)
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 111, in collect
    handler(path, prefixed_path, storage)
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/collectfast/management/commands/collectstatic.py", line 80, in copy_file
    normalized_path = self.storage._normalize_name(path)
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/utils/functional.py", line 214, in inner
    return func(self._wrapped, *args)
AttributeError: 'StaticFilesStorage' object has no attribute '_normalize_name'

I'm trying to put my static files and uploaded media files in separate folders within my S3 bucket. In order to do that, I'm setting the following:

DEFAULT_FILE_STORAGE = 'myapp.s3utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE = 'myapp.s3utils.StaticRootS3BotoStorage'

s3utils.py is defined as:

from storages.backends.s3boto import S3BotoStorage

StaticRootS3BotoStorage = lambda: S3BotoStorage(location='static')
MediaRootS3BotoStorage = lambda: S3BotoStorage(location='media')

That causes a problem in get_lookup(path) with the following code:

if cached is False:
    self.lookups[path] = self.storage.bucket.lookup(path)
    cache.set(cache_key, self.lookups[path])
else:
    self.lookups[path] = cached

The problem is that 'path' doesn't exist. The lookup always fails with a 404 not found error. The path needs to be prefixed with the location ('static'). S3BotoStorage has a method '_normalize_name(name)' which does exactly that.

I'm not sure how to fix this generically, but if you change the code to the following, it works just fine:

if cached is False:
    self.lookups[path] = self.storage.bucket.lookup(self.storage._normalize_name(path))
    cache.set(cache_key, self.lookups[path])
else:
    self.lookups[path] = cached

The Exception.message attribute was removed in python 3 (and deprecated in python 2.6), thus in python 3 a double exception occurs.

At some point, line 38 of collectstatic.py should use e instead of e.message.

Hi,

I am getting following error UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 65: ordinal not in range(128) the full traceback is below:

Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 359, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 294, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 345, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/dist-packages/collectfast/management/commands/collectstatic.py", line 70, in handle
    super(Command, self).handle(**options)
  File "/usr/local/lib/python2.7/dist-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 193, in handle
    collected = self.collect()
  File "/usr/local/lib/python2.7/dist-packages/collectfast/management/commands/collectstatic.py", line 63, in collect
    Pool(settings.threads).map(self.do_copy_file, self.tasks)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 567, in get
    raise self._value
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 65: ordinal not in range(128)

Looking at traceback I am not sure from which file it is complaining about the error. Please help.

Environment:
- Python 2.7.12
- Ubuntu 16.04.2 LTS

Packages:
- Django==1.10.5
- Collectfast==0.5.2
- boto3==1.4.4

The output of locale command:

LANG=en_US.UTF-8
LANGUAGE=en_US:en
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8

Thanks!

Boto v3 support was recently added to django-storages (jschneier/django-storages#57), but when using collectfast with boto v3, an error is generated when calling collectstatic: AttributeError: 's3.Bucket' object has no attribute 'lookup'

This causes the etag comparison to fail and all files to be recopied every time.
See pull request #27

Running into error during post-processing

django/contrib/staticfiles/storage.py", line 101, in hashed_name:    
raise ValueError("The file '%s' could not be found with %r." % (filename, self))

I believe this happens because the underlying collect() command also runs post-process. But because this library overrides collects such that uploads don't happen until the underlying collection function completes. The image has not actually been uploaded yet.

Happy to submit PR to delay the post-process function

I'm using heroku and django-storage
so say I modified myclass.css, push to heroku, run collectstatics
I would get a myclass_1.css on s3

Please help!

Django is 1.10.7,

Traceback (most recent call last):
  File "manage.py", line 29, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 359, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 208, in fetch_command
    klass = load_command_class(app_name, subcommand)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 41, in load_command_class
    return module.Command()
  File "/usr/local/lib/python3.5/site-packages/collectfast/management/commands/collectstatic.py", line 35, in __init__
    if self.storage.preload_metadata is not True:
  File "/usr/local/lib/python3.5/site-packages/django/utils/functional.py", line 235, in inner
    return func(self._wrapped, *args)
AttributeError: 'StaticFilesStorage' object has no attribute 'preload_metadata'

Is there a simple way to purge cached values? I by mistake added a bunch with no expiration to the wrong cache and want to clear them.

Setting AWS_PRELOAD_METADATA = True is not always an option, because it makes some commands in the S3BotoStorage really slow. But when it's not enabled, the command does not work. So, preload_metadata should be set when it's not set or set to false when running the command.

Relevant:
https://bitbucket.org/david/django-storages/issue/106/preload-breaks-things-like-exist
AGoodId/django-s3-collectstatic#2

If AWS_IS_GZIPPED is set to True, then this will check the hash of the un-gzipped item against the etag of the gzipped item, which will obviously not match (though once upon a time it used to, but amazon fixed that).

This means all js and css files on a set-up using gzipping will be copied on every collect.

There should be an option to cache the ETag of the remote files to further decrease deploy time.

I have used this library before with one of my S3 projects.
Recently we have moved a project to GCP storage. Collectstatic from a regular server is excruciatingly slow.
I need the benefits of this library for a GCP backend as well.
Can I expect any such changes in the near future?

Log errors that are ignored by the try-except in copy_file (

Is this package compatible with Django 2.1 & 2.2? Anyone using it in production? If so, we should add these versions to tox.ini and travis.yml

If you don't have boto in your environment (only boto3) has_boto flag will be set to false in boto.py due to an import error and that will fail the condition check to set preload_metadata to True in the initialization of the mgmt command.
As a consequence collectfast will be slow

Running pip install collectfast fails because the long description is set to

long_description=open('README.md').read()

but README.md isn't included in the package.

With 1.7 files will get added to S3 but never updated. This may be an django-storages issue though.

Hi @antonagestam -- what is the roadmap for getting ready for a 0.1.15 release?

There's an informational logging message in the etag implementation:

This is shown if collectstatic is run with verbosity 0 and should not be. This can be fixed in the management command by using logging.disable() if verbosity is zero.

I pip installed the Pypi version, but it still has the last_modified checking in it, and in fact appears to have a bug with that checking mechanism (it skips the upload when local_modified > storage_modified instead of when local_modified < storage_modified , which is working oppositely for me). I notice you've since merged a Pull Request to take out that check all together, with good reason I'm sure.

Anyhow, this is a great library--its taken my collectstatic times down from 10-15 minutes to 2 minutes, which is a huge boost--but its always nice to be able to install a stable version directly from PyPi so updating the PyPi package would be very helpful.

Looks like pypi is current, its just the readme that told me to install the wrong version (0.1.9 instead of 0.1.10)

Introduce setting for which cache to use and whether or not to use cache at all.

Re-running CI on 0.6.2 results in errors

https://travis-ci.org/jayvdb/collectfast/builds/495173846

======================================================================
ERROR: collectfast.tests.test_command (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: collectfast.tests.test_command
Traceback (most recent call last):
  File "/opt/python/3.6.3/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/opt/python/3.6.3/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/home/travis/build/jayvdb/collectfast/collectfast/tests/test_command.py", line 7, in <module>
    from .utils import test, clean_static_dir, create_static_file, override_setting
  File "/home/travis/build/jayvdb/collectfast/collectfast/tests/utils.py", line 10, in <module>
    import boto
  File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/site-packages/boto/__init__.py", line 1216, in <module>
    boto.plugin.load_plugins(config)
  File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/site-packages/boto/plugin.py", line 93, in load_plugins
    _import_module(file)
  File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/site-packages/boto/plugin.py", line 75, in _import_module
    return imp.load_module(name, file, filename, data)
  File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/imp.py", line 235, in load_module
    return load_source(name, filename, file)
  File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/imp.py", line 172, in load_source
    module = _load(spec)
  File "/usr/lib/python2.7/dist-packages/google_compute_engine/boto/compute_auth.py", line 19, in <module>
    from google_compute_engine import logger
ModuleNotFoundError: No module named 'google_compute_engine'
======================================================================
ERROR: collectfast.tests.test_etag (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: collectfast.tests.test_etag
Traceback (most recent call last):
  File "/opt/python/3.6.3/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/opt/python/3.6.3/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/home/travis/build/jayvdb/collectfast/collectfast/tests/test_etag.py", line 7, in <module>
    from storages.backends.s3boto import S3BotoStorage
  File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/site-packages/storages/backends/s3boto.py", line 25, in <module>
    from boto import __version__ as boto_version
  File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/site-packages/boto/__init__.py", line 1216, in <module>
    boto.plugin.load_plugins(config)
AttributeError: module 'boto' has no attribute 'plugin'

When I try to run it on openSUSE build environment, with latest moto, I get different errors:

======================================================================
ERROR: test_basics (collectfast.tests.utils.test_basics)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/moto/core/models.py", line 74, in wrapper
    result = func(*args, **kwargs)
  File "collectfast/tests/utils.py", line 33, in wraps
    boto.connect_s3().create_bucket(django_settings.AWS_STORAGE_BUCKET_NAME)
  File "/usr/lib/python2.7/site-packages/boto/__init__.py", line 141, in connect_s3
    return S3Connection(aws_access_key_id, aws_secret_access_key, **kwargs)
  File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 194, in __init__
    validate_certs=validate_certs, profile_name=profile_name)
  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 569, in __init__
    host, config, self.provider, self._required_auth_capability())
  File "/usr/lib/python2.7/site-packages/boto/auth.py", line 1021, in get_auth_handler
    'Check your credentials' % (len(names), str(names)))
NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV1Handler'] Check your credentials

When i try to do an upload of a file to s3 and get its etag, i get this error

  File "/venv3/lib/python3.5/site-packages/collectfast/management/commands/collectstatic.py", line 84, in get_boto3_etag
    return self.storage.bucket.Object(path).e_tag
AttributeError: 'Bucket' object has no attribute 'Object'

Django = 1.10.1
python = 3.5

Hello!

When using collectfast with threads, there is a problem with call_command:

from django.core.management import call_command

def update():
    call_command('collectstatic', interactive=False, verbosity=3)
    return 0

This immediately return 0 and running threads are killed.
We need to make sure that collectstatic is a blocking call until all work is done.

Greets
Manuel

For an S3 storage class using the HashedFilesMixin (e.g. CachedFilesMixin or ManifestFilesMixin), a ValueError will be raised when collectstatic is run with an empty S3 bucket (or more precisely, a bucket lacking any one of the original static files):

ValueError: The file 'mypath/myfont.eot' could not be found with <myapp.MyStorageClass object at 0x7fa786804978>.

This only occurs when COLLECTFAST_THREADS is not False. When the HashedFilesMixin.hashed_name method uses exists() to determine whether the file for the cleaned file name exists (on S3), a ValueError is raised, since it doesn't. i assume this is due to a race condition.

At first i thought that overriding hashed_name to compute based on file contents of local static media would fix that issue (and be faster?), but for third-party apps, that raises a SuspiciousFileOperation.

Getting a traceback with 0.2.0.

[localhost] local: foreman run ./manage.py collectstatic --noinput
Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/core/management/base.py", line 415, in handle
    return self.handle_noargs(**options)
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/collectfast/management/commands/collectstatic.py", line 178, in handle_noargs
    collected = self.collect()
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/collectfast/management/commands/collectstatic.py", line 60, in collect
    ret = super(Command, self).collect()
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 111, in collect
    handler(path, prefixed_path, storage)
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/collectfast/management/commands/collectstatic.py", line 135, in copy_file
    path, prefixed_path, source_storage)
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 294, in copy_file
    if not self.delete_file(path, prefixed_path, source_storage):
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/collectfast/management/commands/collectstatic.py", line 141, in delete_file
    path, prefixed_path, source_storage)
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 224, in delete_file
    self.storage.modified_time(prefixed_path)
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/storages/backends/s3boto.py", line 334, in modified_time
    return _parse_datestring(entry.last_modified)
  File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/storages/backends/s3boto.py", line 119, in _parse_datestring
    raise ValueError("Could not parse date string: " + dstr)
ValueError: Could not parse date string: 2014-04-11T20:21:13.000Z

Fatal error: local() encountered an error (return code 1) while executing 'foreman run ./manage.py collectstatic --noinput'

Aborting.

Hi,

Thanks for publishing this project. It is very helpful :-)

I've encountered the following issue:

When using Collectfast in disabled mode (for local development without S3), running ./manage.py collectstatic breaks even with --disable-collectfast or COLLECTFAST_ENABLED = False. The reason seems to be the following:

When using filesystem storage, collectstatic crashes with the following:

Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
    utility.execute()
  File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 365, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 216, in fetch_command
    klass = load_command_class(app_name, subcommand)
  File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 37, in load_command_class
    return module.Command()
  File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/collectfast/management/commands/collectstatic.py", line 36, in __init__
    if self.storage.preload_metadata is not True:
  File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/django/utils/functional.py", line 216, in inner
    return func(self._wrapped, *args)
AttributeError: 'StaticFilesStorage' object has no attribute 'preload_metadata'

This happens because StaticFilesStorage doesn't have an attribute called preload_metadata (I believe it is S3-specific).

To reproduce this issue, simply run ./manage.py collectstatic with collectfast in INSTALLED_APPS and a default storage configuration.

I believe this can be solved by something similar to the following:

from storages.backends.s3boto3 import S3Boto3Storage
...
if isinstance(self.storage, S3Boto3Storage) and self.storage.preload_metadata is not True:
    ...

The following is also possible to avoid the import:

if (type(self.storage).__name__ == 'storages.backends.s3boto3.S3Boto3Storage'
    and self.storage.preload_metadata is not True):

What do you think?

Thanks,
Johanan

I'm frequently seeing unspecified, hard-to-identify (let alone debug) errors when running collectstatic using Collectfast. Is there any way to get the errors to print so I can find out what's going on?

These occur within my build process on CircleCI, which adds another layer of complexity.

Hi, thanks for building this promising library. I'm looking forward to using it.

My app uses the default value for STATICFILES_FINDERS

STATICFILES_FINDERS = (
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
)

When asset are collected into static/, I have these files:

$ ls -alh static
total 4200
drwxr-xr-x  16 ben  staff   512B Jan 24 15:49 .
drwxr-xr-x  61 ben  staff   1.9K Jan 25 10:59 ..
-rw-r--r--   1 ben  staff   669B Jan 24 15:49 500.html
drwxr-xr-x   6 ben  staff   192B Jan 24 15:49 admin
drwxr-xr-x   5 ben  staff   160B Jan 24 15:49 debug_toolbar
-rw-r--r--   1 ben  staff   209B Jan 24 15:49 demo_config.yaml
-rw-r--r--   1 ben  staff   671K Jan 24 15:49 demo_deed.pdf
-rw-r--r--   1 ben  staff   101K Jan 24 15:49 demo_statement.pdf
drwxr-xr-x   5 ben  staff   160B Jan 24 15:49 django_extensions
drwxr-xr-x   5 ben  staff   160B Jan 24 15:49 gis
drwxr-xr-x  13 ben  staff   416B Jan 24 15:49 images
-rw-r--r--   1 ben  staff    23B Jan 24 15:49 robots.txt

There are files (500.html, demo_config.yaml) and there are directories (debug_toolbar, images, gis). When I use Collectfast, only the files are uploaded to S3.

$ ./manage.py collectstatic --settings=settings_local --no-input --clear
Copying '/Users/ben/code/minerals/core/static/500.html'
Copying '/Users/ben/code/minerals/core/static/robots.txt'
Copying '/Users/ben/code/minerals/core/static/demo_deed.pdf'
Copying '/Users/ben/code/minerals/core/static/demo_statement.pdf'
4 static files copied.

I expected the files and directories to be uploaded to S3, not just the files. When I use --disable-collectfast, causing storages.backends.s3boto3.S3Boto3Storage to be used directly, the files and directories are uploaded to S3.

$ ./manage.py collectstatic --settings=minerals.settings_local --no-input --clear --disable-collectfast
...(lots of copying of a bunch of files)...
226 static files copied.

It's surprising behavior that Collectfast doesn't upload the directories like S3Boto3Storage does when used directly (I think I also tried with S3BotoStorage). Is there a way to get Collectfast to upload the directories?

Here's my config...

STATICFILES_FINDERS = (
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
)

# I verified that the /var/tmp/collectfast_cache directory is being used as expected
CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://127.0.0.1:6379/2",
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
        }
    },
    'collectfast': {
        'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache',
        'LOCATION': '/var/tmp/collectfast_cache',
        'TIMEOUT': None,
        'OPTIONS': {
            'MAX_ENTRIES': 1000,
        },
    }
}

# Collectfast
STATICFILES_STORAGE = "storages.backends.s3boto3.S3Boto3Storage"
AWS_PRELOAD_METADATA = True
COLLECTFAST_THREADS = 20
COLLECTFAST_DEBUG = True
COLLECTFAST_CACHE = 'collectfast'

# I've also setup Boto3 settings correctly (`AWS_STORAGE_BUCKET_NAME`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_QUERYSTRING_EXPIRE`)

Thanks for any help, and thanks for the time you spend on this library. I was surprised at how slow collectstatic was when used with ManifestStaticFilesStorage and S3Boto3Storage, so I was very excited to find this library.

Here are my relevant settings:

INSTALLED_APPS += ['storages', 'collectfast', ]
AWS_STORAGE_BUCKET_NAME = os.environ['AWS_STORAGE_BUCKET_NAME']
AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_ACCESS_KEY']
DEFAULT_FILE_STORAGE = 's3utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE = 's3utils.StaticRootS3BotoStorage'
COMPRESS_STORAGE = 's3utils.StaticRootS3BotoStorage'
AWS_PRELOAD_METADATA = True

When I run ./manage.py collectstatic I can see that it uploads some files. Then I abort the command via CTRL+C and run it immediately again. I can see that it uploads the same files again.

Is there any way for me to debug this?

I messed up and accidentally had the INSTALLED_APPS order wrong. When I was trying to determine which collectstatic was working I tried using python manage.py collectstatic --version. However, even after I fixed the issue it was still reporting my Django version instead of the version of Collectfast.

Yes, it's a really small and pedantic thing, but --version should be reporting the version of Collectfast.

The readme.rst contains non-ascii chars which fails the setup on python 3 if i install it via pip.

It would work if I do open with encoding='utf-8', but that would fail python 2 i assume (I haven't used python 2 since years)?

Hi ,

I'm trying to move my Django project (which uses collectstatic) from python 2.7 to python 3.6.

When running collectstatic from the command line I get an error (see further). As everything else works fine, I'm a bit lost on what went wrong here.

I did not change my settings.py and all worked fine under python 2.7

Thanks for pointing me in the right direction!

Regards,
Joris

The error:

(platform)baze@baze-mintbox:~/dev/www/projects/ieper/platform > python manage.py collectstatic

Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
    utility.execute()
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/core/management/__init__.py", line 355, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/management/commands/collectstatic.py", line 70, in handle
    super(Command, self).handle(**options)
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 199, in handle
    collected = self.collect()
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/management/commands/collectstatic.py", line 61, in collect
    ret = super(Command, self).collect()
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 124, in collect
    handler(path, prefixed_path, storage)
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/management/commands/collectstatic.py", line 109, in copy_file
    self.do_copy_file(args)
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/management/commands/collectstatic.py", line 86, in do_copy_file
    self.storage, path, prefixed_path, source_storage):
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/etag.py", line 93, in should_copy_file
    remote_storage, source_storage, path, prefixed_path):
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/etag.py", line 83, in has_matching_etag
    storage_etag = get_etag(remote_storage, path, prefixed_path)
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/etag.py", line 58, in get_etag
    etag = get_remote_etag(storage, prefixed_path)
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/etag.py", line 38, in get_remote_etag
    normalized_path = storage._normalize_name(
  File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/utils/functional.py", line 239, in inner
    return func(self._wrapped, *args)
AttributeError: 'StaticFilesStorage' object has no attribute '_normalize_name'\

For reference, my pip freeze output:

boto==2.46.1
boto3==1.4.4
botocore==1.5.40
certifi==2017.4.17
chardet==3.0.4
Collectfast==0.5.2
defusedxml==0.5.0
dj-database-url==0.4.2
Django==1.11
django-allauth==0.32.0
django-postgrespool==0.3.1
django-storages==1.5.2
docutils==0.13.1
futures==3.1.1
gunicorn==19.7.1
idna==2.5
jmespath==0.9.2
newrelic==2.86.3.70
oauthlib==2.0.2
psycopg2==2.7.1
python-dateutil==2.6.0
python3-openid==3.1.0
pytz==2017.2
requests==2.17.3
requests-oauthlib==0.8.0
s3transfer==0.1.10
six==1.10.0
SQLAlchemy==1.1.10
urllib3==1.21.1

I use the follow code so that I could set the prefix of the location to be the same as the one on the local directory:

from django.conf import settings
from storages.backends.s3boto import S3BotoStorage

class StaticStorage(S3BotoStorage):
    location = settings.STATICFILES_LOCATION
    def path(self, name):
        return self.url(name)

class MediaStorage(S3BotoStorage):
    location = settings.MEDIAFILES_LOCATION
    def path(self, name):
        return self.url(name)
~

Prior to 0.5.0, the location will only affect the result directory on s3, but now it tries to look for the static files while taking this variable into account.

For example. If STATICFILES_LOCATION = "static"
In the past, the search path would have been "/static/js/webpack.js"
Now it is "/static/static/js/webpack.js"

this variable location should affect where Collectfast is looking for files. This happens not only to the files under /static but for other static files, such as admin related things. There is an extra "/static" in the path.
This was reproduced in 0.5.1 too

The source distribution isnt testable as-is.

Ideally one of the following should work.

python setup.py test
python `which django-admin` test

I see this project uses tox and runtests.py.

The files to support that are not included in the sdist.

trying out collectfast with django-minio-storage, I stumbled over collectfast.etag.get_remote_etag() causing an
AttributeError: 'MinioStaticStorage' object has no attribute '_normalize_name'
which can easily be replaced by the public storages.utils.safe_join() which is called by storages.backends.s3boto3.S3Boto3Storage._normalize_name()

I really like the option of not having to collect statics that haven't changed, I'd love if this became an option to the collectsatic command itself.

So I was asking myself if you ever discussed that on django-dev?

Thanks for the project!

Current output:

…
Skipping 'dajax/mootools.dajax.core.js'
Skipping 'dajax/jquery.dajax.core.js'
Skipping 'dajax/prototype.dajax.core.js'
Skipping 'dajax/dojo.dajax.core.js'
Copying '/tmp/tmp9FnPsc'

7 static files copied.

Should be less verbose and instead say Skipped 4 files, 7 static files copied.

Collectfast only ever makes a single cache entry on Windows, meaning I can't collectstatic from my development machine.

I believe I have traced this to the use of storage._normalize_name(prefixed_path), which returns a Windows style directory structure with double backslashes instead of forward slashes, i.e. it doesn't normalize paths.

I'll see if there's anything I can do to patch it and I'll create a pull request, unless someone come up with a better place to fix it.

Traceback (most recent call last):
File "", line 17, in
File "/home/abal/isocutter/build/collectfast/setup.py", line 10, in
long_description=open('README.rst').read(),
File "/home/abal/isocutter/lib/python3.4/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 12: ordinal not in range(128)
Complete output from command python setup.py egg_info:
Traceback (most recent call last):

File "", line 17, in

File "/home/abal/isocutter/build/collectfast/setup.py", line 10, in

long_description=open('README.rst').read(),

File "/home/abal/isocutter/lib/python3.4/encodings/ascii.py", line 26, in decode

return codecs.ascii_decode(input, self.errors)[0]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 12: ordinal not in range(128)

This commit adds an import of S3Boto3Storage with no fallback, which will fail with "django.core.exceptions.ImproperlyConfigured: Could not load Boto3's S3 bindings." if boto3 is not installed.

So long as the intention is to continue to support boto as well as boto3, this import should be conditional.

What:
Collectfast==0.6.0 and boto3==1.5.8 don't play well when deploying to Heroku, resulting in

Traceback (most recent call last):
remote:          File "manage.py", line 30, in <module>
remote:            execute_from_command_line(sys.argv)
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
remote:            utility.execute()
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 356, in execute
remote:            self.fetch_command(subcommand).run_from_argv(self.argv)
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 206, in fetch_command
remote:            klass = load_command_class(app_name, subcommand)
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 40, in load_command_class
remote:            module = import_module('%s.management.commands.%s' % (app_name, name))
remote:          File "/app/.heroku/python/lib/python3.6/importlib/__init__.py", line 126, in import_module
remote:            return _bootstrap._gcd_import(name[level:], package, level)
remote:          File "<frozen importlib._bootstrap>", line 994, in _gcd_import
remote:          File "<frozen importlib._bootstrap>", line 971, in _find_and_load
remote:          File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
remote:          File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
remote:          File "<frozen importlib._bootstrap_external>", line 678, in exec_module
remote:          File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/collectfast/management/commands/collectstatic.py", line 8, in <module>
remote:            from collectfast.etag import should_copy_file
remote:          File "/app/.heroku/python/lib/python3.6/site-packages/collectfast/etag.py", line 10, in <module>
remote:            from storages.utils import safe_join
remote:        ImportError: cannot import name 'safe_join'
remote: 
remote:  !     Error while running '$ python manage.py collectstatic --noinput'.
remote:        See traceback above for details.

I'm also using:

django-storages==1.5.0  # pyup: ==1.5.0

Solution:
Downgrade

Collectfast==0.5.2 # pyup: ==0.5.2

I realize its not a collectfast problem, but I spent a long time trying to figure out why caching was only working for a (seemingly random) portion of my static files. Eventually I found the MAX_ENTRIES default setting for Django caches, which is set to 300 by default (I think this only applies to DB or File Based caching, which is what i'm using for my CollectFast cache).

It may be useful for some users to call out the bad default on the "MAX_ENTRIES" option when you call out "TIMEOUT" in the README, since they probably won't be familiar with it and it will almost always reduce the effectiveness of CollectFast if used with affected cache types, since only 300 file hashes will be cached

I apologize in advance if this isn't an issue directly related to this library, but I can't figure out what's going on. I'm testing this out on FreeBSD 11 with memcached as a backend:

python36-3.6.1_4
libmemcached-1.0.18_6
memcached-1.4.34_2

boto==2.47.0
Collectfast==0.5.2
Django==1.11.2
django-storages==1.5.2
pylibmc==1.5.2

I have collectfast listed before django.contrib.staticfiles in INSTALLED_APPS, a dedicated COLLECTFAST_CACHE setup using pylibmc, and COLLECTFAST_THREADS set anywhere from 2 to 20. memcached is started with flags -m 1024 -I 128m.

Using collectstatic --disable-collectfast, the static files are copied to the S3 bucket without any problems. When I try it with collectfast enabled, I get all sorts of memcached errors:

• error 21 from memcached_set
• error 9 from memcached_set: (0x80e83d000) bad command line format, host: 127.0.0.1:11211
• error 11 from memcached_set
• error 31 from memcached_get(fastcache_:1:collectfast05_asset): (0x80ea3d000) A TIMEOUT OCCURRED, host: 127.0.0.1:11211 -> libmemcached/get.cc:314

The odd thing is that I have other things in Django using memcached, connecting to the same server. In the shell, I can get/set from either the default cache or the dedicated collectfast cache perfectly fine.

I've printed out the calls in etags.py to see the keys and etags being set, and I can manually set them to memcached in a Python shell without any problems. This only happens when running collectstatic. When collectstatic finishes running, Python itself ends in a seg fault.

I switched to using a file-base cache and haven't seen any errors. Any ideas of what's happening?