Comments (17)
grantjenks
commented on August 22, 2024
After you "rm -r /var/tmp/django_disk_cache" just restart your web server. The directory and supporting SQLite database will be created for you.
from python-diskcache.
grantjenks
commented on August 22, 2024
Here's the relevant docs:
https://docs.djangoproject.com/en/1.11/topics/cache/#basic-usage
http://www.grantjenks.com/docs/diskcache/api.html#djangocache
To use the API, open a Django shell with "python manage.py shell" then import the cache as you do in your views and call "cache.clear()"
from python-diskcache.
grantjenks
commented on August 22, 2024
Is there a diskcache bug that caused the out-of-space scenario?
from python-diskcache.
silentjay
commented on August 22, 2024
Wow that was quick! No it wasn't a diskcache bug that caused the out-of-space scenario just my stupid fault for setting the cache too large.
I've restarted the and deleted /var/tmp/django_disk_cache multiple times and seen it's been recreated but the error remains, the only way I've been able to fix it is by changing the django cache type.
I'll try cache.clear() thanks, also where is the sqlite db located as I'd like to try deleting that as well
from python-diskcache.
grantjenks
commented on August 22, 2024
The SQLite database is in that directory. I think it's "/var/tmp/django_disk_cache/cache.db"
from python-diskcache.
grantjenks
commented on August 22, 2024
Actually it's a little more complicated. There's 8 or so subdirectories and it's in one of those.
from python-diskcache.
grantjenks
commented on August 22, 2024
Do you have a situation where you assume the cache is warmed? Meaning, are there keys you assume will always be there? It's hard to tell without more stack trace in your exception.
from python-diskcache.
silentjay
commented on August 22, 2024
Still no joy, have deleted /var/tmp/django_disk_cache/, run 'from django.core.cache import cache -> clear.cache()' in a shell, restarted nginx and gunicorn.
I'm sure not what you mean by assuming the keys are always there, I'm using it in conjunction with fancy-cache https://github.com/peterbe/django-fancy-cache, if there's a miss on the keys then I believe it executes the code as normal then caches it. Been working well for a week or two until I deleted that directory.
Here's the stack trace:
Internal Server Error: /x
ValueError at /x
Key ':1:87449ae7b6412baad697135c2326efe3' not found
Traceback:
File "/home/x/venv/local/lib/python2.7/site-packages/django/core/handlers/exception.py" in inner
41. response = get_response(request)
File "/home/x/venv/local/lib/python2.7/site-packages/django/core/handlers/base.py" in _get_response
187. response = self.process_exception_by_middleware(e, request)
File "/home/x/venv/local/lib/python2.7/site-packages/django/core/handlers/base.py" in _get_response
185. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/x/venv/local/lib/python2.7/site-packages/django/utils/decorators.py" in _wrapped_view
141. result = middleware.process_request(request)
File "/home/x/venv/local/lib/python2.7/site-packages/fancy_cache/middleware.py" in process_request
176. self.cache.incr(cache_key)
File "/home/x/venv/local/lib/python2.7/site-packages/diskcache/djangocache.py" in incr
203. raise ValueError("Key '%s' not found" % key)
Exception Type: ValueError at x
Exception Value: Key ':1:87449ae7b6412baad697135c2326efe3' not found
from python-diskcache.
grantjenks
commented on August 22, 2024
That behavior looks expected if the key is not in the cache. See the Django source for reference: https://github.com/django/django/blob/1.11.8/django/core/cache/backends/base.py#L177
So the question then is why does fancy-cache assume the key is in the cache?
from python-diskcache.
silentjay
commented on August 22, 2024
Well I assumed the keys were in this sqlite db I can't find which still has the old keys from before I deleted /var/tmp/django_disk_cache, but this isn't the case?
/var/tmp/django_disk_cache has been recreated but there's no sqlite db there, so I definitely must be using the original
from python-diskcache.
grantjenks
commented on August 22, 2024
Assuming your caches setting looks something like:
CACHES = {
'default': {
'BACKEND': 'diskcache.DjangoCache',
'LOCATION': '/var/tmp/django_disk_cache/',
'OPTIONS': {
'size_limit': int(10e9),
},
},
}You can also test the caching layer with:
$ python manage.py shell
Python 2.7.14 (default, Sep 22 2017, 00:05:22)
>>> from django.core.cache import caches
>>> cache = caches['default']
>>> cache.set(b'test', 0)
True
>>> cache.get(b'test')
0
>>> exit()
$ python manage.py shell
Python 2.7.14 (default, Sep 22 2017, 00:05:22)
>>> from django.core.cache import caches
>>> cache = caches['default']
>>> cache.get(b'test')
0
Does that work?
from python-diskcache.
silentjay
commented on August 22, 2024
Yes that works, except instead of '0' nothing is returned, which I assume is the same
from python-diskcache.
grantjenks
commented on August 22, 2024
Not the same actually. The default return value is None which is not displayed by the Python interpreter. Sounds like your cache is not getting created. Usually that's a file system permissions issue.
from python-diskcache.
silentjay
commented on August 22, 2024
Ah you're right, it had:
drwxr-xr-x silentjay www-data
changed it to:
drwxr-xr-x silentjay silentjay
and it's now working! either that or it's because I got mad and ran the clear cache command about 20 times in a row. I also changed size_limit back from 2 ** 16 to 2 ** 32
from python-diskcache.
grantjenks
commented on August 22, 2024
Glad it's working now!
I wonder why it didn't error earlier? I tried to support this scenario with an informative exception:
In [1]: import diskcache as dc
In [2]: cache = dc.FanoutCache('/tmp/test/cache')
---------------------------------------------------------------------------
EnvironmentError Traceback (most recent call last)
<ipython-input-2-b3b6bb09a133> in <module>()
----> 1 cache = dc.FanoutCache('/tmp/test/cache')
/home/grantj/env27/local/lib/python2.7/site-packages/diskcache/fanout.pyc in __init__(self, directory, shards, timeout, disk, **settings)
36 **settings
37 )
---> 38 for num in range(shards)
39 )
40 self._hash = self._shards[0].disk.hash
/home/grantj/env27/local/lib/python2.7/site-packages/diskcache/fanout.pyc in <genexpr>((num,))
36 **settings
37 )
---> 38 for num in range(shards)
39 )
40 self._hash = self._shards[0].disk.hash
/home/grantj/env27/local/lib/python2.7/site-packages/diskcache/core.pyc in __init__(self, directory, timeout, disk, **settings)
371 error.errno,
372 'Cache directory "%s" does not exist'
--> 373 ' and could not be created' % self._directory
374 )
375
EnvironmentError: [Errno 13] Cache directory "/tmp/test/cache/000" does not exist and could not be created
In [3]: !ls -al /tmp/test
total 8
drwxr-xr-x 2 www-data www-data 4096 Dec 13 17:04 .
drwxrwxrwt 10 root root 4096 Dec 13 17:07 ..
Also, if I setup a dummy Django project, I get that error again:
$ python manage.py shell
Python 2.7.12 (default, Jul 1 2016, 15:12:24)
Type "copyright", "credits" or "license" for more information.
IPython 5.5.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: from django.core.cache import caches
In [2]: cache = caches['default']
---------------------------------------------------------------------------
EnvironmentError Traceback (most recent call last)
<ipython-input-2-c8c8d8058ea1> in <module>()
----> 1 cache = caches['default']
/home/grantj/env27/local/lib/python2.7/site-packages/django/core/cache/__init__.pyc in __getitem__(self, alias)
78 )
79
---> 80 cache = _create_cache(alias)
81 self._caches.caches[alias] = cache
82 return cache
/home/grantj/env27/local/lib/python2.7/site-packages/django/core/cache/__init__.pyc in _create_cache(backend, **kwargs)
53 raise InvalidCacheBackendError(
54 "Could not find backend '%s': %s" % (backend, e))
---> 55 return backend_cls(location, params)
56
57
/home/grantj/env27/local/lib/python2.7/site-packages/diskcache/djangocache.pyc in __init__(self, directory, params)
26 options = params.get('OPTIONS', {})
27 self._directory = directory
---> 28 self._cache = FanoutCache(directory, shards, timeout, **options)
29 self.memoize = self._cache.memoize
30
/home/grantj/env27/local/lib/python2.7/site-packages/diskcache/fanout.pyc in __init__(self, directory, shards, timeout, disk, **settings)
36 **settings
37 )
---> 38 for num in range(shards)
39 )
40 self._hash = self._shards[0].disk.hash
/home/grantj/env27/local/lib/python2.7/site-packages/diskcache/fanout.pyc in <genexpr>((num,))
36 **settings
37 )
---> 38 for num in range(shards)
39 )
40 self._hash = self._shards[0].disk.hash
/home/grantj/env27/local/lib/python2.7/site-packages/diskcache/core.pyc in __init__(self, directory, timeout, disk, **settings)
371 error.errno,
372 'Cache directory "%s" does not exist'
--> 373 ' and could not be created' % self._directory
374 )
375
EnvironmentError: [Errno 13] Cache directory "/tmp/test/cache/000" does not exist and could not be created
from python-diskcache.
grantjenks
commented on August 22, 2024
Oh wait! I just noticed your comment about the size limit. "2 ** 16" is 65,536 which is really much too low to be practical. Your cache was likely evicting keys to meet the size limit just after they were written. It's happened to me before too.
from python-diskcache.
silentjay
commented on August 22, 2024
That would be it then. Might be worth a warning in the docs. Thanks again for the help.
from python-diskcache.
Related Issues (20)
- Mark the distributed package as typed HOT 1
- Long-lived cache management HOT 2
- PR for fork deadlock HOT 1
- Can this awasome add the range selection Feature? HOT 2
- PicklingError: logger cannot be pickled HOT 6
- Q: Config for in-memory shared cache HOT 8
- Use local OS' path separator HOT 3
- Cannot change JSONDisk compression level HOT 2
- Memoize key isn't always invertible
- support pandas.DataFrame in cache.memoize()
- Protect Against Repeated KeyboardInterrupt Signals HOT 1
- Error during reset
- Custom serializer cannot return "None"
- Bug with pandas 2.0 HOT 1
- Retrieve cached file path instead of bytes HOT 1
- Comparison with RocksDict HOT 3
- ImportError: DLL load failed while importing _sqlite3: The specified module could not be found.
- Add typing HOT 1
- Passing ignore arguments to memoized cache from local scope (diskcache python)
- Cache access fails after forking if multiple `Cache` instances are backed by the same database HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from python-diskcache.