Comments (6)
This is the norma behavior for yas3fs, if you change something on S3 outside of yas3fs control, you need to send an SNS message to the topic in order to invalidate the cache.
from yas3fs.
Thanks - understood, however what about an option such as the below?
"Now I understand this is because of the caching mechanism in yas3fs, however maybe an option could be added that, in the instance of a direct/explicit request for a file, that does not exist in the cache, that would force yas3fs to force check S3 for that file before stating it does not exist to the caller. (again this fallback check would be optional, not all folks would want this behavior)"
from yas3fs.
So with this option I should cache positive results form S3 (e.g. a file
exists) but not negative results (e.g. a file does not exists).
But in case of updates you would still miss the updated content (as I would
cache the previous content).
Would that be ok?
How would you call such an option?
On Tue, Apr 22, 2014 at 6:48 PM, bitsofinfo [email protected]:
Thanks - understood, however what about an option such as the below?
"Now I understand this is because of the caching mechanism in yas3fs,
however maybe an option could be added that, in the instance of a
direct/explicit request for a file, that does not exist in the cache, that
would force yas3fs to force check S3 for that file before stating it does
not exist to the caller. (again this fallback check would be optional, not
all folks would want this behavior)"—
Reply to this email directly or view it on GitHubhttps://github.com//issues/18#issuecomment-41064341
.
from yas3fs.
I'm thinking maybe there would be an configurable option/flag such as this:
a) attemptS3ReadOnCacheFileReadFail = if true this means that if the file system gets a read-request for an explicit file, and the cache reports "does not exist"; before yas3fs returns that the "file does not exist" back to the caller/user, yas3fs would force a call to S3 to double-check that the file indeed does not really exist. If it does exist in S3, it would then pull the file down into the cache and return back to the user's read-request with the file contents etc. In short fixing the cache state.
There could probably be several other variants of this kind of flag for other scenarios as you described (positive/negative evaluations of diff scenarios). However for my case, this is the only one I can think would have the most benefit for now. If other flags are added, I'd only suggest that the flags be super granular/explicit in their effect.
This kind of special flag would be useful for my use-case. I'm using a large app that tracks all known file meta-data in a RDBMS, this RDBMS is written to by many application instances distributed globally. The application never does directory listings to determine an inventory of files and all files are immutable. The app just makes explicit read-requests for files (and writes new ones). Hence I'd like to use yas3fs for solving both the GSLBd aspects of this globally available file store, and its CDN benefits for local reads for the app instances.
In short an option like this would really help to compensate for transient/failure scenarios for whatever reason such yas3fs misses a SQS message or there is a failure in delivery or something is out of sync etc. This would be help the system self-heal rather than us having to have some administrator go blow a way an entire cache locally on disk to fix the problem w/ one file.
from yas3fs.
Any thoughts on this idea?
from yas3fs.
#24 should take care of this.
from yas3fs.
Related Issues (20)
- CPU cycles while idling HOT 1
- FUSE permission handling has changed, --no-default-permissions is undocumented
- Include either mountpoint or bucket name in cache-status messages
- --cache-check N seems not to work
- EFAULT bad address in system call argument HOT 4
- ca-central-1 not supported?
- Feature: File Level Encryption
- Mounting public buckets
- Digital Ocean compatible? HOT 5
- Unsupported UTF-8 characters HOT 1
- Trouble Mounting S3 Bucket Inside Docker Container HOT 2
- MacOSX 10.15.4 makes yas3fs read-only?
- Broken package in pypi HOT 1
- /etc/fstab support for ubuntu 18.04 HOT 1
- Big Sur compatibility? HOT 2
- Please tag and push new versions to pip
- Pip install no longer working HOT 1
- Unable to `pip install yas3fs` on Ubuntu server HOT 3
- Backblaze B2 S3 API HOT 1
- Undocumented requests package version requirement HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from yas3fs.