Cache behavior about yas3fs HOT 6 CLOSED

This is the norma behavior for yas3fs, if you change something on S3 outside of yas3fs control, you need to send an SNS message to the topic in order to invalidate the cache.

from yas3fs.

Thanks - understood, however what about an option such as the below?

"Now I understand this is because of the caching mechanism in yas3fs, however maybe an option could be added that, in the instance of a direct/explicit request for a file, that does not exist in the cache, that would force yas3fs to force check S3 for that file before stating it does not exist to the caller. (again this fallback check would be optional, not all folks would want this behavior)"

from yas3fs.

So with this option I should cache positive results form S3 (e.g. a file
exists) but not negative results (e.g. a file does not exists).
But in case of updates you would still miss the updated content (as I would
cache the previous content).
Would that be ok?
How would you call such an option?

On Tue, Apr 22, 2014 at 6:48 PM, bitsofinfo [email protected]:

Thanks - understood, however what about an option such as the below?

"Now I understand this is because of the caching mechanism in yas3fs,
however maybe an option could be added that, in the instance of a
direct/explicit request for a file, that does not exist in the cache, that
would force yas3fs to force check S3 for that file before stating it does
not exist to the caller. (again this fallback check would be optional, not
all folks would want this behavior)"

—

Reply to this email directly or view it on GitHubhttps://github.com//issues/18#issuecomment-41064341
.

from yas3fs.

I'm thinking maybe there would be an configurable option/flag such as this:

a) attemptS3ReadOnCacheFileReadFail = if true this means that if the file system gets a read-request for an explicit file, and the cache reports "does not exist"; before yas3fs returns that the "file does not exist" back to the caller/user, yas3fs would force a call to S3 to double-check that the file indeed does not really exist. If it does exist in S3, it would then pull the file down into the cache and return back to the user's read-request with the file contents etc. In short fixing the cache state.

There could probably be several other variants of this kind of flag for other scenarios as you described (positive/negative evaluations of diff scenarios). However for my case, this is the only one I can think would have the most benefit for now. If other flags are added, I'd only suggest that the flags be super granular/explicit in their effect.

This kind of special flag would be useful for my use-case. I'm using a large app that tracks all known file meta-data in a RDBMS, this RDBMS is written to by many application instances distributed globally. The application never does directory listings to determine an inventory of files and all files are immutable. The app just makes explicit read-requests for files (and writes new ones). Hence I'd like to use yas3fs for solving both the GSLBd aspects of this globally available file store, and its CDN benefits for local reads for the app instances.

In short an option like this would really help to compensate for transient/failure scenarios for whatever reason such yas3fs misses a SQS message or there is a failure in delivery or something is out of sync etc. This would be help the system self-heal rather than us having to have some administrator go blow a way an entire cache locally on disk to fix the problem w/ one file.

from yas3fs.

Any thoughts on this idea?

from yas3fs.

#24 should take care of this.

from yas3fs.