No-Expiry about gun HOT 3 CLOSED

I'm confident that this requires no additional wire protocol.

This thread should be renamed and re-focused. Sharding with respect to GUN is only an interface option. The underlying prerequisite is actually controlling expiry - or more to the point: expiry period. GUN currently has none, which means as you subscribe to more and more data, your RAM usage will "memory leak" and eventually explode causing your app to crash (this is bad. however the nice thing is gun can cleanly survive these types of failures).

My current initial thought is that the expiry mechanism will be LRU (least-recently-used) by default. However if you are running your own physical machines you will necessarily have to shard. If you are not running your own physical machines you should not shard, since GUN's storage plugins for various services (AWS, Google, etc.) utilize more powerful mechanisms than sharding. So even if you do need sharding because of physical constraints, it is still achieved via expiry logic.

Therefore, it would be nice to modify the expiry system, such as NOT expiring certain data. Say because you want to prioritize certain data on a peer (like a user's profile information on a user's device) that doesn't get pushed out when other data flows in (like their friend's status updates) OR for when someone is running physical machines and they need to chunk/partition/shard it to guarantee long term storage.

Most of these options will realistically be per-storage-plugin specific, since that is where the majority of the responsibilities lie. However, that said, it might be nice to have generic options that plugins expect, just to keep things easier for implementers striving for compatibility with other plugins. HOW to interface with that will be in the other discussion you referenced (I'll repeat a couple of the thoughts here there).

Extra clarification: by "expiry" here I do not mean "cache-invalidation" (that already happens by default from the realtime updates and stuff, use journal/logging to store the revision history to rewind/revert to if desired), but meaning "what data do I discard such that I do not explode my physical resource constraints like RAM/disk". If you can think of a better word for this, that would be good. I also want to distinguish between expiry on RAM and disk, the behave nearly the same but not exactly the same. In RAM we definitely have to swap, so LRU (or something similar) has to be used. So "expiry" here must mean "expiry" of the data across the entire system for non-infinite storage systems (ones where new hard drives are not being added automatically). Meaning different machines would have slightly different "no-expiry" keys (via some API) which then creates "the sharding" behavior. Meaning sharding is an emergent result of lower level primitives. If you have physical machines and you do not increase physical storage space you will eventually run out of space, in which new data obviously will be not persisted if you are using no-expiry clauses, however if you had expiry-clauses then new data would slowly start replacing old data (which MIGHT be desired or MIGHT not). NOW, with that aside, even if you have finite-storage space, GUN will still be caching everything it can on various machines, it is just specifically at the point in which it hits its limits that it then discards of either LRU or (in a shard context) data that is the inverse of the no-expiry options.

BLAH. We need to figure out how to take these word dumps of mine and organize them into logical components so I don't confuse people with all my text barfs of incremental (attempts) edge case explanations.

from gun.

Note: Sharding won't be core, but I suspect expiry/LRU or something like that will need to be a core feature. As a result, I'm gonna leave this issue open.

from gun.

Oh actually, this is specifically about the no-expiry sharding, compared to #68 which is about internal expiry. Ah, gonna close this since it isn't core, and leave #68 open. This should be implemented as an option with storage drivers and stuff. Sure, we'll probably eventually standardize on an options interface, but the code for this will be in its own repo. Therefore closing here.

from gun.