Alright, I have an initial implementation ready. I will start working on writing tests but I would really appreciate some early feedback :)

I've worked on the assumption that PRs #72 and #66 will be merged, so my code builds on top of those changes. Given this, what would be the best way to create a PR? If I create it against any of the existing branches it will be a nightmare to review only the changes related to persistence.

Let me know how to proceed 🙂

Cheers.

from firebase-js-sdk.

I see news travel fast around here (faster than me on the bike anyway).

Yeah, I was really busy with work recently, which prevented me from spending any time on this, and now I'm taking some time off to travel. I will eventually get back to it, I promise. Sorry if anyone was counting on this being ready sooner.

In the meantime, anyone else could contribute to move this forward faster. I think everything is explained in previous comments but off the top of my head, this is what's pending:

• Limit multi-tab write access. (There might be other edge cases too, I'll see if I can give this some thought)
• Fix the current tests so they pass again, after some changes I had to make to the existing code (some methods were returning values synchronously and now return a Promise, to account for the possibility of having to access asynchronous storage)
• The motherload: write the whole test suite for the new features. I'd like to do this myself but don't hesitate giving it a go if you want.

If you do any work towards this then feel free to open a PR against my fork (also linked elsewhere here).

I promise I'll finish this as soon as I get my hands on a laptop but, like I said, it might be a while.

Cheers,

Josep

from firebase-js-sdk.

I was thinking about a database persistence similar to what is supported with the Firebase iOS and Androd SDKs available even when the web app is reopened in a new browser tab offline.
Right now i need to implement my own offline persistence layer on top of Firebase to support offline first scenarios.
There was even a Firebase I/O session last year regarding PWAs and offline first. For this demo Polymer with an index db mirror was used on top because the functionality was not provided by the Firebase JS SDK itself. https://www.youtube.com/watch?v=SobXoh4rb58
With this approach I'm limited on the offline functionality of Polymer without the ability to directly query the Firebase database.
Would be great if such an index db mirror could be part of Firebase JS SDK itself.

from firebase-js-sdk.

Ok, I've started looking into this. Some general considerations:

• Persistence should be off by default and the user can enable it at will. Same behavior as in iOS & Android, AFAIK.
• Both server and user-initiated writes should be persisted.
• For user writes:
  • All set() and update() operations get persisted.
  • transaction() operations are only persisted once acknowledged by the server (committed). [@puf, when you said that transactions are not persisted in iOS/Android I assume you meant the intermediate states, right?]
  • Keep track of pending writes in case they fail, so that we can roll back persisted data.

Thoughts so far?

I'll start working on a simple proof of concept implementation to get the ball rolling. If I'm not mistaken the most obvious first "point of attack" seems to be the SyncTree so I'll focus on that. Feel free to offer any comments and suggestions, though :)

P.S.: I'll try to be as independent as possible while working on this to avoid bothering you all too much, but I might ask for some guidance from time to time. Hope that's ok!

from firebase-js-sdk.

A quick overview of what's implemented so far regarding persistence:

• Pending user writes: they are persisted until acknowledged/refused by the server. Only set() and update() operations but not transactions(). During initialization, if any persisted pending user writes are found they are retried.
• Server cache: any data coming from the server because of a user query is persisted. This cached data, if any is found, is used to immediately notify the user when querying data and also to calculate the appropriate hashes when attaching a listener on the server (just like with the current in-memory cache).
• A storage adapter model allowing the user to supply any storage engine to be used for persistence, as long as it complies with a defined API.
• An IndexedDB-based storage adapter implementation to be used as the default if none is provided, which will probably be the case most of the time.

Next steps:

• Testing (1): first of all I want to get the current tests to pass again. From a quick look at the results I don't think I broke any functionality, but quite a few of the tests are failing due to several changes I had to make to the current code. @jshcrowthe I will probably have to coordinate this with you.
• Testing (2): then I'll start writing the test suite for the new features and any other necessary tests.
• Investigate implementing compound hashes and range merging in order to avoid receiving all the data for a query when attaching a listener if only parts of that data have changed compared to what we have persisted/in memory.
• keepSynced()... maybe.

About that last point: in the JavaScript SDK we can't really implement the keepSynced() functionality, at least not in the same way it works in the iOS and Android ones. In those platforms the SDK has complete control over when and what data to prune from the persisted cache, so it can choose to never prune paths that have been marked to keep (barring the OS randomly deciding to wipe the app's data). That's not the case here since with IndexedDB it's 100% up to the browser to decide when to evict data from the cache.

If we decide to implement this we could force accessing the cached data for the keepSynced paths every so often even if we don't need it. Most IndexedDB implementations follow a LRU cache policy so that would keep that data "fresh" in the eyes of the browser. This would still be a best-effort approach though, since we couldn't guarantee that the data doesn't get evicted if the cache fills up and other parts have been used more recently.

In short: I would forget about keepSynced(), at least for now. Let's focus on everything else.

Edit: see comments below about possible keepSync support.

from firebase-js-sdk.

For the record, while I am excited about Firestore for some use-cases, the realtime database is still the best fit for my application (many small updates). I'm still anxiously waiting for this feature to be implemented and wishing I had the time to help move it forward. Sending ya'll who work on it lots of love and good wishes! 😄 Thanks for your hard work.

from firebase-js-sdk.

@knpwrs I think this is something that we could totally accept as a PR! Love to have your contributions. The notion of different storage adapters is also an interesting idea that I'd love to see more details on.

In addition, I'd encourage everyone, for all Feature Requests, to make sure you are signed up for the Firebase Alpha Program where you can keep up on all the upcoming features and products.

from firebase-js-sdk.

To me the best case scenario for true offline support would be if it was completely transparent. Downloaded data would be available from persistent storage, and new data would be written to persistent storage and synched automatically once the device is online again.

I'd like to add (since I haven't seen this mentioned) that using localStorage on mobile is not very persistent since it can be deleted at any time by the OS. This would be much more inconvenient on Cordova, React Native, NativeScript, apps. Users expect more persistence from an app than a website.

I agree that right approach would be having an API and write adapters on separate modules (official or third party) to reduce bloating on the main SDK.

from firebase-js-sdk.

It's integrated into the SDK. That allows the SDK to know what data is cached, which prevents the server from sending data that's already known. This significantly reduces bandwidth usage in most cases.
When using a caching solution external to the SDK, like most of us are doing now, the server will still send all the data back to the client when attaching a listener.

Besides, we shouldn't be putting the burden of implementing their own persistence on the library's users when using the Web SDK, considering the iOS and Android ones already support this feature. The way I've implemented it, the user just needs to run firebase.database().enablePersistence() and forget about it.

Internally it uses IndexedDB by default if available but for more complex use cases, like a Cordova o React Native app, the enablePersistence() method also accepts a storage adapter to use as long as it complies with a specific but simple API. It should be trivial to wrap whatever storage engine you want to use, as long it follows a key-value model.

from firebase-js-sdk.

Hi everyone. I started looking into this again to see if we can move it forward. I’m gonna put some thoughts into writing here, along with some of the decisions I’ve made so far. Please feel free to comment about any of it and ask any questions you have.

(cc @mikelehen @schmidt-sebastian)

Data storage

There was some concern about how data is stored internally. My current implementation follows the same approach as in the iOS sdk, where an entry is created for each value with its full path as the key. As mentioned, this results in many entries with long keys and usually small values. That's fine in the iOS sdk because it uses LevelDB, which uses prefix compression on the keys.

I've looked into it and both Blink (Chrome, Opera) and WebKit (Safari, Chrome iOS) use LevelDB as the underlying engine for IndexedDB so we're probably good there, although we should still do some profiling to be sure the performance is acceptable. I couldn't find any information about Gecko (Firefox) nor EdgeHTML (Edge).

As for other platforms, like React Native, I think we should offload the decision on how to store it to their specific StorageAdapter implementation. The "core" could still keep using these deep keys while, depending on what's most efficient in that platform, the specific storage implementation might decide to group several entries that share a common prefix into a single value. When retrieving data from storage, the persistence manager already makes a single request to the storage adapter to get all the entries whose keys begin with a certain prefix (the path we want) so it should be fairly trivial to use an alternative approach in each adapter.

That aside, I don't see any easy alternative to the current implementation. Even though IndexedDB is our best option as a default, it's really not ideal to store arbitrary JSON-like data and that imposes certain limitations. We can't just dump the whole thing into a single entry since that would be very inefficient, so data needs to be split somehow. But we don't really know how to split that data other than by its deepest key. In the long run we could implement some heuristic to determine where to split/group (for example, by determining which paths are commonly written to or read from, among other things) but in the meantime our best approach is to just keep using deep keys. Obviously I’m open to any suggestions here, of course.

Multi-tab access

This was another concern back when we were discussing this last year. Since it has already been solved in Firestore, I think the best approach here will be to mirror their solution. To keep things simple, though, I will probably begin by not allowing persistence to be enabled in more than one tab. This can be achieved by using a minimal implementation of what's currently being used in Firestore, using LocalStorage to coordinate between tabs. Once an initial version is stable and working we can look into adding proper multi-tab support later on.

Cache policy

To prevent the cache from growing too big (and thus increasing the chances of the browser nuking all persisted data) I implemented a “Least Recently Used” cache policy, with pruning triggered when the persisted storage reaches a certain size.

With IndexedDB, though, there’s no direct way to obtain the size of the database so I ended up following this approach as an approximation:

• If it's a number, count 8 bytes.
• If it's a string, count its length in bytes.
• If it's a boolean, count half a byte.
• If it's an array, loop its elements and apply the same algorithm recursively.
• Also, for each entry add half the length of the key to the count.

I’ll need to do some testing to see if it’s a good approximation or whether it needs some tweaking. Any suggestions as to how to make it more accurate are definitely welcome.

packaging

When I implemented most of this last year I put persistence into its own separate module, in order to limit the impact this change would have on the size of the database bundle. Since the internal structure of the SDK has changed quite a bit these last few months, when I adapted my changes into it I opted to just put it into the database package for now. That also seems to be the approach followed by Firestore, but if you think it would still be a good idea to have it separate (something like @firebase/database-persist) let me know. This can be decided further down the road, though. There’s plenty of work to do before this is of any concern.

from firebase-js-sdk.

Transactions are explicitly not persisted to disk. They do not survive app restarts.

This was an explicit decision by the team at the time. It might be good to revisit the discussion at some point. But for a first iteration, I'd recommend aiming for feature parity with iOS and Android and not persisting transactions.

from firebase-js-sdk.

I'm experimenting with extracting persistence into its own module and I'm seeing pretty good results, with only a small increase in size in the firebase-database.js module.

Here's a handy comparison table with the differences, all the other files remain the same:

It would be included like any other of the firebase services, either by:

import firebase from 'firebase/app';
import 'firebase/database';
import 'firebase/database-persistence';

or by:

<script src="firebase-app.js"></script>
<script src="firebase-database.js"></script>
<script src="firebase-database-persistence.js"></script>

I think this is the way forward, since it introduces very little penalty to users who don't want to use persistence in their apps.

from firebase-js-sdk.

+1

from firebase-js-sdk.

It's no big deal except you have to manage Redux in addition to Firebase. You have no control over when Firebase syncs to the server, you're restricted by Firebases local cache limits and persisting the cache isn't trivial. And Redux certainly overlaps with what the Firebase SDK does for offline. All could be mitigated should Firebase support a few modules/adapters for Offline-first in a similar method to how Redux-Offline defines. I'm just suggesting that we use Redux-Offline as a guide for what parts could be made modular.

from firebase-js-sdk.

Persistence should be off by default

Then persistence should be a separate library; don't want to ship dead code to the client.

from firebase-js-sdk.

Hey everyone! I see you got to this before I did but just wanted to give another plug for Cloud Firestore.

As has been mentioned, Firestore supports web offline, and it may be a great fit for your application. So please take a look!

That said, I've changed the title of this issue to point at the RTDB as this is still a valid feature request.

from firebase-js-sdk.

Just wanted to say I'm still hoping for this feature. I'm quite happy with RTDB and don't see any other reason to migrate to Firestore. My only alternative is to implement my own caching system which won't be as good as official support.

@jsayol I'm cheering for you! 👏

from firebase-js-sdk.

This is something I have talked to Firebase support about in the past and I was actually just about to open my own issue until I saw this one. When I think about Firebase (at least, my usage of Firebase) and offline functionality I think of storage.

I think what would make the most sense would be to refactor the current implementation to support storage adapters. The current implementation could become the default, "in-memory" adapter. Other community-developed or officially supported adapters could be published as well. IndexedDB is an obvious choice, it's what PouchDB uses by default. A less obvious adapter I would like to implement for use in Electron would be a sqlite adapter.

Just spitballing here, but there could also be a proxy adapter to use two adapters together. For instance, I could use the in-memory adapter along with my sqlite adapter for performance purposes.

The Firebase SDKs were only just recently open sourced. Would these types of features be welcome for pull requests?

from firebase-js-sdk.

@jshcrowthe What exactly features and products Firebase Alpha Program offers at this time? I've completed and submitted the form 3 days ago. I wonder how long to wait the admission into the Alpha Program?

from firebase-js-sdk.

Adding some form of local storage in our apps to cache the data retrieved from the Firebase database (like most of us are doing, I guess) works quite well to enable offline use even when cold-starting an app, but the main problem I see with not having persistence built into the SDK like it is in the Android or iOS ones is this: when an app starts, the SDK has no idea what data is stored locally so, when it attaches listeners, the hash field is empty and the server responds with all the data. Every single time. That means that data usage with the JS SDK is significantly higher than with the native ones.

I understand building a reliable and truly cross-browser local storage solution into the Firebase SDK is no easy feat, maybe even impossible, but it doesn't need to be perfect. It just needs to be better than not having it, even if only in some situations. It could be implemented gradually, first for whatever browsers have the best IndexedDB or SQLite support and then slowly with others, if possible.

Another possible solution, albeit a radically different approach to what is being currently used in the other platforms, would be for the SDK user to pass whatever data it has for a certain database location before attaching a listener.

This might be better explained with an example: let's say we already know what the data at /messages contains, because we had it locally stored somehow:

let data = {
  "-Kgx5lyGUg7w9nnNAKss": {
    "from": "Bob",
    "text": "Hey there"
  },
  "-Kgx9en1kPcRyy1uk7j7": {
    "from": "Alice",
    "text": "Sup?"
  }
};

So there would be a way to "bootstrap" the data at that location before attaching a listener, letting the SDK know what we know:

firebase.database().ref('messages').bootstrap(data).on('child_added', snap => { /* */ });

This leaves some open questions, though: should the SDK always accept that data, or should it ignore it when it is positive the data it has is fresh? (maybe because there's already an active listener on that path).

This would only be a temporary solution anyway, since it puts most of the burden on the developer using the library (figuring out how to store the data locally, passing it to the SDK, etc.) Not my favorite approach but it would certainly be a step up.

from firebase-js-sdk.

This is great! Firestore is much more aligned to my needs than a realtime db. I don't mean to get this issue off topic but it's somewhat relevant: how does sync work? Is it similar to the rtdb implementation mentioned earlier in this thread (compound hashes)? OT-based? How are conflicts handled?

Edit: Looks like it's last-write-wins like the realtime db.

from firebase-js-sdk.

Did @jsayol ever make it to Barcelona?

from firebase-js-sdk.

Did @jsayol ever make it to Barcelona?

I did.

I've been looking into this again. The internal structure of the SDK has changed quite a bit since last time so I'm slowly solving all the merge conflicts on my local copy to get it working again. Once that's done I'll keep working on it :)

from firebase-js-sdk.

I believe this was talking about adding offline support to the Realtime Database SDK (not the Cloud Firestore SDK, which already supports web offline).

FWIW- In theory you could use the Cloud Firestore SDK 100% offline by calling firebase.firestore().disableNetwork() but it's not really optimized for this use case right now, so performance may degrade over time.

from firebase-js-sdk.

from firebase-js-sdk.

@jsayol I've been looking at how PouchDB stores data offline. They've taken the approach of basing their storage around LevelUP. From that point you can plug in different backends such as MemDOWN (in-memory), level.js (IndexedDB), or even something like SQLdown (sqlite3, PostgreSQL, and MySQL). There's even an Abstract LevelDOWN project which can be used to implement compatible backends. Basing storage around LevelUP could be potentially interesting because then we inherit a large offering of various storage backends. By default we could use MemDOWN and offer the ability to use different backends such as level.js.

from firebase-js-sdk.

@FluorescentHallucinogen, the Alpha Program is a great way to work with developers who are willing to donate their time to help make Firebase an awesome platform. There is really good discussion going on, so the invitation is to make sure we get to work with all of you in that space as well!

from firebase-js-sdk.

Upon start of the app, the queue of pending writes is read from disk and just becomes the head of the write queue that always exists.

from firebase-js-sdk.

How is your solution better than using a caching lib like LaddaJS?

Ladda is built around the idea of CRUD, and not really subscriptions and un-subscriptions to realtime data.

Another factor is that promises are one time only. You need callbacks or observers a la RX to get subscriptions working. For example with on(...).

While it's possible to use REST with Firebase, losing realtime would defeat its main purpose, no?

from firebase-js-sdk.

More about those quirks:
after this small patch everything seems to work as expected 😊!

in src/database/persistence/ServerCacheStore.ts line 111:

const subpath = item.key.substring(baseKey.length)
                .split('/').filter(part => part.length > 0);

to:

const subpath = item.key.substring(item.key.indexOf(baseKey) + baseKey.length)
                .split('/').filter(part => part.length > 0);

from firebase-js-sdk.

Hey @jsayol, how is .enablePersistence() going? Right now we are using redux-offline for persistence but your solution might be more useful for us. If you could provide somewhat of a roadmap or a list of things that need to be done before integrating your fork into the main repo we might be able to help and make this a reality!

from firebase-js-sdk.

It appears @jsayol went on a bike trip from Poland to Barcelona for 3 months, so this might take a while...

😄

https://twitter.com/jsayol/status/905564254379630594

from firebase-js-sdk.

Just saw this today.

Enables offline data access via a powerful, on-device database. This local database means your app will function smoothly, even when your users lose connectivity. This offline mode is available on Web, iOS and Android.

It seems Firebase announced another db option separate from the realtime db. I haven't looked into the details yet but it claims to support offline capabilities similar to this issue. Just thought I would drop this here since I've been watching this issue closely and I'm sure others like me might find this option suitable.

from firebase-js-sdk.

@cfilipov the Firestore web SDK is also open source here: https://github.com/firebase/firebase-js-sdk/tree/master/src/firestore

Most of the offline code is in the local folder. Firestore uses a versioning system instead of hashing to do conflict resolution, which is better and worse in someways to the hashing method.

Most of the offline caching of queries is here: https://github.com/firebase/firebase-js-sdk/blob/master/src/firestore/local/indexeddb_query_cache.ts

We do cache all the documents that are returned here as well:
https://github.com/firebase/firebase-js-sdk/blob/master/src/firestore/local/indexeddb_remote_document_cache.ts

Although we do a lot of other indexes we keep for performance reasons and we also persist any writes you do.

from firebase-js-sdk.

Don't lose heart @awardrop!

The browser limit is 50% of remaining disk space (https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Browser_storage_limits_and_eviction_criteria) which is much more than 4mb in most cases. And in the cases it is not, you can let the user know to clean up disk space for offline functionality when getting the QuotaExceededError.

In addition, if firebase is ejected because it has been unused for awhile this is certainly ok. The main reasons to cache the data locally are:

1. increase the speed of startup and data-load
2. allow for offline usage
3. decrease bandwidth usage and costs

If you have to fetch data from the server because it has been ejected since last fetch, that is certainly an acceptable situation. Offline storage enhances an app, but the data still exists elsewhere so you're not in trouble if it times out. You're only in trouble if you only store it locally and not anywhere else.

I'm storing all my data locally in indexeddb and syncing it to firebase. I would love to take advantage of the bandwidth savings persistence mode provides, but I realized yesterday that if I timestamp all the saves I can use a query to only get updates newer than last fetch (and I can store the last timestamp in local storage or indexeddb).

I think most people on this thread of come up with workarounds or other solutions for this problem and are past the point where they need persistence. But newcomers will certainly benefit from the feature (and we all will on our next app) so I still feel there is a lot of value to it. I just wish I could commit time to solving it rather than onlyt contributing to the discussion.

from firebase-js-sdk.

(Update at the bottom)

Hi there.

I just remembered that back when I was implementing this last year I ran into an issue that could become a potential breaking change.

Some background: when a new listener is attached with the current implementation, either via .on() or .once(), then SyncTree.addEventRegistration() returns a list of events to be raised immediately (synchronously) based on the data that can already be found on the in-memory cache.

By adding persistence into the mix, now addEventRegistration() needs to account for the possibility that it might need to access the disk cache, which is asynchronous. This means that now it needs to return a Promise that resolves to that list of events instead.

With that in mind, take the following code from one of the current Transaction tests:

it('New value is immediately visible.', function() {
  const node = getRandomNode() as Reference;
  node.child('foo').transaction(function() {
    return 42;
  });

  let val = null;
  node.child('foo').on('value', function(snap) {
    val = snap.val();
  });
  expect(val).to.equal(42);
});

I've already modified several other tests that were directly inspecting the list of raised internal events synchronously, since that is not a real use case. But this example is different since people's code might be currently relying on this behavior. It's uncommon, but I'd say it's perfectly valid to do that if you know the value you're looking for has already been cached.

If you don't want to introduce a breaking change here, which is perfectly understandable, the only alternative I see would be to revert addEventRegistration() back to synchronously returning the list of events to be raised based only on the in-memory cache, and then asynchronously raise any events based on the disk cache. From a practical point of view, it would behave just as if new data had come from the server after attaching the listener.

Would that be an acceptable solution?

Update: I went ahead and made these changes, since it actually seemed like the only option really. I guess I needed to put it in writing to realize ¯\_(ツ)_/¯

from firebase-js-sdk.

Hey there! I couldn't figure out what this issue is about, so I've labeled it for a human to triage. Hang tight.

from firebase-js-sdk.

I have some ideas of what I think you mean by "offline first support," but the term itself is rather vague especially when viewed in the context of the entire JS SDK as each part of firebase (auth, database, storage, messaging) would have different approaches to "offline first support."

That said, I like what this issue calls attention to, and it is something that I'd love to pursue. Can you help me understand what specific things were difficult for you in doing offline first development?

from firebase-js-sdk.

I was looking at achieving this type of functionality with Firebase and Redux-Offline. If the Firebase JS SDK was to be made modular with defaults it seems that it should do so in other layers of the SDK than just storage to achieve all the Offline-first criteria as specified in Redux-Offline EG: like exposing functions for implementing custom reconciliation of optimistic update failures/rollbacks.

from firebase-js-sdk.

Making the data Offline-first available using react-redux and firebase is no big deal. Here is a working example: https://github.com/TarikHuber/react-most-wanted
But: the data is offline available only for the client. The firebase database listeners don't know that you already have most of the data in your local storage. It would be great if firebase itself would manage Offline-first. Maybe they could figure out how to then just load the data that is missing in the local storage and not all of it like it's done with a running app that has connection. For example: you loaded 10 tasks in your application and go offline or close the application. After you reconnect firebase uses hes own cache not only to give you the already loaded 10 tasks but to also just load 2 tasks that where added afterwards and edits to the existing 10.

from firebase-js-sdk.

+1 to what @mbleigh said.

Lets try and keep this thread on topic though 😄 . Further questions on the Alpha program would be better directed to our support or discussion channels (link here: https://firebase.google.com/support/)

from firebase-js-sdk.

Some more thoughts: a possible solution to add local storage support would be to use localForage, maybe wrapping it like ionic-storage is doing. This would allow to use whatever the best solution is in every scenario/browser.

@knpwrs's idea of storage adapters also seems quite interesting. Skimming through the code, it seems like there's already support for in-memory, LocalStorage, and SessionStorage. So it seems it would be a matter of implementing a way to allow the user to provide their own adapter with a compatible API. I would also consider this a temporary solution though, like the bootstrapping I mentioned in my previous comment. [EDIT] I was looking in the wrong place, nevermind what I said here[/EDIT] The ultimate goal should be for the SDK to handle all this for the user, the same way it's done in the native SDKs.

from firebase-js-sdk.

Having a storage mechanism like redux-persist would allow for complete browser/native coverage. Then the user would only have to specify an environment flag for the correct storage adapter to be used.

from firebase-js-sdk.

I believe Firabase developers already have a solution. That's why @jshcrowthe suggests to sign up Firebase Alpha Program.

@jshcrowthe, @mbleigh, isn't it?

from firebase-js-sdk.

AFAICT this discussion primarily emphasizes two things

1. Providing offline access to data
2. Leveraging offline data on boot to prevent unneeded network traffic

The notion of pluggable storage adapters is something that I think is a cool idea and I'd love to see a demo implementation of this in context of the SDK. This could be separate from minimizing the network traffic as the amount of network traffic would be no different than what it is today. Once we had an agreed upon implementation of persistence, reducing network overhead is just the next logical step.

In the iOS SDK (Github Repo: https://github.com/firebase/firebase-ios-sdk) we are synchronizing only the delta between the local device and server state. In principle we could port that same functionality over to web, and then integrate it with the persistence layer discussed above.

@jsayol / @knpwrs I'd love to see a sample implementation of the storage adapters concept, sounds like a solid strategy to allow for flexible browser/environment requirements.

from firebase-js-sdk.

In the iOS SDK (Github Repo: https://github.com/firebase/firebase-ios-sdk) we are synchronizing only the delta between the local device and server state.

That's very interesting. You mean that if the hashes don't match when attaching a listener, only the difference is synchronized? If so that's pretty cool, and quite different from the web SDK where the whole thing is resent in that situation.

How is it implemented? Do you traverse the tree checking the hashes at every node to figure out what's up to date and what isn't? I'm trying to locate the relevant code in the iOS repo but I can't seem to find it (and not being familiar with ObjC doesn't help either 😄).

from firebase-js-sdk.

@jshcrowthe Time permitting I may be able to get something done. What do you think about utilizing LevelUP as suggested in my previous comment? Obviously assumes a compatible data model. If it's not compatible then we'd need to design our own adapters.

from firebase-js-sdk.

@knpwrs I looked at LevelUP and it seems like a really solid library, however I don't know that we need all that it provides. With the database already being quite large, adding another large persistence library is probably a hard sell (I just ran LevelUP through a quick webpack build, 103kb min).

Same story goes for something like LocalForage (although this one is admittedly lighter coming in at around ~25kb).

IMO I'd start w/ just the raw primitives until we need the abstraction (we are going to have to build our own abstraction layer already to allow it to be pluggable).

from firebase-js-sdk.

@jsayol so we currently are using a hash function that can be found here:

https://github.com/firebase/firebase-js-sdk/blob/master/src/database/js-client/core/Repo.js#L204-L236

This hash is a "simple" hash of the data in the node. We then send that hash to the server when we call listen in the PersistentConnection (see https://github.com/firebase/firebase-js-sdk/blob/master/src/database/js-client/core/PersistentConnection.js#L183)

By leveraging "compound" hashing (which is a hash of key ranges instead of the entire node, iOS implementation found here: https://github.com/firebase/firebase-ios-sdk/blob/master/Firebase/Database/Core/FCompoundHash.m) we could minimize traffic over the wire. We would just need to implement the ability to merge the range updates that we receive with what we already have in memory. (see https://github.com/firebase/firebase-ios-sdk/blob/master/Firebase/Database/Core/FRangeMerge.m)

All that said, I think the right first step is to allow for persistent offline through IndexedDB (or an adapter structure), and then work towards this.

from firebase-js-sdk.

Thanks for the links @jshcrowthe!

I knew about the hash function (a few months ago it took me a while of digging through minified code to figure that one out :P) but I had no idea about the whole compound hashing implementation. I'll definitely look into it!

I agree with you though, none of it will be very useful without persistence so let's focus on that first. I think a solid first approach would be to simply use IndexedDB, since that would cover most use-cases. (Safari's implementation of IndexedDB is known to have issues though, so it might be worth looking into WebSQL too. Maybe. I don't know.)
If we also want to support Node.js then we'd have to look into other options too, but having direct access to the file system opens a whole lot of other possibilities there.

You raised a valid point in a previous comment about bundle size. Ideally we'd keep this change as small as possible but if it ends up getting too large for comfort it could just be implemented into its own sub-module, as an optional feature to be added by the user if they want to use persistence. Something like this:

const firebase = require('firebase/app');
require('firebase/database');
require('firebase/db-persistence');

I'll start looking into how IndexedDB could fit in into the current implementation. Off the top of my head, we'd have to build a system to consistently synchronize the contents of the MemoryStorage with what's being persisted, and probably ensure we're not hitting persistence too often during read or write bursts to avoid performance issues. This synchronization could happen after a certain time of inactivity on the database, like for example 10 seconds, with a maximum interval of time between operations to minimize the risk of ending up with stale data in the event that the app would crash or suddenly be shut down somehow.

Thoughts?

P.S.: I still think the storage adapter idea is an interesting one that can be added later, but basic browser persistence should be provided by the SDK out of the box anyway.

from firebase-js-sdk.

Side thought: how do the iOS and Android SDKs (and this SDK, for that matter) handle transactions when offline?

from firebase-js-sdk.

@knpwrs That is a great question! Paging @schmidt-sebastian since I don't know off the top of my head.

from firebase-js-sdk.

As far as I know, the iOS and Android SDKs will keep track and try to complete the transaction even across app restarts. The JavaScript SDK will only keep the transaction alive during the same session, since it doesn't persist the transaction state. This is definitely something that can be improved with this whole persistence "overhaul".

Other than that, they work the same way: when offline, the transaction callback will receive either the latest known value or null if it isn't known, will trigger optimistic updates unless instructed not to do so, and won't trigger the completion callback until the transaction is actually committed by going back online.

Edit: Oops, turns out I was wrong about transactions. See Frank's comment below for an explanation.

from firebase-js-sdk.

As far as I know the iOS and Android clients keep two types of data in their disk cache:

1. Data that was recently listened to.
2. A queue of all pending write operations, excluding transactions.

Note that (again: as far as I know) the pending writes are not aggregated into the data cache. That only happens when a listener updates it.

But I'd love @schmidt-sebastian or @mikelehen to give their take on this.

from firebase-js-sdk.

Note that (again: as far as I know) the pending writes are not aggregated into the data cache. That only happens when a listener updates it.

That actually makes a lot of sense, there's no point in persisting data that might never be listened to.

I presume the disk cache for pending writes is to be able to retry them after an app restart, right? That might be an interesting feature to add too, maybe after basic persistence is working.

from firebase-js-sdk.

Yeah, I've been giving some thought to that. Once everything is working there should be a discussion about it, but now might be a bit premature. Good point though.

from firebase-js-sdk.

@jsayol How is your solution better than using a caching lib like LaddaJS? I was about to use Ladda in just a few minutes until I saw this thread.

Edit: noteworthy difference may be if there is a change notification.

BTW, I requested caching years ago before FireBase got bought by Google; glad you're working on it!

from firebase-js-sdk.

'enablePersistence()' method also accepts a storage adapter

That's a smart idea! Great if you need IE8-9 compatibility, & for build/server-rendering. I've been looking at some newer faster Key-Value engines.

from firebase-js-sdk.

@jsayol I tried to gulp build your fork but get a java/closure error so I can't 'compile'. Might have to do something about me on Win10, but I'm from an innocent time back in the 90s when Java was supposed to make code platform-independent. ;)

While it's possible to use REST with Firebase, losing realtime would defeat its main purpose, no?

Well, my immediate purpose is to hack source code from here: https://hnpwa.com I'm just checking to see if FireBase added caching yet; I was expecting it since Google owned it for a while now...

But thinks for pointing the differences out @PierBover!

from firebase-js-sdk.

thanks @jsayol for the initial heavy lifting on this issue!

i have been waiting for a proper implementation for quite some time and had to built my own workarounds to accomplish at least some kind of offline usage.

after successfully building your fork i quickly realized that "IndexedDB" is not an option for react-native:
FIREBASE WARNING: Failed to initialize database persistence. It will be disabled.

it would be great if you could provide an additional (simple?) "AsyncStorage" storage adapter for testing on react-native.

cheers

p.s. i actually just signed in to github to leave this hopefully motivating comment 😉..

from firebase-js-sdk.

Hey @kr31n3r, thanks for trying this out. I just wrote a very crude wrapper around React Native's AsyncStorage. I'm planning on publishing a proper implementation once everything's settled, for RN and other platforms too, but this should get you up and running for now.

Just copy this file into your project, import the RNAsyncStorageAdapter object from it, and then enable persistence like this:

firebase.database().enablePersistence(RNAsyncStorageAdapter);

Let me know how it goes for you.

from firebase-js-sdk.

@jsayol This is sounding pretty awesome. Quick question about keepSynced() and eviction... I was under the impression that the browser doesn't do any "eviction" more finer-grained than at the database-level (so it would never purge /part/ of the server cache, for instance). Unless you're using multiple databases? So I would expect that the web client could (and ideally should) implement the same garbage collection and keepSynced() code that iOS has. Though this isn't necessary for an initial "proof-of-concept" implementation. But not implementing it probably actually makes it more likely that the browser would purge the entire offline cache, because it could get too big.

Also, FWIW- keepSynced() is actually just a trivial wrapper to keep an active listener attached to the data. It doesn't actually interact with persistence at a low-level, so it'd be trivial to implement on web, even without any of your persistence work. :-)

from firebase-js-sdk.

I was under the impression that the browser doesn't do any "eviction" more finer-grained than at the database-level (so it would never purge /part/ of the server cache, for instance)

@mikelehen I was just doing some reading about that and it looks like you're absolutely right. When space is running low, the browser will evict all the data for the least recently used origin or API, depending on the browser. But yeah, the whole database is cleared in any case, not just parts of it. Thanks for pointing that out!

I'll start looking into implementing an equivalent to the PruneForest in the iOS SDK.

Determining the size of the database to decide when to prune is going to be an issue though, since there's currently no standard way to do that with IndexedDB. A very naive approach to estimating the size could be to load the whole server cache, stringify it, and check how long it is. I don't like how inefficient that would be though, so I'm certainly open to suggestions there :)

from firebase-js-sdk.

Yeah, for what it's worth, an approximation that's off by a factor of 2 or whatever would probably be good enough. It'll still put a bound on the size of the cache, which is the important thing. I'd try to avoid stringifying the /entire/ server cache, but we probably can't get away from walking the entire cache in some form. But we could probably keep a running approximate size without having to materialize a giant string (either stringify and .length each entry as we walk the cache or apply a less-precise heuristic to estimate sizes).

On that note, how exactly are you persisting data in the server cache? (Sorry, I haven't looked at your code; feel free to link me to the relevant bit if you prefer). The iOS client stores a separate LevelDB record for every leaf node in the cache. This results in long keys (like "/chats/xyz12345/messages/abc67890/sender" with small values, but LevelDB does prefix compression that makes that very efficient (common key prefixes are shared from one row to the next so the storage isn't duplicated). I'm not sure if that approach would be as efficient with IndexedDB, so I'm curious what you have chosen. :-)

from firebase-js-sdk.

@jsayol This is way exciting! I love the idea of extracting this into its own module as that will minimize the impact that this change will have on other users. When you are ready to coordinate on the testing, get a PR open and we can get @mikelehen and @jdimond to help with the validation.

As for storage adapters, we probably want to formalize that interface into a spec that others can use to write storage adapters for the platforms that aren't built in (ReactNative, Cordova, etc, etc). When you are happy with that interface (i.e. https://github.com/jsayol/firebase-js-sdk/blob/db-persistence/src/database/persistence/storage/StorageAdapter.ts#L11-L130), let's iterate on a spec doc that we can publish alongside this.

from firebase-js-sdk.

The iOS client stores a separate LevelDB record for every leaf node in the cache. This results in long keys (like "/chats/xyz12345/messages/abc67890/sender" with small values

@mikelehen I followed the same approach (the iOS SDK code was a great source of information while I was implementing this so it shouldn't come as a surprise if my solution is similar overall ;))
Now that you bring it up I might look into more performant alternatives, though it needs to account for the fact that IndexedDB won't be the only storage engine used. In certain scenarios the user might choose to use a different one (like we've seen just a couple comments above with React Native's AsyncStorage) so we shouldn't make any assumptions about the underlying engine. Unless we want to prioritize better IndexedDB performance at the expense of any other alternatives, which I guess one could make a case for since it will be the default.

As for approximating the size of the IndexedDB database, we could loop through all the stored values and accumulate a total based on the type of the value. For example:

• If it's a number, count a fixed weight for it. Like 4 bytes, for example.
• If it's a string, count its length in bytes (taking encodings like UTF-8 or 16 into account)
• If it's a boolean, count only a small weight. Maybe half a byte (1/8th seems too optimistic, in general)
• If it's an array, loop its elements and apply the same algorithm recursively.
• (Also, for each entry we'd add the length of the key to the count)

It's important to note that this check wouldn't be done very often, only after a certain number of data updates from the server. The iOS SDK does it every 1000 server updates or so which is a good starting point, and could be tweaked after some experimenting.

It could also be worth trying if any browser-specific APIs are available to get the size of the database. For example, Chrome has navigator.webkitTemporaryStorage.queryUsageAndQuota(). If that or any other API are available in the platform and work then we can use that number instead, otherwise we keep going with the manual check.

from firebase-js-sdk.

@jshcrowthe Thanks! :D I'll have to put all this on hold for a few days as I'll be busy with other stuff but I'll start working on the tests next week, or maybe a bit later if I'm still implementing pruning and all that.

As for the StorageAdapter and StorageAdapterWriteBatch interfaces there might still be some changes there so that's very much a work in progress. I've made an effort to document them thoroughly though so it shouldn't take much work to write that spec towards the end.

It might be a good idea to have some sort of brainstorming session somehow to figure out if any other methods might be necessary, even if we don't use them right now. Once all this goes live I'd like to keep changes to those interfaces to a minimum, or ideally none at all.

from firebase-js-sdk.

@jsayol Yep! I think that kind of size approximator would be perfect... and using internal APIs where available is also a great idea. :-)

from firebase-js-sdk.

@jsayol As for the storage format, it'd be interesting to do some performance profiling and see how things look. I believe Chrome uses LevelDB for its IndexedDB storage under the covers which may make it similar to LevelDB on iOS, depending on how much extra overhead there is. Other browsers (and ReactNative AsyncStorage) may have a much harder time with lots of small rows, but I don't know!

One other big question. What are doing (or planning to do) about multi-tab access? That was always one of the big concerns with tackling web offline... If you have multiple tabs modifying the offline cached data, you can easily end up with data corruption / inconsistencies. iOS and Android don't have this problem since apps are single-instanced on mobile.

The easy / safe option is to detect and prevent multi-tab access so that it's not a concern, but obviously that makes it harder to build an ideal UX for web apps. It would probably be 100% fine for React Native, PWAs, etc. though.

from firebase-js-sdk.

One other big question. What are doing (or planning to do) about multi-tab access?

@mikelehen Ah, good point. Hadn't thought about that. I think for now the best option is to prevent multi-tab access and revisit that decision in the future.

Reading data from multiple tabs is not an issue, so we would only need to lock writes to a single tab (whichever requests access first, I'll look into how that can be implemented). With that approach, the worst-case scenario is that some data won't be persistence in some situations, so basically no worse than it is now, but most of the time it would still be persisted. I'm OK with that.

from firebase-js-sdk.

Regarding initial RNAsyncStorageAdapter testing:

first of all thanks again for that stunning fast response and providing this wrapper around React Native's AsyncStorage. Despite some initial quirks my first tests look promising and right now my current project works with persistence enabled.

now about these quirks:

• In RNAsyncStorageAdapter.js getAll() a "return" is missing. it should obviously look like this:

getAll(dbName, storeName, prefix) {
    return this.keys(dbName, storeName, prefix)
        .then(...

i guess it was to obvious for me to find this typo quickly without deeper diving into the code for debugging. it wasn't easy to track down as i only got a Failed to initialize database persistence. It will be disabled. again. (i have seen you already added a // TODO for this issue inside your code 👍)

• the usage of ref.once('value') does not work as expected and returns the whole tree.
  e.g. i get

getProblemIDs:{
  "tE7ldHnJRv4zHiHjPoDlizpTAgw_server_": {
    "boards": {
      "666": {
        "problems": {
          "1497964075540": true
         ...

instead of

getProblemIDs:{
  "1497964075540": true
  ...

• the usage of ref.on('value') shows the same behavior and sometimes returns the including storeKey defined in RNAsyncStorageAdapter.js. as above. the good thing is it always returns the expected values later on.
• i had to add a couple more snap.val() checks for null that where not needed before

i'll keep you updated about further testing next week.

from firebase-js-sdk.

Here's another small one in
src/database/persistence/query/TrackedQueryManager.ts hasActiveDefault() line 165:

(map: TrackedQueryMap) => map[Query.DefaultIdentifier].active);

should check for undefined as well

(map: TrackedQueryMap) => {
    const trackedQuery = map[Query.DefaultIdentifier];
    return trackedQuery && trackedQuery.active;
})

This eliminates RN red screen of death undefined is not an object (evaluating 'map[_Query.Query.DefaultIdentifier].active')

from firebase-js-sdk.

Thanks @kr31n3r! Hopefully I will soon have some time to start writing tests and catch all these little things.

from firebase-js-sdk.

Hello. Thanks very much for all this great work. I've tried to pull the branch mentioned above and build it but am having issues. To shortcut my frustration does anyone have any of the built 'dist' folder that they could share with me? Am really interested in trying this. Perhaps @kr31n3r ?

from firebase-js-sdk.

Firestore Announcement blog post
Firestore Docs
Firestore and Web introduction video - discusses nearer the end how syncing works in practice (not implementation)

I can confirm that Firestore does indeed support offline mode in Web in addition to the mobile platforms.

from firebase-js-sdk.

Firestore looks great, but glad you're keeping the feature request active.

update: Fully migrated to Firestore. I started with some analytical collections, not intending to move further. But, like RTDB, the API is clean and clear, so why not... Kudos to the team for a great product.

from firebase-js-sdk.

I'm using the storage APIs and storage event in https://github.com/dabblewriter/browserdb to help with cross-window (browser tab) updates whether online or off. PouchDB does the same. I think this would be the best option for that.

from firebase-js-sdk.

Actually I realized that multiple firebase connections writing to one database for my app isn't ideal, so for my app I will be implementing leader-election from the Raft consensus algorithm and only allowing 1 tab to be connected to Firebase at a time.

from firebase-js-sdk.

Is offline first supported for Cloud Firestore?
I got lots of errors with my existing setup after adding enableOffline.

from firebase-js-sdk.

Yes it is.

enableOffline? I believe the call is enablePersistence, see the docs here

What does the error say? I would suggest opening another issue for this (since this is tracking persistence for the Realtime Database).

from firebase-js-sdk.

FWIW, I've moved to Firestore for my PWA, which seems to support offline OK (via firebase.firestore().enablePersistence()), but it throws a lot of console error messages when offline and definitely isn't a good offline first experience.

from firebase-js-sdk.

@paulpv - with firestore do you get the two events back from an onSnapshot query when you set it to includeQueryMetadataChanges? (background: https://groups.google.com/forum/#!topic/google-cloud-firestore-discuss/cLqy_zH2no4)

from firebase-js-sdk.

@callagga Yes, I do get two onSnapshot when using .onSnapshot({ includeQueryMetadataChanges: true }, (querySnapshot) => { ... }), and one when not using { includeQueryMetadataChanges: true }

from firebase-js-sdk.

I'm looking into solving offline again for a crossplatform web app (mobile, desktop, Chrome OS).

So, persistence hasn't been solved yet for the RTDB, right?

I also took a look at Firestore but I saw this in the docs:

For the web, offline persistence is an experimental feature that is supported only by the Chrome, Safari, and Firefox web browsers.

Which doesn't inspire much confidence to be honest... and knowing Edge is not supported is a deal breaker for us which represents about 25% of our users.

Is there work being done on offline persistence for Firestore for the web or is this feature going to remain "experimental"?

from firebase-js-sdk.

@PierBover Firestore web persistence is definitely under active development. Right now we're focused on implementing multi-tab support. The Edge issue is unfortunate. If Microsoft implements https://developer.microsoft.com/en-us/microsoft-edge/platform/status/indexeddbarraysandmultientrysupport/ then we should be able to support Edge easily (feel free to add some upvotes to their roadmap :-)). Barring that, we're going to have to rework how we store / index our persisted data in order to work around the limitation.

from firebase-js-sdk.

Thanks for your fast answer @mikelehen !

For my use case multi-tab support is not a priority, but Edge support is.

And what about offline persistence for the RTDB @jsayol ?

from firebase-js-sdk.

I am also hoping for RTDB persistence. I have a custom solution using IndexedDB and will be using https://github.com/dabblewriter/tab-election for multi-tab support so that only one tab commits updates to the database. You could use that (or something like it) and have a single tab be the one with all the watches. Sorry I can't help contribute more to this!

from firebase-js-sdk.

Thanks @jacwright my app will not run in multiple tabs. We are working on our own native wrappers using the web engine of each OS (Android, iOS, Mac, Windows).

from firebase-js-sdk.

I am currently caching data myself and I wonder if we could expose the hashing in Firebase to at least allow it to be taken advantage of outside the persistence feature. I would really like to take advantage of the bandwidth savings and am happy to control the caching myself.

from firebase-js-sdk.

Folks, I have been looking into this for some time now and the answer is clear but not what we want to hear.

Browsers "parse" JS and thus we rely on browser technology for local storage as we do not store data on disc.

Android / OS compile onto the disc if necessary so the access to disc storage and light databases like sqlite is possible.

Browser local cache or localdb or indexeddb only allow 4mb of data per app before recycling old data... So ... Offline first would mean user cannot create more than 4mb of data. This is not realistic in the long term.

You could choose to only make certain tables offline first.... But even then it's unclear if they might one day loose old data and sync will ruin your cloud store as well

The only option is to make your site into an online portal with more functionality and allow offline for your android app... Both feed the same node API.

I am waiting for a miracle in browsers but don't loose any sleep on it
Hope this helps

from firebase-js-sdk.

What we ended up doing in previous projects is to abstract the persistent storage medium depending on the platform for Cordova, Electron, or UWP web apps. Basically we ended up saving .json files to disk (which are truly persistent) with the complete Vuex state of our Vue app.

It's rudimentary... but it works. We have a couple thousands users with very bad connectivity that remain offline like 80% of the time.

from firebase-js-sdk.

@jacwright very true... I didn't mean to sound pessimistic :D.

The only thing I can't wrap my head around is the security layer when working with on disc files @pierbober... Was wondering if you could share some pearls of wisdom down on us mere mortals about how you managed to make the local database on client side secure (given its fully under their control potentially)... It's the last piece of my puzzle

from firebase-js-sdk.

Was wondering if you could share some pearls of wisdom down on us mere mortals about how you managed to make the local database on client side secure

I think you may be overestimating what we did... I'd like to clarify that we do not have the entire DB there, only the state of our app from Vuex (like Redux for Vue).

On iOS, Android, and Chrome OS the user has no access to the JSON files we are saving.

On Mac and Windows (via Electron or UWP) we simply encrypt the files before writing to disk and decode them in memory.

from firebase-js-sdk.

Is this thread about implementing offline persistence or true offline first capabilities? Ie, is the target to be able to add and query items in a locally persisted Firestore-compatible database without requiring any network activity or even a cloud Firestore database setup?
With PouchDB, this is completely possible since remote syncing is optional and can be added at a later stage.

from firebase-js-sdk.

Is there a solid answer for how to use offline persistance with real time Db yet? I had to upgrade my angular app and the old angularfire2-offline has been depricated. And to move over from RTDb to firestore I cant do just yet.

from firebase-js-sdk.

I have come up with a system to do it myself adding a modified field on every record. Then using indexeddb and the time offset I keep the local records and the remote records in sync whenever online. This reduces the traffic by only grabbing everything that has been modified since I last asked (I store the last modified timestamp received in an indexeddb store too). It's been working well, but having built-in support like Firestore does would be much nicer. It took me awhile to put this together and puts certain requirements on my data structure.

from firebase-js-sdk.

Hey @jsayol,

Good luck with this!

Data Storage
If you wanted to pursue an approach to avoid many entries with small values, you could take a look at what we do on Android. Each row contains a tree node (possibly containing an entire subtree). If the node would be too large, then we split it into multiple rows. This works pretty well, though breaks down if you have large lists of very small nodes. The list will be too big to fit in a single node, but the individual nodes are very small, leading to poor storage efficiency. Anyway, for more details see https://github.com/firebase/firebase-android-sdk/blob/c9f213cac589595580a04a89784a44e0ff19c39b/firebase-database/src/main/java/com/google/firebase/database/android/SqlPersistenceStorageEngine.java#L70 and https://github.com/firebase/firebase-android-sdk/blob/master/firebase-database/src/main/java/com/google/firebase/database/android/SqlPersistenceStorageEngine.java#L842. But you may as well stick with what you're doing initially and see if it's a problem.

Multi-tab access
The key to the Firestore approach is a "lock" in IndexedDb. Every tab assigns itself an ID and one tab writes its ID to the "lock" object store in IndexedDb. Then in every subsequent transaction it verifies that it still holds the lock before doing any writes. To guard against crashed tabs holding the lock there's also a timestamp in the lock and other tabs can take over the lock if it's too stale. Even with multi-tab we use a similar approach to nominate one tab as the "primary."

Cache policy
That sounds very reasonable. We do something similar in Android: https://github.com/firebase/firebase-android-sdk/blob/c9f213cac589595580a04a89784a44e0ff19c39b/firebase-database/src/main/java/com/google/firebase/database/core/utilities/NodeSizeEstimator.java#L34

from firebase-js-sdk.

Yep! Agreed that's the only option. 👍

from firebase-js-sdk.

Any progress on this feature? Would be really useful for my website.

from firebase-js-sdk.

@jsayol do you have a copy of what you have so far? I am interested in seeing how much work it is. I would personally skip trying to store the data efficiently in indexDB, for my personal needs very little data beyond the user record needs to be stored, but startup time is paramount. A store ineffecient offline first implementation would be better than nothing!

from firebase-js-sdk.