dalenguyen / firestore-backup-restore Goto Github PK

There is no key "databaseURL" in "servicceAccount"
=> This affects to Initialize App when import Json to Firestore in my case.

export const initializeApp = (serviceAccount: object, name = '[DEFAULT]') => {
  if (
    admin.apps.length === 0 ||
    (admin.apps.length > 0 && admin.app().name !== name)
  ) {
    admin.initializeApp(
      {
        credential: admin.credential.cert(serviceAccount),
        databaseURL: serviceAccount['databaseURL'],
      },
      name
    )
    admin.firestore().settings({ timestampsInSnapshots: true })
  }
  return true
}

Hey

I asked not too long ago on how this one was coming along: #25 Shortly after you release a new version supporting refs! 👏

However I can't seem to get it to work. A example:

The object I want to reference has the path: /testCollectionA/documentA/

I then create a new collection which looks like this: /testCollectionB/documentB/subCollectionB/subDocumentB. However the refs just get input as strings.

I am probably not understanding the documentation. I tried entering refs: ['references'] and refs: ['testCollectionA/documentA']. I just assume I am doing it wrong. What's the correct way to use the refs option?

I also just tried entering a field named exampleReference with a single ref in subDocumentB, and using that field in refs: ['exampleReference']. Still comes out as string.
testCollectionA/documentA

Do both collections have to be in the same json file for refs to work? I use a JSON file per collection.

I just exported an collection and try to import it but when I do the geopointers are numbers and not as geopointer

my import code:
restore('export.json', { dates: [], geos: ['location', 'polylines', 'centerPoint'], refs: [], })

The small version of the json I try to import:

{
    "locations": {
        "doc": {
	    "initialZoom": "15",
            "centerPoint": { //works fine
                "_latitude": 51.5883, 
                "_longitude": 4.77626
            },
            "polylines": [{ //works fine
                    "_latitude": 51.58859,
                    "_longitude": 4.77626
                }, {
                    "_latitude": 51.58895,
                    "_longitude": 4.77604
                }
            ],
            "markers": [{
                    "title": "Punt 1",
                    "location": {
                        "_latitude": 51.58858, //inserted as number
                        "_longitude": 4.7764  //inserted as number
                    }
                }, {
                    "title": "Punt 2",
                    "location": { 
                        "_latitude": 51.58878,  //inserted as number
                        "_longitude": 4.77624  //inserted as number
                    } 
         	}
            ]
        }
    }
}

deleted

How to prevent overwriting existing documents within a collection.
Importing new data to existing collection overwrites the documents present.

Hi @dalenguyen

I think you merged out my changes for the support of nested dates, geos and refs. Can it be?

I sending a Json to my database in Firebase, but all time i sending the collections is rewriting..

Hi, i'm trying to import a JSON with field nested timestamp 2 level, but i don't know how to do it.
Sample bellow with 3 field: date, date1, date2

Please, Help for me!
Thanks!

{
  "test": {
    "first-key": {
      "website": "dalenguyen.me",
      "array": ["1", "2", "3"],
      "date": {
        "_seconds": 1534046400,
        "_nanoseconds": 0
      },
      "custom": {
        "lastName": "Nguyen",
        "firstName": "Dale"
      },
      "location": {
        "_latitude": 49.290683,
        "_longitude": -123.133956
      },
      "email": "[email protected]",
      "author" : {
        "name": "dalenguyen.me",
        "date1": {
          "_seconds": 1534046400,
          "_nanoseconds": 0
        }
      },
      "subCollection": {
        "test/first-key/details": {
          "33J2A10u5902CXagoBP6": {
            "dogId": "2",
            "dogName": "hello",
            "date2": {
              "_seconds": 1534046400,
              "_nanoseconds": 0
            }
          },
          "MSZTWEP7Lewx0Qr1Mu5s": {
            "dogName": "lala",
            "dogId": "2",
            "date2": {
              "_seconds": 1534046400,
              "_nanoseconds": 0
            }
          }
        }
      }
    }
  }
}

Hello;
i truly love this but i need some help, i did a sample import but the issue is that all the things i imported went from 0,1,3,4,5,6,7,8,9,10,11.. is there anyway to get random document id?

https://github.com/complexorganizations/firestore-export-import

Just going to rewrite this in golang cause of the amount of errors.

RangeError [ERR_FS_FILE_TOO_LARGE]: File size (20954288616) is greater than possible Buffer: 2147483647 bytes
RangeError [ERR_FS_FILE_TOO_LARGE]: File size (20954288616) is greater than possible Buffer: 2147483647 bytes
    at FSReqWrap.readFileAfterStat [as oncomplete] (fs.js:264:11)

As the subject says - this is the only output I'm receiving from the example providing.

Hey,

I've been testing this out and was wondering if there were any risks of exporting data, and what they are?

I have a collection containing ~80000 documents that can go from 50kB to ~900kB. When trying to fetch the entire collection to push it inside a backup file, I get the following error:

Error: Deadline exceeded
    at Http2CallStream.<anonymous> (<node_modules>\@grpc\grpc-js\build\src\call.js:68:41)
    at Http2CallStream.emit (events.js:208:15)
    at <node_modules>\@grpc\grpc-js\build\src\call-stream.js:71:22
    at processTicksAndRejections (internal/process/task_queues.js:75:11) {
  code: 4,
  details: 'Deadline exceeded',
  metadata: Metadata { options: undefined, internalRepr: Map {} }
}

Here is the code I'm using to fetch data:

const firestoreService = require('firestore-export-import');
const serviceAccount = require('./service-account.json');

// Initiate Firebase App
firestoreService.initializeApp(serviceAccount, 'my-database-url');

// Start exporting your data
firestoreService
  .backup('lists')
  .then(data => {
    console.log(Object.keys(data.lists).length);
    fs.writeFileSync('./backup-lists.json', JSON.stringify(data))
  });

I got a json files with >20k entries. Each one got datetime fields like that : "2018-01-02T01:00:00"
After importing, firestore set this field to string. Is there any way to import them with the correct type: Timestamp ?

Regards,

Fairy

I noticed this package requires you to explicitly state the Timestamp fields in the documents when restoring.

Would you be interested in having an option that allows for automatic timestamp parsing in documents? We've implemented it several times before and can submit a PR 👍

I try to export a large collection. The code I use, to do the export looks like this:

firestoreService.initializeApp(serviceAccount);
const collectionToExport = 'intensityDetails';

firestoreService
    .backup(collectionToExport)
    .then(
        (data) => {
            console.log(Object.keys(data[collectionToExport]).length);
            fs.writeFileSync(collectionToExport + '.json', JSON.stringify(data));
        });

When I run the export, I get this error message:

FAILED_PRECONDITION: The requested snapshot version is too old.

Is there any solution to this?

There is any flag/attribute/command to clear the collection prior to a restore?

Hi Dale,

Import export works great, however it does not import relations as relations but as maps. Imported reference contains three fields containing maps: _converted, _firestore and _path.

How about instead of setting the map you create a real reference with the _path values? This way the information would remain as same means as in the original.

Thanks!
Jose

Hi there!

I would like to ask how to correctly use restore for date timestamp when data looks like this:

 "wFLrjgdW7Jjmjs5sX": {
  "_id": "wFLrjgdW7Jjmjs5sX",
  "notifications": [
    {
      "date": {
        "_seconds": 1589998976,
        "_nanoseconds": 490000000
      },
      "type": "new",
      "message": "Contract has been created.",
      "user": "Philip Rawsen"
    },
    {
      "date": {
        "_seconds": 1589999521,
        "_nanoseconds": 985000000
      },
      "type": "payment",
      "message": "Customer payment for USD $ 67.92",
      "user": "Lena Perks"
    }
  ]
}

When I use the command

// Start exporting your data
firestoreService
  .restore(`./collections/${collectionName}.json`, {
    dates: ['notifications.date']
  });

I cannot get it to restore it as a timestamp. It restores as a map field.
I have also tried with 'notifications[].date' and simply 'date' unsuccessfully.

Any help is greatly appreciated. Thanks.

https://firebase.google.com/docs/firestore/manage-data/export-import

if document contains a subcollection but no fields, export method will ignore the document.

Sorry, this is more of a question. I am new to web development.

Can this tool be used to migrate db from Realtime to Firestrore? Will it support deeply nested documents and a single large json file (around 50 mb)?

Hi, i'm trying to import a JSON with field timestamp, but i don't know how to do it.

I am trying to import some data, but my data in the documents is nested. For example:

location: {
  geohash: 'xxx',
  geopoint: {
    _latitude: '...',
    _longitude: '...'
  }
}

When I try to import now with the following, it doesn't work. Would be very nice, if the restore function would traverse the JS-Object, when a "." is present.

firestoreService.restore('collections.json', {
    geos: ['location.geopoint'],
});

Firestore supports batched operations, this library would be really good if you can support this. a batch can support 500 operation max, here's the URL for batched operations

https://firebase.google.com/docs/firestore/manage-data/transactions#batched-writes

// Get a new write batch
let batch = db.batch();

// Set the value of 'NYC'
let nycRef = db.collection('cities').doc('NYC');
batch.set(nycRef, {name: 'New York City'});

// Update the population of 'SF'
let sfRef = db.collection('cities').doc('SF');
batch.update(sfRef, {population: 1000000});

// Delete the city 'LA'
let laRef = db.collection('cities').doc('LA');
batch.delete(laRef);

// Commit the batch
return batch.commit().then(function () {
  // ...
});

Please some help, Im using node but when I try to restore a collection I got this error, any help please?

errorInfo: {
code: 'app/no-app',
message: 'The default Firebase app does not exist. Make sure you call initializeApp() before using any
of the Firebase services.'
},
codePrefix: 'app'
}

hi i am i get successful backup and upload file to bucket now i want to resotre that backup but i cant get i facing some error

path must be a string or Buffer

and give direct URL to function give same error how i restore data

please help me thanks in advance..

firestoreService.backups()

firestoreService.backups(['col1', 'col2' ]) not working

single collection import export is working perfectly , but all collection backup not working.

I would like to export all the collections in a given database. I have come up with this code that works (pending pull request #6):

    const db = firestoreService.admin.firestore();
    // get all the root-level paths
    db.getCollections().then((snap) => {
        let paths = [];
        snap.forEach((collection) => paths.push(...collection._referencePath.segments));
        // fetch in parallel
        let promises = [];
        paths.forEach((segment) => {
            let result = firestoreService.backup(segment);
            promises.push(result);
        });
        // assemble the pieces into one object
        Promise.all(promises).then((value) => {
            let all = Object.assign({}, ...value);
            console.log("All is " + JSON.stringify(all));
        });
    });

Is there a better or more efficient way to get all of the data?

I'd like to save the data I export from firestore to a file in my local storage disk instead of doing console.log(). Is there a way to do so by adding to the current code?

Hello,

You shall add a 3rd optional name?: string param in your initializeApp function in order to give a unique app name to initialize multiple apps.
This way we could use you great tool to copy collection from a Firebase environment to another.

Hi, i'm trying to import a JSON with some reference fields, but i don't know how to do it. If i put this field in JSON as string, obviously, it's imported as string.

It's that possible? Thanks!

Would be nice not only be able to export collections, but also users.

How can I backup specific document with a subcollection in a collection. I tried this code

const { backup, backups, initializeApp } = require('firestore-export-import')
const serviceAccount = require('./service-account.json')

initializeApp(serviceAccount)

backup('data/00Grg9QoguND4o9PiR2Eb1p23sn2sXXSS').then((data) =>
  console.log(JSON.stringify(data))
)

I got this error

Error: Value for argument "collectionPath" must point to a collection, but was "data/00Grg9QoguND4o9PiR2Eb1p23sn2sXXSS". Your path does not contain an odd number of components.

I want to have a selective backup per user Id. Any Advice how to this?

when geopoint is exported, it is serialised as plain json object and after importing such file in database it will be a plain map, not a 'GeoPoint' field. I'd consider adding some 'type' discriminator field during export and using it to restore a proper type later on.

First of all, great package here, its amazing!

Just want to ask if subcollections can be imported, the front page says yes but I can't seem to find relevant code samples.

Hi,
When I try to export my data I got objects as [Object] (not the real data).

so e.g.:
{key: { disciplines: [Object], finance: [Object] }}

Could you tell my why?

Hello,

i've just try to export a collection with some reference fields into maps but it doesn't seem to work.

Export result :

Is there a way to handle this case with options refs ?

Thanks

I've found that if you do an export without any options, and one of your fields is of type DocumentReference, then the library thinks the DocumentRef is a JSON object and exports the entire object. This is very dangerous because this includes any firebase settings (Private Keys included!).

If in the wrong hands, your private keys could be exposed. This is not good.

Attaching a screenshot of the response.

I noticed this issue with another library and I stopped using that one because of the risk. Something possible to try is checking the field values like so:

if (rawValue.constructor.name === 'DocumentReference')

https://github.com/complexorganizations/firestore-manager

It pretty much does the same thing but written in golang so there is no errors nor memory limits or any other issues in between.

give it a try.

i think the problem is with this line because this overwrite the extisting data so the last sub collection is only printed
data[collectionName][doc.id]['subCollection'] = subColData;

or is there is something wrong with my code

firestoreService
  .backup('collection-name')
  .then((data) => { 

    let json = JSON.stringify(data, null, 2);
      
      console.log(json)
      
      fs.writeFileSync('backup.json', json);

  });

Hey guys,

The restores method doesn't return a Promise, don't should return like others methods?

I implemented that resource in firebase functions (google cloud), but because no exist promise returned is not work propely, just sometimes, becouse the request responde before finsh the restore process.

Thanks for any help.

Thanks for your sharing. But I have issue on export the subcollection

This is, the following works
'node export.js users'

But the following does not work
'node export.js users userbehaviour'

Am I using it correctly? Please kindly advise. Thanks.

I'm trying to import locations with Geopoints, but the Geopoints are showing as Maps in Cloud Firestore

Import Code:
// Imports
const firestoreService = require('firestore-export-import');
const firebaseConfig = require('../config/config.js');
const serviceAccount = require('../config/serviceAccount.json');

// JSON To Firestore
const jsonToFirestore = async () => {
  try {
    console.log('Initialzing Firebase');
    await firestoreService.initializeApp(serviceAccount, firebaseConfig.databaseURL);
    console.log('Firebase Initialized');

    await firestoreService.restore('./services-output/locations.json');
    console.log('Upload Success');
  }
  catch (error) {
    console.log(error);
  }
};

jsonToFirestore();

locations.json

{
  "locations" : [
    {
      "email"   : "[email protected]",
      "website" : "dalenguyen.me",
      "Location": {
        "_latitude": 49.290683,
        "_longitude": -123.133956
      }
    }
  ]
}

Cloud Firestore (Shows as Map)

Hi,

if my date field is inside an object, adding ['objectkey.date'] in the optional parameter is not working to add date as a timestamp. it throws an error as below

Please check your date parameters!!! [ 'updated.date' ]

Any idea, how to add optional date keys when the date is inside an object in the JSON file?

please help @dalenguyen

Thanks

Hello,

I have an issue when import with the right keys that have references:

firestoreService.restore('./users.json', {
    dates: ['time', 'birthday', 'lastLogin','createdAt','updatedAt','date'],
  **refs: ['following', 'followers'],**
});

return:

{ status: false,
  message:
   'Value for argument "documentPath" is not a valid resource path. Path must be a non-empty string.' }

How can I fix this?

Thank you

I'm trying to run automated backups using a pubsub firebase function, but when I run firestoreService.initializeApp(), it conflicts with my admin.initializeApp(). The Error is:

Error: The default Firebase app already exists. This means you called initializeApp() more than once without providing an app name as the second argument. In most cases you only need to call initializeApp() once. But if you do want to initialize multiple apps, pass a second argument to initializeApp() to give each app a unique name.

Any idea?

Hello, i am trying to import local data to firestore using example code firestoreService.restore.... but i am getting Unhandled promise rejection error, screenshot below. Any ideas on how to fix this?

For testing purposes, we want to export a small subset of a Firestore database.

Ideally we'd like to get only a single document, from all collections and subcollections,
So instead of 10,000 documents from the users collection, just export 1 document.

Is this currently possible?

It would be cool to be able to restore a JSON backup using this tool to the Firestore Emulator.

It would be great to do the following:

1. ✔️ Backup your database to JSON using this tool
2. ✔️ Manually delete most irrelevant data, expect for some test data.
3. Use this tool to restore the JSON to the emulator in a test like this

I have an array of locations like below:

"path": [
        {
          "_latitude": 42.55318642480909,
          "_longitude": 69.16420140448128
        },
        {
          "_latitude": 42.553242362145895,
          "_longitude": 69.16332591108834
        },
        {
          "_latitude": 42.55337671961003,
          "_longitude": 69.16292894415413
        },
        ...
]

How to set the param geos? I tried geos: ['path', 'path[]', '[path]']