arro / mongo-airtable Goto Github PK

Is it possible to not pull hidden fields from an airtable view? This would allow one collection to be used for more API calls and act more dynamically.

I'm considering to use your tool to pull our entire Airtable base to our Staging webserver's MongoDB, from which I will then generate pages for our website. We then deploy all this static content as complete stand-alone ZIP to our Live web server.

I already use the official Airtable API JS client, but with your tool I hope to avoid problems with the Airtable API limitations (5 requests per second).

This won't be a big improvement to using the Airtable API directly, because I can delay the API requests and we are OK with waiting 10-30min. However, I'm wondering if at some point I can have the MongoDB directly on our Production and threfore skip generating the static pages on Staging server and then read/write dynamic content (e.g. Orders, User profiles) directly from the MongoDB, which in background will be synced by your tool every 10min or so.

Are you using it for scenarios like this?
Or only to sync our offline copy of the base, like for local development for example?

Asking because I've read this:
The problem is that sometimes you don't have internet access, so pulling and saving data using their API isn't feasible.

Thanks for your input!

When I delete any record in airtable and then call "npm run update", it's not deleted in mongodb
also if I delete a field in airtable and call "npm run update", the field is not deleted inside mongodb

I tried mongo-airtable to pull my base to mongoDB, my Airtable base has 12 tables and more than 6500 records...
it took 190 seconds to finish pulling and inserting
I tried tp modify some functions to make it work async, all tables start pulling at once but as soon as any table finish pulling it will start inserting into mongodb directly ... this improved the performance and took 92 seconds instead of 190 seconds
this still too much as I am intending to fire this syncing process every 5 minutes intervals or less to always keep mongodb synchronized with airtable

here is my edits to reach 92 secs... added this function to airtable.js:

 export async function initialPullAndInsert(config) {
   for (const table_to_sync of config.sync) {
     pullTable({
       ...table_to_sync,
       auth_key: config.auth.airtable
     }).then(() => {
       console.log(`end sync of ${table_to_sync.primary}`)
       const last_pulled_filename = path.resolve(`${__dirname}/../build/last-pulled.txt`);
       writeFile(last_pulled_filename, moment().format('ddd MMM D YYYY h:mm A ZZ'), 'utf-8').then(() => {
         initialInsertTable(table_to_sync);
       })
     })
   }
 }

and added this to mongodb.js:

 export async function initialInsertTable(table_to_sync) {
   const { primary, collection, database } = table_to_sync;
     await putIntoDB({
       primary,
       collection,
       database
     })
   }

and in pull.js I called await initialPullAndInsert(config);

Hi
I ran the tests and all 6 passed. When I run "npm run push", I get the follow error. I tried both simple and complex mongodb collections
(node:8198) UnhandledPromiseRejectionWarning: TypeError: (0 , _mongodb.lookForChanges) is not a function