Added field in under Extension configuration did not trigger Typesense update about firestore-typesense-search HOT 9 CLOSED

When you changed the extension configuration, could you confirm that Firebase re-deployed both the backfill function and the indexingOnWrite function?

Could you try re-installing the extension from scratch with the new field added, and see if that helps?

Also, do you see any logs in the backfill functions in the Firebase web console, showing any errors?

from firestore-typesense-search.

Apologies I said there was no attempt by the extension function to update. I couldn't find anything under indexToTypesenseOnFirestoreWrite function log, but now I found below under backfillToTypesenseFromFirestore function log. Let me know if you find something below, and still want me to try re-installing. I probably will try with a new, smaller setup as I don't want to disrupt the live app.

--- backfilling function triggered ---

{
  "textPayload": "Backfilling [a, b, c (last one newly added)] fields in Firestore documents from products into Typesense Collection products on xyz.typesense.net",
  "insertId": "insertID",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "region": "us-central1",
      "project_id": "projectID",
      "function_name": "ext-firestore-typesense-search-backfillToTypesenseFromFirestore"
    }
  },
  "timestamp": "2024-01-10T04:09:17.988118Z",
  "severity": "INFO",
  "labels": {
    "execution_id": "executionID",
    "runtime_version": "nodejs16_20231231_16_20_2_RC00",
    "instance_id": "instanceID"
  },
  "logName": "projects/projectID/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/projectID/traces/traceID",
  "receiveTimestamp": "2024-01-10T04:09:18.307724632Z"
}

Following that is an error:

Import error ImportError: 0 documents imported successfully, 1000 documents failed during import. Use `error.importResults` from the raised exception to get a detailed error reason for each document.
    at ImportError.TypesenseError [as constructor] (/workspace/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
    at new ImportError (/workspace/node_modules/typesense/lib/Typesense/Errors/ImportError.js:25:28)
    at Documents.<anonymous> (/workspace/node_modules/typesense/lib/Typesense/Documents.js:154:39)
    at step (/workspace/node_modules/typesense/lib/Typesense/Documents.js:48:23)
    at Object.next (/workspace/node_modules/typesense/lib/Typesense/Documents.js:29:53)
    at fulfilled (/workspace/node_modules/typesense/lib/Typesense/Documents.js:20:58)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  importResults: [
    {
      code: 409,
      document: '{"document content"}',
      error: 'A document with id <DocID> already exists.',
      success: false
    },
.... 

Then these two messages:

Done backfilling to Typesense from Firestore
Function execution took 9275 ms, finished with status: 'ok'

And this error again:

Import error ImportError: 0 documents imported successfully, 7366 documents failed during import. Use `error.importResults` from the raised exception to get a detailed error reason for each document.
    at ImportError.TypesenseError [as constructor] (/workspace/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
    at new ImportError (/workspace/node_modules/typesense/lib/Typesense/Errors/ImportError.js:25:28)
    at Documents.<anonymous> (/workspace/node_modules/typesense/lib/Typesense/Documents.js:154:39)
    at step (/workspace/node_modules/typesense/lib/Typesense/Documents.js:48:23)
    at Object.next (/workspace/node_modules/typesense/lib/Typesense/Documents.js:29:53)
    at fulfilled (/workspace/node_modules/typesense/lib/Typesense/Documents.js:20:58)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  importResults: [
    {
      code: 409,
      document: '{"document content"}',
      error: 'A document with id <Doc ID> already exists.',
      success: false
    },
...

from firestore-typesense-search.

Ah this is because during a backfill, we do action=create, instead of action=upsert, so Typesense throws an error saying the document being backfilled already exists.

We didn't account for this use-case of backfilling a 2nd time when a new field is added to an existing collection. Will push out an update for this shortly.

from firestore-typesense-search.

Yay! Thanks for arranging update for this. That would be very useful. A kind note, please also take into account the user-case that a field is removed (if that's not too much to ask).

from firestore-typesense-search.

Could you try installing this pre-release version of the extension and let me know how it goes:

https://console.firebase.google.com/project/_/extensions/install?ref=typesense/[email protected]

A kind note, please also take into account the user-case that a field is removed (if that's not too much to ask).

Yup, action=upsert does this automatically

from firestore-typesense-search.

I did, below steps and result.

1. setup a new collection in firestore (3 docs, 3 fields)
2. setup a new collection in typesense
3. installed 1.3.0-rc.0, in config, included only 2 of 3 fields in Firestore Collection Fields
4. upon completion, nothing show up in typesense, even though I had a typesense_sync backfill collection in place (before this whole setup)
5. deleted the typesense_sync backfill collection, created a new one
6. all docs appeared in typesense, with 2 fields
7. reconfigured the extension to include the third field
8. upon completion, typesense is not updated, all docs still have 2 fields only
9. repeated step 5
10. all docs in typesense updated, with 3 fields

Edit: no logs available

from firestore-typesense-search.

4. upon completion, nothing show up in typesense, even though I had a typesense_sync backfill collection in place (before this whole setup)

This is expected, because only after the extension is installed, it starts listening for the creation of the typesense_sync backfill collection. It does not look at an existing typesense_sync collection.

8. upon completion, typesense is not updated, all docs still have 2 fields only

This is also expected, since the backfill only runs when explicitly triggered by creating the typesense backfill collection. It does not run automatically when the extension is re-configured / installed / re-installed.

So it sounds like things are working as designed, since all docs in Typesense now have 3 fields in Step 10?

from firestore-typesense-search.

Yes. I also tried to remove one field in config, and all work out as expected. Thanks for your lightning-fast communication and implementation!

from firestore-typesense-search.

Happy to help! And thank you for all the feedback.

I'll publish this version live shortly.

from firestore-typesense-search.