[feature] Add new field to store document reference for persistent storage about guac HOT 8 CLOSED

+1 on doing it for all verbs.

We should not combine fields with separators. I think doing so will open us to issues when we'd need to escape the separator further down.

from guac.

cc @nchelluri

from guac.

I say we do it for all verbs. I think it's a bit more extra work for helping not just from an evidence perspective but debuggability and maintainability perspective for GUAC.

We can use the blobs whether or not they are API responses or documents or something else for help debugging when something goes wrong as we have the content saved. It also helps if we parse a document, API request/response, etc. differently in the future and need to re-parse the data.

from guac.

+1 , this looks good on adding this to SourceInformation

This will allow users to find the original document (or re-ingest in the case of failure) if they need.
i would say that this will not quite directly meet the usecase of "re-ingest in the case of failure". There needs to be another solution for that - something that is more on the pubsub and having a reprocessing pipeline, but that seems out of scope for this issue.

As an aside, one of the issues we've run to before with one of our other projects that has a similar data pipeline is the reprocessing is the issue of duplicates (@mdeicas has been looking at this).

from guac.

This will allow users to find the original document (or re-ingest in the case of failure) if they need.
i would say that this will not quite directly meet the usecase of "re-ingest in the case of failure". There needs to be another solution for that - something that is more on the pubsub and having a reprocessing pipeline, but that seems out of scope for this issue.

oh yes, this is not the solution to "re-ingest in case of a failure". This is your database blows up and you have to start from scratch.

from guac.

As an aside, one of the issues we've run to before with one of our other projects that has a similar data pipeline is the reprocessing is the issue of duplicates (@mdeicas has been looking at this).

Interesting, if there is lessons learned we can apply here that would be great.

from guac.

Sorry for the late response, but I think the lesson learned is that an ingestion pipeline may have been designed with an assumption of only ingesting documents once, or otherwise to be idempotent, and so it won't support re-ingesting documents to pick up new parsing features. It might be prudent to document this somewhere for clients?

from guac.

hmmm that is an interesting case. I added to our agenda to discuss

from guac.