[BUG] Chroma DB Duplicate ID Error about mem0 HOT 8 CLOSED

Hey @ArmanJR @luisrock @missioncfwf @ramaschneider @Harin329 : merged PR #160 which addresses this issue.

from mem0.

Same issue here

from mem0.

Same issue here. I had no problem uploading small files, but big ones cause this issue for me

from mem0.

Same here - is there any update regarding this issue?

from mem0.

I think this could be because ids are clashing.
Right now, the ids are generated using a hash of chunk text and url. There may be scenarios where same chunk is occurring multiple times in a doc, hence the error.

@ramaschneider / @missioncfwf / @ArmanJR : possible for anyone of you to share doc file?
I just want to double check if this is the issue or something else.

from mem0.

@taranjeet sure: https://www.sufi.ir/books/download/english/molavi-en/masnavi-en.pdf

from mem0.

I opened a PR for this, setting a random bit in the ID hashing function seems like a harmless and quick fix. Not entirely sure of the implications of turning the lists into a set, but that may affect the composition of the dataset

from mem0.

Released in version 0.0.18
https://test.pypi.org/project/embedchain/0.0.18/

Closing this issue.

from mem0.