Stuck on Error: The text contains a special token that is not allowed: <|endoftext|> about obsidian-smart-connections HOT 6 OPEN

thank you so much, i finally got it to work! cheers!

from obsidian-smart-connections.

@seanrose thanks for bringing that to my attention.

That seems to be an incompatibility issue with the OpenAI tokenizer.

In the future, we could try to manually filter out tokens like the <|endoftext|> mentioned in the error.

For now, you could either switch to a non-OpenAI embedding model (may or may not work), remove/change any mentions of <|endoftext|> in your notes, or exclude any notes that contain <|endoftext|> using the exclusion settings.

🌴

from obsidian-smart-connections.

Removing mentions of those text did help. Hopefully they're rare, but it'd be useful if there were some built in way to see these exceptions outside of the console to self-resolve,

from obsidian-smart-connections.

@seanrose glad it helped!

A couple things would help:

1. preventing errors from halting the process and listing erroneous notes in the settings or somewhere else for review.

2. known "stop" words listed in the model config (varies per model) and handling accordingly.

The first is already in a long list of todos. And while it won't necessarily show the exact issue, users may still be able to identify the issues given that the erroneous notes likely have some commonality, like the "stop" word.

The second will likely become a feature in the modules I'm building for securely integrating the AI models in v2.0/2.1.

🌴

from obsidian-smart-connections.

I have the same error. this is my console:

REMOVED COPY AND PASTED LOGS

from obsidian-smart-connections.

@marketfailuree The error your receiving (429) is from OpenAI rate limits. These errors, where OpenAI is blocking/restricting your account, are often solved by pre-paying $5 on your account and making sure your billing info is properly setup.

Please screenshot the logs in the future instead of copy & paste.

🌴

from obsidian-smart-connections.