Another 400 error about obsidian-smart-connections HOT 13 CLOSED

@vguillet check out #24 (comment) and let me know if you think it's the issue (free plan rate limits). If so, I have an update in the works that should alleviate some/all of the problem.

from obsidian-smart-connections.

Hi @Fusion

Thanks for the detailed report.

The skipping block (mtime) was a remnant of development. It's just saying that it's skipping getting embeddings because the file modified time hasn't changed. I removed this logging in the most recent patch.

The next issue is being worked on here #12

It's partially fixed, and the null issue is handled, but we're still figuring out what is causing OpenAI to respond with a 400 error in the first place. The more recent patch version also adds information to the log about the 400 error, but I'm not sure that will give us anything else.

I'm glad you added the details about the request. Particularly the <some enormous escaped page content here> part. It's making me think that a string that exceeds the token limit is making it to the request, which would cause the 400 error. To address this, I just decreased the maximum length string that will be passed into the request from 30K characters to 25K characters. In the future, this can be improved by better calculating the exact number of tokens, but reducing the current character limit should be a good indicator of whether the length is the issue or something else like character encoding.

Let me know if you continue having issues after the latest patch (1.0.30).

Thanks again for the report,
Brian

from obsidian-smart-connections.

Unfortunately, even after performing a force reload, I am still seeing the issue.

Please note that, below, the "enormous" content, encoded, is 33716 characters long.

Error:

undefined
plugin:smart-connections:595 Error: Request failed, status 400
    at new t (app.js:1:710073)
    at Aw (app.js:1:710265)
    at app.js:1:710942
    at app.js:1:235505
    at Object.next (app.js:1:235610)
    at a (app.js:1:234349)
plugin:smart-connections:510 Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'save_embeddings_to_file')
    at process_batch (plugin:smart-connections:510:20)
    at async SmartConnectionsPlugin.get_file_embeddings (plugin:smart-connections:400:11)
    at async Promise.all (index 1)
    at async SmartConnectionsPlugin.get_all_embeddings (plugin:smart-connections:189:9)
    at async SmartConnectionsPlugin.find_note_connections (plugin:smart-connections:656:7)
    at async SmartConnectionsView.render_note_connections (plugin:smart-connections:1020:21)
process_batch @ plugin:smart-connections:510
await in process_batch (async)
eval @ plugin:smart-connections:958
e.tryTrigger @ app.js:1
e.trigger @ app.js:1
t.trigger @ app.js:1
t.activeLeafEvents @ app.js:1
s @ app.js:1
l @ app.js:1

Response:

{
    "status": 400,
    "headers": {
        "date": "Thu, 12 Jan 2023 05:11:06 GMT",
        "content-type": "application/json",
        "content-length": "294",
        "access-control-allow-origin": "*",
        "openai-organization": "user-<redacted>",
        "openai-processing-ms": "138",
        "openai-version": "2020-10-01",
        "strict-transport-security": "max-age=15724800; includeSubDomains",
        "x-request-id": "<redacted>"
    }
}

Request:

{
    "usedParams": {
        "model": "text-embedding-ada-002",
        "input": "<some enormous escaped page content here>"
    },
    "reqParams": {
        "url": "https://api.openai.com/v1/embeddings",
        "method": "POST",
        "body": "{\"model\":\"text-embedding-ada-002\",\"input\":\"<some enormous escaped page content here>\"}",
        "headers": {
            "Content-Type": "application/json",
            "Authorization": "Bearer <redacted>"
        }
    }
}

from obsidian-smart-connections.

@Fusion Thanks for the update!

That size, 33716 characters long, is undoubtedly problematic.

I recently put in some limits that should have prevented that long of an input. I also tested on a 90K character length note and couldn't reproduce this.

Can you confirm which version you're using?

from obsidian-smart-connections.

from obsidian-smart-connections.

That is the error that's intentionally thrown when the embeddings.json file is missing. It's intentional to halt processes that might otherwise overwrite data.

When that happens, you should see a button to "Create embeddings.json" in the right-hand sidebar. There is also a retry button in cases where the embeddings.json file should already exist.

from obsidian-smart-connections.

from obsidian-smart-connections.

Not a problem at all. It helps me solve this for others who might otherwise move on without reporting the issue.

If the view with the dialog options isn't visible, you can try the Open: View Smart Connections command or disable and re-enable the plug-in. It will take ~10 seconds after doing either of those before the prompt appears.

Thanks for your help!

from obsidian-smart-connections.

@Fusion were you able to make it work?

from obsidian-smart-connections.

Not sure what happened there (now v1.0.45):

Note the number of retries before I managed to unload the plugin.

from obsidian-smart-connections.

@Fusion with that many 429 retries, I think it's worth double checking your API key is working, as we discovered in this issue #13 (comment)

from obsidian-smart-connections.

@Fusion latest version backs off the API exponentially to prevent further 429s and implements skipping of previously failed files. These files can be manually retried from the settings but should allow the plugin to work with partial vault embeddings.

from obsidian-smart-connections.

I am getting a 429 error

Error: Request failed, status 429 at new t (app.js:1:710073) at Aw (app.js:1:710265) at app.js:1:710942 at app.js:1:235505 at Object.next (app.js:1:235610) at a (app.js:1:234349)

Can't seem to find a way around it

EDIT: Solved by upgrading to pay-as-you-go account, might be worth specifying in the doc for future users!

from obsidian-smart-connections.