brianpetro / obsidian-smart-connections Goto Github PK

Hello,

I have this error. I understand that is a limit about ChatGPT API. There's something we can do about the limit? Try to use FASS as an index and split the chunks?

Thanks!

So something like this:

1. Open a file, the plugin makes connections
2. Click on a "lock" icon to lock the results
3. Open another file, the results won't updated since they are locked

BTW, thank you so much for the blazing fast update for #45, now it works flawlessly.

you should hide the text of the api key field.

Below is an example

Great plugin!

My request is to only show the filename in the list, rather than the entire path. Perhaps show the path upon hover?

I'd like to save the embeddings file to a specific folder to avoid syncing it

Hi,

what I am seeing is 24 x "skipping block (mtime)"

followed by:

plugin:smart-connections:508 Error: Request failed, status 400
    at new t (app.js:1:710073)
    at Aw (app.js:1:710265)
    at app.js:1:710942
    at app.js:1:235505
    at Object.next (app.js:1:235610)
    at a (app.js:1:234349)
plugin:smart-connections:466 Uncaught (in promise) TypeError: Cannot read properties of null (reading 'usage')
    at SmartConnectionsPlugin.get_embeddings (plugin:smart-connections:466:51)
    at async Promise.all (index 0)
    at async SmartConnectionsPlugin.get_file_embeddings (plugin:smart-connections:454:5)
    at async Promise.all (index 0)
    at async SmartConnectionsPlugin.get_all_embeddings (plugin:smart-connections:201:9)
    at async SmartConnectionsPlugin.find_note_connections (plugin:smart-connections:628:7)
    at async SmartConnectionsPlugin.render_note_connections (plugin:smart-connections:564:21)
get_embeddings @ plugin:smart-connections:466
await in get_embeddings (async)
eval @ plugin:smart-connections:116
e.tryTrigger @ app.js:1
e.trigger @ app.js:1
t.trigger @ app.js:1
t.activeLeafEvents @ app.js:1
s @ app.js:1
l @ app.js:1

Adding a breakpoint in the catch block, this is the response:

{
    "error": {
        "status": 400,
        "headers": {
            "date": "Sun, 08 Jan 2023 05:53:01 GMT",
            "content-type": "application/json",
            "content-length": "294",
            "access-control-allow-origin": "*",
            "openai-organization": "user-<redacted>",
            "openai-processing-ms": "117",
            "openai-version": "2020-10-01",
            "strict-transport-security": "max-age=15724800; includeSubDomains",
            "x-request-id": "<redacted>"
        }
    }
}

Request:

{
    "usedParams": {
        "model": "text-embedding-ada-002",
        "input": "<some enormous escaped page content here>"
    },
    "reqParams": {
        "url": "https://api.openai.com/v1/embeddings",
        "method": "POST",
        "body": "{\"model\":\"text-embedding-ada-002\",\"input\":\"<same enormous page content as above>\"}",
        "headers": {
            "Content-Type": "application/json",
            "Authorization": "Bearer sk-<redacted>"
        }
    }
}

What if Smart Connections was connected to external platforms?

Imagine seeing the relevant content you've created outside of Obsidian from within your vault through Smart Connections.

Below is a screenshot showing this feature in action. I integrated my blog, so posts are included in my Smart Connections.

This would save a lot of time for someone like myself by helping me remember what I wrote in emails and published on my blog or elsewhere.

But I need your help!

I need help focusing on integrating Smart Connections with the platforms that will matter most.

Don't miss out on this opportunity to have your voice heard and make Smart Connections even smarter.

Respond here with the names of the platforms you want to see integrated into Smart Connections!

I'm loving it so far.

One thing I've noticed is that it still suggests connections even when they're already linked from the file.

In general I'm looking for items that are not already linked (similar to the non-AI functionality).

The "Previously failed files" list is very large (well over 100 files in a vault that contains almost 600 files). Excluding a few miscellaneous files, all of the failed files are in the same folder (Zettelkasten). These comprise almost the entirety of the Zettelkasten folder. All files are .md files.

I got the above result even after doing a Force Refresh. File is attached below.

GW-thinking-box ARCHIVE 1:21:23.zip

Probably similar to #32, #33, I'm currently using Windows 11. Luckily I don't have a lot of notes (~50) with average around 10000 characters (majority are Journals), so I only lost average around 200K tokens every time it re-indexing, LOL 😅

1. only one notification is necessary.
2. the notification should not appear if it has appeared with some time or based on some action.

Re: #13 (comment)

Absolute hog for CPU.
Please let the user change how often is the cycle for calculating smart connections

I see an error saying my platform is not supported. I'm on an iPad.

Getting this in the logs

I must confess to being an NLP novice and not fully understanding the concept of embeddings yet. But would it be possible to only trigger generation of embeddings on a per file basis when that file is opened?

Somewhat related to:

• #33
• #5 (comment)
• #32
• #31

Hi,

This is an amazing plugin!
I've been having a small problem in that my embeddings.json file remains empty, even after the connections have been made and are displayed in Obsidian. This means that the connections are remade each time I open up Obsidian.

I'm getting the following error, as well as a couple of failed embeddings, but I'm not sure this is related:

Error: net::ERR_FAILED
    at SimpleURLLoaderWrapper.<anonymous> (node:electron/js2c/browser_init:101:7169)
    at SimpleURLLoaderWrapper.emit (node:events:390:28)

The failed embeddings are being written to the failed-embeddings.txt. The embeddings.json file however contains only a pair of curly brackets. A force refresh makes a new embeddings file, with a time stamp, but this too remains empty.

Thanks!

This is such a powerful tool. I am using it to write a paragraph on a certain theme and then having it pick up similar text across my vault. Would it be possible to add the ability to save or embed this search as a new document or in a document? Are the results of the smart search ranked in some way? It would be useful to be able to set a limit on the number of records retrieved if that is the case?

It would be helpful to be able to see some context in the results pane. Perhaps a toggle to show a part of the related section? Obsidian's native search UI is a good example.

error msg: "Error getting embeddings for some-note-title.md"

I think this error is being generated before all the vault's notes are initially processed. Any ideas on how to fix this? I can't wait to see what this plugin can do for me.

Thanks for this plugin, but it is economically not feasible to use if it happens to use so many tokens.
Any suggestions?

Two parts:

1. Indicate link is clickable with a hand cursor or something.
2. Enable dragging a suggested note into the current note to add a link, or drag to a new pane/tab to open.

Would it be possible to have the option to store the embeddings in Pinecone?

Thanks for doing this. I love the idea to have OpenAI help look through my notes to find connections.

So, I'm a noob. I don't know why my notes are "excluded".

What is happening? haha
Regards

A small tweak which would improve readability. Once other file types start to be also parsed it should probably be included, but for md files I feel like it is a bit superfluous given the context.

Solution: Do not include the .md file extension at the end of the file name in the smart links file list

I posted an idea here: https://twitter.com/Sarah_A_Bentley/status/1606338249529561088?s=20&t=XTp7SsyB1_pYt9Hy3khiGg
based off of this: https://twitter.com/rileytomasek/status/1603854647575384067?s=20&t=XTp7SsyB1_pYt9Hy3khiGg

Here's what I said: "I want to be able to GPT3-query my own notes and receive answers that are fine-tuned to my style and voice. Semantically search my content for relevant passages and feed them in with the prompt. That's the first step."

Looks like you've gotten the first part. Is the rest doable?

Hi, Unfortunately this isn't working for my vault. I'm using 1.0.48

I first tried in on the openai's free plan, but have upgraded to a paid plan to see if that helps, but it doesn't.
The API key is working, as verified in your plugin, and because I see this in openai's usage logs:

13:20 Local time: 19 Jan 2023 at 13:20 text-embedding-ada-002-v2, 54 requests 3,183 prompt + 0 completion = 3,183 tokens 13:25 Local time: 19 Jan 2023 at 13:25 text-embedding-ada-002-v2, 16 requests 3,415 prompt + 0 completion = 3,415 tokens

However, there are no smart connections being created in my obsidian vault, and every single console entry reports status 429.

Here is a typical example:

failed embedding: 5 Daily Notes/2022-11-14-Mon.md plugin:smart-connections:717 undefined plugin:smart-connections:721 Error: Request failed, status 429 at new t (app.js:1:710073) at Aw (app.js:1:710265) at app.js:1:710942 at app.js:1:235505 at Object.next (app.js:1:235610) at a (app.js:1:234349)

Do you have any ideas? How can I help fault trace this?

Love the plugin so far. I was wondering, since the plugin is already generating embeds for the entire vault, could it be possible to add a chat_gpt like query tool, with the goal of synthesising an answer from the vault's embedding along with the preexisting training data?

Loving the plugin so far!

It would be great if, when I click on a result in the side pane, I'm taken to the relevant header section in the related page and the header is highlighted. Right now it just starts at the top of the page.

I am getting a flock of popup notices indicating failed file skip, these do not disappear but persist.

It uses a million tokens everytime it connects to OpenAI.... can you do something about this?

Either reduce the token request payload for updates? or give us a control for setting it to update once a month or so... ?

Thanks

I don't think your latest plugin updates are getting picked up by BRAT for some reason. Am I doing something wrong?

It would be very helpful to have a setting to exclude various folders. The excluded file option doesn't really satisfy my needs.

Hello , what's could be the problem if the plugin is not accepting any OpenAI APIs I add on the settings?

Does OpenAI provide some sort of score for 'matching goodness of fit' of the results? If so, could we get an option that allows us to limit results e.g. to options with a score better than 0.60 for example (if scoring from 0.00 to 1.00). Alternatively, maybe we could limit the results to the top N most relevant connections?

Brian,

After installing via BRAT and adding my OpenAI token several days ago, I'm still getting:

Making smart connections...

I have only 4K files.

Is there a log I could check?

Thanks,

Bill Petro
billpetro.com

Can these embeddings be used for finetuning?
If so, what would that workflow look like with GPT-3?

Can I send these embeddings as prompts instead of large text?

Hello, this plugin is amazing but it would be great if there was a way to completely block or hide headers from showing up in the Connections pane. I know you can exclude them individually currently, but I would like to do it for all headers. Thanks!

When the embeddings file is tool large, it is problematic to sync it with github, which has a size limit of 100 MB. And it warns, if the file exceeds 50 MB. My embeddings file is 65MB. Maybe it would be possible to split large files into a smaller ones?

Hello,
I have obsidian on both windows and mac and I use obsidian sync.
I ran smart connections correctly 48 hours ago on windows, but it was not working on mac.
Then when I went back to windows, it told me that there were failed files.
Now it tells me that my API key is not working correctly when I do the API key test from Obsidian.
To have the file on the mac properly addressed, I tried to copy my .smart-connections file from windows to my macbook.
Also, in the sync settings, do I have to sync all other file types for it to work correctly (the option was disabled in my case)?
Also, in the Api key test it tells me that: Api key is not working as expected.
Knowing that there was an update on 07.02.23, I wonder if there is not an error somewhere. Before the update in any case, everything was working correctly on my windows computer.
I restarted a sync to see by deleting all the files and starting again on new (restarting the embedding).
Oh and thank you for this great Plugin, I can't wait to see the rest.

The plugin was not getting responses from the API even though the free account was supposed to allow 60 req/min. As soon as I upgraded to paid, the responses came back and everything works fine now.
Thanks for the awesome plugin!

Plugin has been embedding for a long time. Not sure if it's doing something productive or running into an issue. I'm seeing an error 429 frequently:

Obsidian is also just generally laggy while this is going on.

I'd like to explore the smart connections for awhile without having the plugin fetch new smart connections with the files mtime. It would be nice to have a setting to turn off fetching (I spent $7 USD total for 10k files by the way!)

I installed it manually because it won't show up in the community plugins section. Added the API key. But even after hours I just see "Making smart connections..." in the smart connections pane. I must be doing something wrong. Any ideas?

I think this might partially be my large embeddings file, but on startup obsidian will lock up for a good 5ish minutes, and use a large amount of memory:

here are the filesizes:

Is there a way to customize the prompts sent to the API? It would be nice to be able to give it some specific direction and info like the date and time or personality direction and a name and all that.

Hi, your plugin has amazing potential to unlock lots of new info that can be added to your vault, ultimately also be linked to other vaults.. one thing that triggered me to write this comment that Users should be aware that (large parts of) the content of their vault will be shared with OpenAI. One of the great benefits of having your own vault is that you can build something only for yourself.. So my advice would be that you make clear in your plugin summary what parts of your vault are shared with others and that you can shield off parts of your vault that will never shared..

See the attached two images. The first one ("benefits" in title) has multiple smart connections. Look at the top one ("practicing gratitude...") The text of this note is the second image, labeled "practicing".

Clicking this link does not cause the "practicing" note to appear.

Here are some things I find of interest.

1. The "gratitude" note has a tag, "#c001", at the bottoom
2. The top smart connection in "benefits" has text in two sizes; the larger size begins with ">>c001"
3. Could the "#" changing to a ">>" be relevant to the fact that clicking the smart connection link does not result in the "practicing" being displayed?
4. Now that I think about it, should the ">>c001 key quote re. gratitude...circuits" be showing up? After all, it's part of the text of the note, not its title.

Thanks for your help.

PS: In the next few days, I'll be sending you an email with some ideas. My name is Gregg Williams.

Process keeps stalling with the below errors:

Sorry for the noobie question but i ve not been able to use this plugin (which could be SUPER interesting)

1. I ve created a NEW vault
2. added a long note and 50+ kindle
3. set up the API (and tested them) - i am on a PAID APi account
4. waited an hour
5. asked questions with "I"
6. got the ... dot
   7 test the writing file and got an error: ENOENT - not such a file open .... /path/.smart-connections/embedding-test.json

I searched how to solve it with avail.
can you please help?

Hi!

I would appreciate it if you could utilize titles provided by https://github.com/snezhig/obsidian-front-matter-title instead of the default file names. The API for integration can be found at https://github.com/Snezhig/front-matter-plguin-api-provider.

:)