Comments (2)
@bbecausereasonss, thanks for sharing this! With all that I'm building, it's hard to find the time to experiment like this myself. Great insights!
ChatGPT is still the most comprehensive at analysis, especially for takeaways, bullets, and subpoints in long articles.
Gemini is VERY good at 'human' writing. Followed by Claude and by far the worst is ChatGPT. For instance Gemini will pass most AI detectors, while Claude Opus may 40% of the time and ChatGPT 0% of the time. Even when you prompt ChatGPT to write more human by explaining the dynamics, it still fails.
Claude is VERY good at creativity
Gemini is VERY good at synthesis
This is very good to know! Is there anything you feel comfortable sharing about your use-case? It might be helpful to have a little more context.
Inference speed is WAY higher in OpenAI models, both Claude/Gemini take a long time to respond (seem to take much longer to parse the embedding data)
I think it would be useful if I added a tokens/second calculation because, currently, it's hard to compare based on the differences between the APIs. Primarily, OpenAI has a far superior streaming implementation, and Anthropic/Claude doesn't have streaming when using the API in an environment like Obsidian (hopefully, this will change soon!). The Google streaming, to me, seems like they're faking it since the stream is a sentence or more per chunk, while OpenAI is more like a word per chunk, and Google's comes seemingly all at once at the end ~80% of the request time.
The context retrieval could take longer if it retrieves more than is being retrieved for the OpenAI models. But this should be almost negligible. The other difference is the intermediate HyDE completion. So a slower model will feel even slower if it's waiting on the HyDE completion. But, the streaming probably accounts for most of the difference.
Something I thought of that would be absolutely next level is the ability to chain multiple models together, almost like a broad mixture of experts.
This is a really interesting idea. I haven't put much thought into it, but I think it would be a great future addition to allow simultaneous responses from multiple models.
There would be a way to use the same embeddings, but to ping pong between the 3 models based on the prompt. Of course there are many 'agent' frameworks out there that could make this even more interesting.
While not really an agent framework, I've been looking at the "Cannoli" Obsidian plugin and think it would be great to integrate with Smart Connections. I messaged the creator just the other day to ask about enabling Smart Connections embeddings to be used in Cannoli. Besides that, to build the workflow you described, it would just need to be able to access the various models, which could be accomplished through a similar Smart Connections integration.
I could develop this idea further, if interested but each model has a very unique strength and weakness.
If you have any more thoughts, please do share.
Aside: The in-chat, context conditioning is working much better than before. The models seem to query all the notes, and refer to the chat window simultaneously, before this was impossible and the model would get tripped up and confused.
😊 I'm happy you noticed this. The way the Smart Chat utilizes chat history and the available context window have both been improved in v2.1
.
I'm also still working towards improving this more. v2.1
is a re-write, but implements essentially the same features. One way I plan on further improving the retrieval is by making the retrieval logic more accessible. Currently, it's just a "bunch of code." But, this code can be modularized so that the "retrieval strategy" looks more like a "list of easy-to-understand functions" rather than the "bunch of code." Eventually, this should make it easy to change the retrieval strategy in the chat and make contributions of new and novel strategies as easy as re-arranging the order of these functions.
Also, looking ahead, I expect to implement actions in the Smart Chat v2.2
similar to those provided to ChatGPT via the Smart Connect software. I like the concept of the AI utilizing actions (aka "tools" or "function calls") because it effectively implements agent-type abilities directly in the chat.
💡 Based on your idea about switching models, I'm making a note (...) to add an action that triggers switching models. This way, implementing the abc
workflow you described above could be as easy as creating a system prompt, which would describe when to switch models. Thanks for the idea!
That reminds me of a new feature I haven't previously mentioned: you can now use @
in the chat input to select a system prompt. In the' v2.1' settings, you can specify a folder for system prompts. This is a very early/experimental feature, but it should insert the specified system prompt before the user message. And due to the aforementioned lack of time, I've hardly had time to experiment with it. So I'd be interested to know if/how you manage to utilize it.
It was great to read these insights; I'm sure others will find them valuable, too. Thanks for your continued contributions to the community!
🌴
from obsidian-smart-connections.
Just wanted to update that after more testing I find myself using Claude 60% of the time, switching the 3 model sizes depending on how much power I need, ChatGPT 30% of the time to fill in gaps and be reliable and Gemini 10% (I know Gemini is powerful but for what ever reason it keeps giving me very short and boring replies and analysis on my notes)
from obsidian-smart-connections.
Related Issues (20)
- Issues with retrieval in Smart Chat HOT 12
- Feature Request: Enhance the functionality of specifying API parameters HOT 1
- smart chat , top level folder exclusion HOT 4
- Always getting the context_length_exceeded error. Is transcript parsing possible? HOT 1
- Stalls at 2090 files out of 2324 Files While Initial Indexing HOT 1
- Smart Connect - Notes on ChatGPT not being able to read my notes HOT 2
- Minor typo: `if a local model if failing` => `if a local model is failing`
- Your chats are currently using a community account..." HOT 6
- In beta "/" does not bring up a modal-dialogue like it used to. HOT 2
- Ajuda com o Plugin "Smart Connections" no Obsidian
- seems to have stopped working after update HOT 3
- Smart Chats HOT 2
- Issues with Ollama local model output HOT 3
- Chat History should allow a custom sub-folder HOT 1
- Embedding hangs HOT 5
- Unrelated content also has a high degree of similarity, which local model is more appropriate?
- Smart chat stops working after a few questions, a restart of Obsidian is required HOT 1
- Unexpected end of JSON input while trying to connect to local llama3.1 ollama server HOT 3
- [Feature Request] Only use knowledge from selected notes
- Max Token Error
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from obsidian-smart-connections.