dragon-okinawa / dragon Goto Github PK
View Code? Open in Web Editor NEWDynamic Retrieval Augmented Generation for Optimized Nimble
Home Page: https://dragon.okinawa
License: MIT License
dragon's People
Contributors
![renovate[bot] avatar](https://avatars.githubusercontent.com/in/2740?v=4 "renovate[bot]")
Stargazers
Watchers
Forkers
eltocieardragon's Issues
Returning sources from RAG response
Since langchain4j 0.31.0 we can retrieve sources :
Ingestor : Notion.so
dRAGon Frontend GUI
- Ability to use dRAGon with a web user interface instead of raw APIs calls
Ingestor : Wiki.js
Apache Tika : common / shared metadata extraction
Granary (Farm Content Retriever) : OpenAPI Client
- Ability to retrieve content from an API about a user query
- Similar to #79 Harvester but no need to let LLM to choose the Tool. The content is queried as same as we would query a Silo
- Don't forget to add #89 Query Routing
- Ability to use an OpenAPI Spec file (or URL) : https://github.com/swagger-api/swagger-parser
RaaG Chat History Strategies
- MessageWindowChatMemory
- TokenWindowChatMemory
Supports of pgvector Embedding Store
- Ability to add settings for using a pgvector database
- Ability to use env var to setup user / password
Adding support for audio, video
Health API Endpoint
- Use job states and results to reflect the true health response
API Authentication
- Backend endpoints should use login / password / jwt
- Other API endpoints should use tokens
RaaG Retriever Tool : Show documents location
- Ability to ask where documents are stored
JobRunr recurring job is sometimes not created / updated after a Silo PATCH
After some time of activity if we create / update a Silo the recurring job (JobRunr) is not created / updated at all.
Looks like the "update" event is not working anymore.
Logs API Endpoint
- Ability to retrieve SLF4J logs from API
- Ability to filter by level
Granary (Farm Content Retriever) : SQL Database
Since langchain4j 0.31.0 we can use a new retriever for RAG :
- https://github.com/langchain4j/langchain4j/releases/tag/0.31.0
- langchain4j/langchain4j#1056
- langchain4j/langchain4j-examples@e980910
- Like #15 (Web Search Engine)
Farm Re-Ranking
Ability to Re-Rank results issued from Content Retriever : https://github.com/langchain4j/langchain4j-examples/blob/17e4f94bc94604ce562878681cc5c676809c3ddc/rag-examples/src/main/java/_3_advanced/_03_Advanced_RAG_with_ReRanking_Example.java#L85
[BUG] LibreChat don't use RaaG to generate conversation title
Describe the bug
When using LibreChat integration title of the conversation is not generated
Log and Stack trace
No error on both LibreChat and dRAGon
To Reproduce
- Configure LibreChat to use dRAGon
- Chat with the RaaG
- No title is generated (summary of the query)
Expected behavior
A summary of the query put in the title of the conversation
Please complete the following versions information:
------------------------------------------------------------
Gradle 8.7
------------------------------------------------------------
Build time: 2024-03-22 15:52:46 UTC
Revision: 650af14d7653aa949fce5e886e685efc9cf97c10
Kotlin: 1.9.22
Groovy: 3.0.17
Ant: Apache Ant(TM) version 1.10.13 compiled on January 4 2023
JVM: 17.0.11 (Microsoft 17.0.11+9-LTS)
OS: Linux 6.6.31-linuxkit aarch64
NPM: 10.5.2
Node: v20.13.1Additional context
/
Granary (Farm Content Retriever) : Web Search Engine
Since langchain4j 0.31.0 we can use a new retriever for RAG :
- https://github.com/langchain4j/langchain4j/releases/tag/0.31.0
- langchain4j/langchain4j#642
- langchain4j/langchain4j-examples@588f19f
- https://github.com/langchain4j/langchain4j-examples/blob/main/rag-examples/src/main/java/_3_advanced/_08_Advanced_RAG_Web_Search_Example.java
- Check ability to launch a search request when user put some keywords on query (Language Query Router)
- Where to put this Content Retriever? Farm seems a better location, like #79
- Does this content retriever is a built-in Harvester?
- or a "Granary" linked to a Farm?
classDiagram
FarmEntity : +UUID uuid
FarmEntity : +String name
FarmEntity : +String raagIdentifier
FarmEntity : +[UUID] silos
FarmEntity : +[UUID] granaries
GranaryEntity : +UUID uuid
GranaryEntity : +String name
GranaryEntity : +[String] granaryConfiguration
FarmEntity .. GranaryEntity
Does "Granary" is an entity? Or just a static array of built-in retrievers?
RaaG Query Rewriting
RaaG / n8n : Ollama Server API Support
In order to support n8n Integration through "Ollama Server API like" we need to support some Ollama API endpoints :
URL Ingestor
- Ability to ingest one or more URLs
- Extract texts to RAG
Dependencies Upgrade Dashboard
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
Rate-Limited
These updates are currently rate-limited. Click on a checkbox below to force their creation now.
- chore(deps): update docker/build-push-action action to v6
- chore(deps): update mcr.microsoft.com/openjdk/jdk docker tag to v21
- fix(deps): update dependency chart.js to v4
- fix(deps): update dependency primeicons to v7
- fix(deps): update dependency primevue to v4
- 🔐 Create all rate-limited PRs at once 🔐
Open
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
- chore(deps): update dependency typescript to ~5.5.0
- chore(deps): update plugin org.openapi.generator to v7.7.0
- fix(deps): update dependency chart.js to v3.9.1
- fix(deps): update dependency primevue to v3.53.0
- fix(deps): update docusaurus monorepo to v3.4.0 (
@docusaurus/core,@docusaurus/module-type-aliases,@docusaurus/preset-classic,@docusaurus/theme-mermaid,@docusaurus/tsconfig,@docusaurus/types) - chore(deps): update dependency @vitejs/plugin-vue to v5
- chore(deps): update dependency @vue/eslint-config-prettier to v9
- chore(deps): update dependency eslint to v9
- chore(deps): update dependency prettier to v3
- chore(deps): update dependency vite to v5
- Click on this checkbox to rebase all open PRs at once
Detected dependencies
dockerfile
Dockerfile
mcr.microsoft.com/openjdk/jdk 17-ubuntu
github-actions
.github/workflows/build.yml
actions/checkout v4actions/setup-java v4actions/setup-node v4gradle/actions v3codecov/codecov-action v4.5.0docker/setup-buildx-action v3docker/login-action v3docker/metadata-action v5docker/build-push-action v5
gradle
settings.gradle
build.gradle
pl.allegro.tech.build.axion-release 1.18.2backend/gradle.properties
backend/build.gradle
org.springframework.boot 3.3.2io.spring.dependency-management 1.1.6org.springdoc.openapi-gradle-plugin 1.9.0org.springframework.boot:spring-boot-starter-thymeleaf 3.3.2org.springdoc:springdoc-openapi-starter-webmvc-ui 2.6.0dev.langchain4j:langchain4j 0.33.0dev.langchain4j:langchain4j-core 0.33.0dev.langchain4j:langchain4j-embeddings 0.33.0dev.langchain4j:langchain4j-pgvector 0.33.0dev.langchain4j:langchain4j-easy-rag 0.33.0dev.langchain4j:langchain4j-open-ai 0.33.0dev.langchain4j:langchain4j-mistral-ai 0.33.0org.dizitart:nitrite-bom 4.3.0net.sourceforge.argparse4j:argparse4j 0.9.0org.jobrunr:jobrunr-spring-boot-3-starter 7.2.2org.awaitility:awaitility 4.2.1org.projectlombok:lombok 1.18.34org.projectlombok:lombok 1.18.34org.projectlombok:lombok 1.18.34org.projectlombok:lombok 1.18.34checkstyle 10.17.0jacoco 0.8.12docs/build.gradle
com.github.node-gradle.node 7.0.2org.openapi.generator 7.5.0frontend/build.gradle
com.github.node-gradle.node 7.0.2
npm
docs/package.json
@docusaurus/core 3.3.2@docusaurus/preset-classic 3.3.2@docusaurus/theme-mermaid 3.3.2@mdx-js/react ^3.0.0clsx ^2.0.0docusaurus-lunr-search ^3.4.0prism-react-renderer ^2.3.0react ^18.0.0react-dom ^18.0.0@docusaurus/module-type-aliases 3.3.2@docusaurus/tsconfig 3.3.2@docusaurus/types 3.3.2typescript ~5.2.2node >=18.0frontend/package.json
chart.js 3.3.2primeflex ^3.3.1primeicons ^6.0.1primevue 3.49.1vue ^3.2.41vue-i18n ^9.13.1vue-router ^4.1.5@babel/core ^7.12.16@babel/eslint-parser ^7.12.16@rushstack/eslint-patch ^1.1.4@vitejs/plugin-vue ^4.1.0@vue/eslint-config-prettier ^7.0.0eslint ^8.22.0eslint-plugin-vue ^9.3.0prettier ^2.7.1sass ^1.55.0vite ^4.2.1
- Check this box to trigger a request for Renovate to run again on this repository
Documents : Save them inside database
- If document is not changed don’t embedded it again
- Documents not indexed in the last ingestion : delete them and their fragments
Log queries and results
- Ability to log queries from user
- Ability to log responses from LLM
- Log documents (stats) used and sent to the LLM
- LangChain4j :
Farm : Smart Silo Routing
- FarmEntity : add setting boolean => smartRouting
- If smartRouting is not enabled : call each Silo + Granary
- If smartRouting is enabled : ask LLM to choose one or more Silo + Granary
- Use FallbackStrategy => retrieve on all? Or none?
- SiloEntity + GranaryEntity : add String field "description" => mandatory or not?
- Description will be used in order to help the LLM to choose the best ones
- Use langchain4j "LanguageModelQueryRouter" => choose one or more Silos / Granaries to retrieve
Harvester (Farm Tool OpenAPI Client)
Harvesters are API Clients as a Tool for LLMs. If a LLM needs to call a Tool to reply the user then a Harvester is used on a Farm.
- Ability to use an OpenAPI Spec file (or URL) : https://github.com/swagger-api/swagger-parser
- Build a langchain4j tool : https://docs.langchain4j.dev/tutorials/tools
Like:
classDiagram
FarmEntity : +UUID uuid
FarmEntity : +String name
FarmEntity : +String raagIdentifier
FarmEntity : +[UUID] silos
FarmEntity : +[UUID] harvesters
Harvester Entity : +UUID uuid
Harvester Entity : +String name
Harvester Entity : +String openApiSpecs
FarmEntity .. HarvesterEntity
Dependency Dashboard
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
Rate-Limited
These updates are currently rate-limited. Click on a checkbox below to force their creation now.
- fix(deps): update dependency @vueuse/core to v10.11.1
- chore(deps): update dependency eslint to v9.9.0
- fix(deps): update dependency tailwind-merge to v2.5.0
- 🔐 Create all rate-limited PRs at once 🔐
Open
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
- chore(deps): update dependency @iconify/json to v2.2.235
- chore(deps): update dependency npm-check-updates to v17.0.6
- chore(deps): update plugin pl.allegro.tech.build.axion-release to v1.18.4
- fix(deps): update dependency org.awaitility:awaitility to v4.2.2
- fix(deps): update dependency org.jobrunr:jobrunr-spring-boot-3-starter to v7.2.3
- fix(deps): update dependency vue to v3.4.37
- chore(deps): update dependency @types/node to v22.2.0
- chore(deps): update dependency tsx to v4.17.0
- chore(deps): update docker/build-push-action action to v6
- chore(deps): update gradle/actions action to v4
- chore(deps): update mcr.microsoft.com/openjdk/jdk docker tag to v21
- Click on this checkbox to rebase all open PRs at once
Detected dependencies
dockerfile
Dockerfile
mcr.microsoft.com/openjdk/jdk 17-ubuntu
github-actions
.github/workflows/build.yml
actions/checkout v4actions/setup-java v4actions/setup-node v4pnpm/action-setup v4gradle/actions v3codecov/codecov-action v4.5.0docker/setup-buildx-action v3docker/login-action v3docker/metadata-action v5docker/build-push-action v5
gradle
gradle.properties
settings.gradle
build.gradle
pl.allegro.tech.build.axion-release 1.18.2backend/gradle.properties
backend/build.gradle
org.springframework.boot 3.3.2io.spring.dependency-management 1.1.6org.springdoc.openapi-gradle-plugin 1.9.0org.springframework.boot:spring-boot-starter-thymeleaf 3.3.2org.springdoc:springdoc-openapi-starter-webmvc-ui 2.6.0dev.langchain4j:langchain4j 0.33.0dev.langchain4j:langchain4j-core 0.33.0dev.langchain4j:langchain4j-embeddings 0.33.0dev.langchain4j:langchain4j-pgvector 0.33.0dev.langchain4j:langchain4j-easy-rag 0.33.0dev.langchain4j:langchain4j-open-ai 0.33.0dev.langchain4j:langchain4j-mistral-ai 0.33.0org.dizitart:nitrite-bom 4.3.0net.sourceforge.argparse4j:argparse4j 0.9.0org.jobrunr:jobrunr-spring-boot-3-starter 7.2.2me.paulschwarz:spring-dotenv 4.0.0org.awaitility:awaitility 4.2.1org.projectlombok:lombok 1.18.34org.projectlombok:lombok 1.18.34org.projectlombok:lombok 1.18.34org.projectlombok:lombok 1.18.34checkstyle 10.17.0jacoco 0.8.12frontend/gradle.properties
frontend/build.gradle
com.github.node-gradle.node 7.0.2
npm
frontend/package.json
@better-scroll/core 2.5.1@iconify/vue 4.1.2@vueuse/core 10.11.0clipboard 2.0.11dayjs 1.11.12echarts 5.5.1lodash-es 4.17.21naive-ui 2.39.0nprogress 0.2.0pinia 2.2.1tailwind-merge 2.4.0vue 3.4.36vue-draggable-plus 0.5.3vue-i18n 9.13.1vue-router 4.4.3@elegant-router/vue 0.3.8@iconify/json 2.2.234@soybeanjs/eslint-config 1.4.0@types/lodash-es 4.17.12@types/node 22.1.0@types/nprogress 0.2.3@unocss/eslint-config 0.61.9@unocss/preset-icons 0.61.9@unocss/preset-uno 0.61.9@unocss/transformer-directives 0.61.9@unocss/transformer-variant-group 0.61.9@unocss/vite 0.61.9@vitejs/plugin-vue 5.1.2@vitejs/plugin-vue-jsx 4.0.0eslint 9.8.0eslint-plugin-vue 9.27.0lint-staged 15.2.8sass 1.77.8tsx 4.16.5typescript 5.5.4unplugin-icons 0.19.2unplugin-vue-components 0.27.3vite 5.4.0vite-plugin-progress 0.0.7vite-plugin-svg-icons 2.0.1vite-plugin-vue-devtools 7.3.7vue-eslint-parser 9.4.3vue-tsc 2.0.29node >=18.12.0pnpm >=8.7.0frontend/packages/axios/package.json
axios 1.7.3axios-retry 4.5.0qs 6.13.0@types/qs 6.9.15frontend/packages/color/package.json
colord 2.9.3frontend/packages/hooks/package.json
frontend/packages/materials/package.json
simplebar-vue 2.3.5typed-css-modules 0.9.1frontend/packages/ofetch/package.json
ofetch 1.3.4frontend/packages/scripts/package.json
@soybeanjs/changelog 0.3.24bumpp 9.4.2c12 1.11.1cac 6.7.14consola 3.2.3enquirer 2.4.1execa 9.3.0kolorist 1.8.0npm-check-updates 17.0.3rimraf 6.0.1frontend/packages/uno-preset/package.json
frontend/packages/utils/package.json
colord 2.9.3crypto-js 4.2.0localforage 1.10.0nanoid 5.0.7@types/crypto-js 4.2.2
- Check this box to trigger a request for Renovate to run again on this repository
"None" Ingestor
Ability to don't ingest documents by dRAGon.
Useful for example if we ingest documents in a pgvector store outside dRAGon but we would like to query it through RaaG.
Temperature / Max tokens / User ID
Ingestor Scheduling Improvements
- Create an abstract class AbstractIngestorSettings
- Move ingestoreSchedule (SiloEntity) to this new class
- Add scheduleTimezone to the settings class
-
Can we have a way to launch manually a job only?=> PT87600H
Dashboard
- Show numbers
- Silos
- Farms
- Documents
- Show stats
- Number of RAG / RaaG queries
Ingestor : SharePoint
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.