Giter VIP home page Giter VIP logo

lobehub / chat-plugin-web-crawler Goto Github PK

View Code? Open in Web Editor NEW
47.0 2.0 16.0 184 KB

🧩 / 🕸 WebsiteCrawler - This plugin automatically crawls the main content of a specified URL webpage and uses it as context input.

Home Page: http://web-crawler.chat-plugin.lobehub.com

License: MIT License

JavaScript 0.06% Shell 0.03% TypeScript 0.63% HTML 99.29%
lobe-chat-plugin function-calling lobe-chat chatgpt openai ai crawler

chat-plugin-web-crawler's Introduction

Web Crawler
LobeChat Plugin

This plugin automatically crawls the main content of a specified URL webpage and uses it as context input


Changelog · Report Bug · Request Feature

Table of contents

TOC

🤯 Usage

This is a function calling plugin for Lobe Chat, you can install it in plugin setting page.

📦 Plugin Ecosystem

Plugins provide a means to extend the Function Calling capabilities of LobeChat. They can be used to introduce new function calls and even new ways to render message results. If you are interested in plugin development, please refer to our 📘 Plugin Development Guide in the Wiki.

  • lobe-chat-plugins: This is the plugin index for LobeChat. It accesses index.json from this repository to display a list of available plugins for LobeChat to the user.
  • chat-plugin-template: This is the plugin template for LobeChat plugin development.
  • @lobehub/chat-plugin-sdk: The LobeChat Plugin SDK assists you in creating exceptional chat plugins for Lobe Chat.
  • @lobehub/chat-plugins-gateway: The LobeChat Plugins Gateway is a backend service that provides a gateway for LobeChat plugins. We deploy this service using Vercel. The primary API POST /api/v1/runner is deployed as an Edge Function.

⌨️ Local Development

You can use Github Codespaces for online development:

Or clone it for local development:

$ git clone https://github.com/lobehub/chat-plugin-web-crawler.git
$ cd chat-plugin-web-crawler
$ bun install
$ bun dev

🤝 Contributing

Contributions of all types are more than welcome, if you are interested in contributing code, feel free to check out our GitHub Issues to get stuck in to show us what you’re made of.

🔗 Links

  • 🤖 Lobe Chat - An open-source, extensible (Function Calling), high-performance chatbot framework. It supports one-click free deployment of your private ChatGPT/LLM web application.
  • 🧩 / 🏪 Plugin Index - This is the plugin index for LobeChat. It accesses index.json from this repository to display a list of available plugins for Function Calling to the user.


📝 License

Copyright © 2023 LobeHub.
This project is MIT licensed.

chat-plugin-web-crawler's People

Contributors

arvinxx avatar canisminor1990 avatar lobehubbot avatar semantic-release-bot avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

chat-plugin-web-crawler's Issues

[Bug] 在使用gemini1.5 pro的情况下无法使用

💻 系统环境 | Operating System

Windows

🌐 浏览器 | Browser

Chrome

🐛 问题描述 | Bug Description

在使用gemini1.5 pro的情况下无法使用

🚦 期望结果 | Expected Behavior

No response

📷 复现步骤 | Recurrence Steps

No response

📝 补充信息 | Additional Information

No response

[Request] 支持非 vercel , 非 BROWSERLESS 的实现方式

🥰 需求描述 | Feature Description

当前商店里提供的插件, 非常好用, 可以解决很多问题, 但是有些时候, 需要抓非公开的, 内网里的文件, 此时本插件就派不上用场, 希望能提供方式, 把插件部署至内网, 让他可以抓取内网的内容.

另外实现方式中, 是调用了 browserless 的 token, 这样肯定是能适配很多的 SPA 应用, 但同时也让部署变得复杂, 希望能提供不使用 browserless 的实现.

🧐 解决方案 | Proposed Solution

提供 docker 等通用部署方案, 提供非 broserless 的实现

📝 补充信息 | Additional Information

我自己尝试了下自己做一个, LeoQuote/lobe-webcrawer-self-host#1 但老是报请求失败.

本地开发的时候, 也是非要先过服务端的网关, 非常的不方便, 难道我必须要 lobe 和插件都在本地运行才可正常进行调试?

只消耗token,没有任何返回[Bug]

💻 系统环境 | Operating System

macOS

🌐 浏览器 | Browser

Chrome

🐛 问题描述 | Bug Description

调用插件,然后只是消耗大量的token,没有任何返回信息,或者无效返回信息

🚦 期望结果 | Expected Behavior

No response

📷 复现步骤 | Recurrence Steps

提问“搜索eccDNA的最新发现”,调用gpt-4-32k模型,输入28981提示词,消耗3$额度,返回为“无法提供“。
提问”搜索GeminiPro的最新消息“,调用gpt-4-1106-preview, 输入29052和29498提示词,消耗2$额度,返回为空

📝 补充信息 | Additional Information

No response

[Bug] 页面内容太长会报错

💻 系统环境 | Operating System

macOS

🌐 浏览器 | Browser

Chrome

🐛 问题描述 | Bug Description

希望可以根据当前剩余token来限制读取内容的长度

🚦 期望结果 | Expected Behavior

No response

📷 复现步骤 | Recurrence Steps

image

📝 补充信息 | Additional Information

No response

[Bug] Does not work with Dolphin-Mixtral

💻 系统环境 | Operating System

Windows

🌐 浏览器 | Browser

Chrome

🐛 问题描述 | Bug Description

With dolphin-mixtral, I contend that it's not able to use the plugin and crawl a web page.

🚦 期望结果 | Expected Behavior

No response

📷 复现步骤 | Recurrence Steps

No response

📝 补充信息 | Additional Information

No response

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.