skywalkerdarren / chatweb Goto Github PK

ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key points.

Python 89.19% Jupyter Notebook 9.74% Dockerfile 0.98% Shell 0.10%

chatgpt embedding gpt-35-turbo news-extractor newspaper openai pgvector postgresql vector-database faiss

chatweb's Introduction

Hi there 👋

🚀 I‘m FULL STACK programmer
🔭 I’m currently working on Android and Flutter and Backend
🌱 I’m currently learning Flutter and GO

chatweb's People

Contributors

Stargazers

Watchers

Forkers

quorafind xieren58 qiaoshouqing xiaojay cxapython banduoba yummyship wrhsd1 jiacheng135 xianyucoder yifree cced3000 wodole lyhiving weykon ai-jie01 itiiss gaoxiaojun cfwvip shinny-hongyan ylrepertory enockipp glaceage colin-coder aoxiang594 xinggonglie dtdly92 westwind027 wzl-bxg licesun tianyang1027 baifengbai wangchen59 iamsail chenlip chatechat jhui008 ec0000 taijichadao tiwentichat bearnlee mason0510 panzhi1 zhaosuqi showjim sheng-jie mayimchen han111 sotowang rivien2010 itsharex arryboom stephenyang1989 xiaoxieshop qixiaobo zalan159 fcgca zhaozhongke marscube ard-skelling iabroad cheniac leegang mkchens 450083aa stampkiller alihongbo dl-link dgo2dance quduoduo zhongpei aicodehunt bob532 tomtomzhang1970 ouyoung protonlight llj090801 martianxhc millsoft shad0wperson anvychow jinghai w1r4 ganquanemail sc-glong running-chen911 richiesh litanlitudan learning-group1 universeroc henryhesz jianpengsu2 laihear menonpg hantop jfontestad wuyx songyongkang feifei05 lihan-hub

chatweb's Issues

I installed it with docker, but it ran wrong.

This is the wrong details：

INFO:     :35837 - "POST /crawler_url HTTP/1.1" 500 Internal Server Error
chatweb-chatweb-1  | ERROR:    Exception in ASGI application
chatweb-chatweb-1  | Traceback (most recent call last):
chatweb-chatweb-1  |   File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 435, in run_asgi
chatweb-chatweb-1  |     result = await app(  # type: ignore[func-returns-value]
chatweb-chatweb-1  |   File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
chatweb-chatweb-1  |     return await self.app(scope, receive, send)
chatweb-chatweb-1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 276, in __call__
chatweb-chatweb-1  |     await super().__call__(scope, receive, send)
chatweb-chatweb-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 122, in __call__
chatweb-chatweb-1  |     await self.middleware_stack(scope, receive, send)
chatweb-chatweb-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 184, in __call__
chatweb-chatweb-1  |     raise exc
chatweb-chatweb-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 162, in __call__
chatweb-chatweb-1  |     await self.app(scope, receive, _send)
chatweb-chatweb-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
chatweb-chatweb-1  |     raise exc
chatweb-chatweb-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
chatweb-chatweb-1  |     await self.app(scope, receive, sender)
chatweb-chatweb-1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
chatweb-chatweb-1  |     raise e
chatweb-chatweb-1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
chatweb-chatweb-1  |     await self.app(scope, receive, send)
chatweb-chatweb-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 718, in __call__
chatweb-chatweb-1  |     await route.handle(scope, receive, send)
chatweb-chatweb-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 276, in handle
chatweb-chatweb-1  |     await self.app(scope, receive, send)
chatweb-chatweb-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 66, in app
chatweb-chatweb-1  |     response = await func(request)
chatweb-chatweb-1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 237, in app
chatweb-chatweb-1  |     raw_response = await run_endpoint_function(
chatweb-chatweb-1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 163, in run_endpoint_function
chatweb-chatweb-1  |     return await dependant.call(**values)
chatweb-chatweb-1  |   File "/app/api.py", line 34, in crawler_url
chatweb-chatweb-1  |     contents, lang = web_crawler_newspaper(req.url)
chatweb-chatweb-1  |   File "/app/contents.py", line 14, in web_crawler_newspaper
chatweb-chatweb-1  |     raw_html, lang = _get_raw_html(url)
chatweb-chatweb-1  |   File "/app/contents.py", line 35, in _get_raw_html
chatweb-chatweb-1  |     with webdriver.Chrome(options=chrome_options) as driver:
chatweb-chatweb-1  |   File "/usr/local/lib/python3.9/site-packages/selenium/webdriver/chrome/webdriver.py", line 84, in __init__
chatweb-chatweb-1  |     super().__init__(
chatweb-chatweb-1  |   File "/usr/local/lib/python3.9/site-packages/selenium/webdriver/chromium/webdriver.py", line 101, in __init__
chatweb-chatweb-1  |     self.service.start()
chatweb-chatweb-1  |   File "/usr/local/lib/python3.9/site-packages/selenium/webdriver/common/service.py", line 100, in start
chatweb-chatweb-1  |     self.assert_process_still_running()
chatweb-chatweb-1  |   File "/usr/local/lib/python3.9/site-packages/selenium/webdriver/common/service.py", line 113, in assert_process_still_running
chatweb-chatweb-1  |     raise WebDriverException(f"Service {self._path} unexpectedly exited. Status code was: {return_code}")
chatweb-chatweb-1  | selenium.common.exceptions.WebDriverException: Message: Service /root/.cache/selenium/chromedriver/linux64/114.0.5735.90/chromedriver unexpectedly exited. Status code was: 127

生成向量用的是text-embedding-ada-002

ai.py 里embedding = openai.Embedding.create(model="text-embedding-ada-002", inpu
t=text)，但是你首页说明里说生成向量用的是gpt-3.5-turbo

报错：zsh: segmentation fault python3 main.py

上传pdf文档后，查询的时候报上述错误退出

环境：macos 13.2.1 (22D68) + Python 3.10.9

The version of chrome cannot be detected. Trying with latest driver version

不管提交什么url都报错The version of chrome cannot be detected. Trying with latest driver version
请问是什么原因

Darren，有可能多开发一步，可以部署到vercl那种嘛？

如题

its possible in english or spanish?

Hello its possible translate to english or spanish please.
thanks you

How to access api through external network?

As title says.

怎么可以做到把文件里面的所有内容发给chatgpt，让他根据文件里面的所有内容（大约3000个字）来生成回答呢？

为什么只总结有限的几个章节内容，就可以出全篇的该要了呢？？

请问提取和全篇内容最相近的几个段落的embedding，然后只总结这几个段落的内容就可以得到全篇的总结，这个的依旧是什么？其他被排除的章节，内容岂不是丢失了。谢谢·

AttributeError: 'generator' object has no attribute 'choices' in ai.py

Not familar with your codes, report error as below:

Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\envs\py39\lib\site-packages\gradio\routes.py", line 414, in run_predict
output = await app.get_blocks().process_api(
File "C:\ProgramData\Anaconda3\envs\py39\lib\site-packages\gradio\blocks.py", line 1323, in process_api
result = await self.call_function(
File "C:\ProgramData\Anaconda3\envs\py39\lib\site-packages\gradio\blocks.py", line 1051, in call_function
prediction = await anyio.to_thread.run_sync(
File "C:\ProgramData\Anaconda3\envs\py39\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "C:\ProgramData\Anaconda3\envs\py39\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "C:\ProgramData\Anaconda3\envs\py39\lib\site-packages\anyio_backends_asyncio.py", line 867, in run
result = context.run(func, *args)
File "D:\coding\notNowUsing\nlp2000\News_nltk\Example_test\OpenaiFinancialReports\chatWeb\webui.py", line 130, in respond
kw = self.ai.get_keywords(message)
File "D:\coding\notNowUsing\nlp2000\News_nltk\Example_test\OpenaiFinancialReports\chatWeb\ai.py", line 86, in get_keywords
result = self._chat_stream([
File "D:\coding\notNowUsing\nlp2000\News_nltk\Example_test\OpenaiFinancialReports\chatWeb\ai.py", line 47, in _chat_stream
print(response.choices[0].message.content.strip())
AttributeError: 'generator' object has no attribute 'choices'