When I search through oogabooga the results are solid with this extension,
But when using SillyTavern with oogabooga API, search doesnt really search like it does natively.
Are there way to make it work? any way?

selenium is installed, but it fails to load.

2023-11-03 10:46:04 ERROR:Failed to load the extension "web_search".
Traceback (most recent call last):
File "/home/mouse/test/text-generation-webui/modules/extensions.py", line 36, in load_extensions
exec(f"import extensions.{name}.script")
File "", line 1, in
File "/home/mouse/test/text-generation-webui/extensions/web_search/script.py", line 3, in
from selenium import webdriver
ModuleNotFoundError: No module named 'selenium'

any ideas? (Linux)

In other news, Brave is including a privacy oriented llama2 chatbot into the browser, should be out in a few days. It will be interesting to see how well it integrates with the browser. I imagine they'll go with the heavily aligned llama2 chat version to keep themselves out of trouble. You should check it out when it arrives, since using a local model gives the user unlimited, fast usage without alignment. Your little extension is about to become more relevant, and I'll bet the brave integration will give you a lot of ideas.

What is the proper way to update this extension?

The issue i'm encountering stems from how Python's strip() method works. The strip() function removes the characters provided in its argument from the beginning and end of a string. In your example, when you call strin.strip("search").strip(), you're removing all instances of the characters "s", "e", "a", "r", "c", and "h" from both the beginning and end of the string "what is the current record of the Yankees".

To remove only the word "search" from the beginning of a string, you can use Python's replace() method, which replaces occurrences of a substring with another substring.

query = user_input.replace("search", "").strip()

After using this extension for a while, the search function has seemed to stop working. It produces this output when it attempts to search. I have tried this on the webui and through the API (I mainly use the API) and this occurs on both. I have Opera, Opera Beta, and Google Chrome browsers installed on a Pop os 22.04 install.

127.0.0.1 - - [15/Oct/2023 12:26:36] "POST /api/v1/chat HTTP/1.1" 200 -
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 32970)
Traceback (most recent call last):
  File "/home/hav0x/Downloads/text-generation-webui/installer_files/env/lib/python3.10/socketserver.py", line 683, in process_request_thread
    self.finish_request(request, client_address)
  File "/home/hav0x/Downloads/text-generation-webui/installer_files/env/lib/python3.10/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/home/hav0x/Downloads/text-generation-webui/installer_files/env/lib/python3.10/socketserver.py", line 747, in __init__
    self.handle()
  File "/home/hav0x/Downloads/text-generation-webui/installer_files/env/lib/python3.10/http/server.py", line 433, in handle
    self.handle_one_request()
  File "/home/hav0x/Downloads/text-generation-webui/installer_files/env/lib/python3.10/http/server.py", line 421, in handle_one_request
    method()
  File "/home/hav0x/Downloads/text-generation-webui/extensions/api/blocking_api.py", line 89, in do_POST
    for a in generator:
  File "/home/hav0x/Downloads/text-generation-webui/modules/chat.py", line 297, in generate_chat_reply
    for history in chatbot_wrapper(text, state, regenerate=regenerate, _continue=_continue, loading_message=loading_message):
  File "/home/hav0x/Downloads/text-generation-webui/modules/chat.py", line 207, in chatbot_wrapper
    text = apply_extensions('input', text, state, is_chat=True)
  File "/home/hav0x/Downloads/text-generation-webui/modules/extensions.py", line 224, in apply_extensions
    return EXTENSION_MAP[typ](*args, **kwargs)
  File "/home/hav0x/Downloads/text-generation-webui/modules/extensions.py", line 82, in _apply_string_extensions
    text = func(*args, **kwargs)
  File "/home/hav0x/Downloads/text-generation-webui/extensions/web_search/script.py", line 52, in input_modifier
    search_data = google_results(query,state)
  File "/home/hav0x/Downloads/text-generation-webui/extensions/web_search/script.py", line 21, in google_results
    driver = webdriver.Chrome(service=service,options=options)
  File "/home/hav0x/Downloads/text-generation-webui/installer_files/env/lib/python3.10/site-packages/selenium/webdriver/chrome/webdriver.py", line 45, in __init__
    super().__init__(
  File "/home/hav0x/Downloads/text-generation-webui/installer_files/env/lib/python3.10/site-packages/selenium/webdriver/chromium/webdriver.py", line 53, in __init__
    self.service.start()
  File "/home/hav0x/Downloads/text-generation-webui/installer_files/env/lib/python3.10/site-packages/selenium/webdriver/common/service.py", line 98, in start
    self._start_process(self._path)
  File "/home/hav0x/Downloads/text-generation-webui/installer_files/env/lib/python3.10/site-packages/selenium/webdriver/common/service.py", line 204, in _start_process
    self.process = subprocess.Popen(
  File "/home/hav0x/Downloads/text-generation-webui/installer_files/env/lib/python3.10/subprocess.py", line 837, in __init__
    errread, errwrite) = self._get_handles(stdin, stdout, stderr)
  File "/home/hav0x/Downloads/text-generation-webui/installer_files/env/lib/python3.10/subprocess.py", line 1638, in _get_handles
    c2pwrite = stdout.fileno()
ValueError: I/O operation on closed file
----------------------------------------

Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0
/bin/sh: 1: google-chrome: not found
/bin/sh: 1: google-chrome-stable: not found
/bin/sh: 1: google-chrome-beta: not found
/bin/sh: 1: google-chrome-dev: not found
/bin/sh: 1: google-chrome: not found
/bin/sh: 1: google-chrome-stable: not found
/bin/sh: 1: google-chrome-beta: not found
/bin/sh: 1: google-chrome-dev: not found
16:39:45-461903 ERROR Failed to load the extension "web_search".
Traceback (most recent call last):
File "/home/lenovoubuntu/Downloads/text-generation-webui-main/modules/extensions.py", line 37, in load_extensions
exec(f"import extensions.{name}.script")
File "", line 1, in
File "/home/lenovoubuntu/Downloads/text-generation-webui-main/extensions/web_search/script.py", line 19, in
service = Service(ChromeDriverManager().install())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lenovoubuntu/Downloads/text-generation-webui-main/installer_files/env/lib/python3.11/site-packages/webdriver_manager/chrome.py", line 40, in install
driver_path = self._get_driver_binary_path(self.driver)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lenovoubuntu/Downloads/text-generation-webui-main/installer_files/env/lib/python3.11/site-packages/webdriver_manager/core/manager.py", line 40, in _get_driver_binary_path
file = self._download_manager.download_file(driver.get_driver_download_url(os_type))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lenovoubuntu/Downloads/text-generation-webui-main/installer_files/env/lib/python3.11/site-packages/webdriver_manager/drivers/chrome.py", line 32, in get_driver_download_url
driver_version_to_download = self.get_driver_version_to_download()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lenovoubuntu/Downloads/text-generation-webui-main/installer_files/env/lib/python3.11/site-packages/webdriver_manager/core/driver.py", line 48, in get_driver_version_to_download
return self.get_latest_release_version()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lenovoubuntu/Downloads/text-generation-webui-main/installer_files/env/lib/python3.11/site-packages/webdriver_manager/drivers/chrome.py", line 64, in get_latest_release_version
determined_browser_version = ".".join(determined_browser_version.split(".")[:3])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'split'

Any ideas? I just downloaded chrome from the new debian package. When i run google-chrome it launches chrome. I don't know why this does not work.

I get this when booting up, after git pull and installing requirements.

2023-09-01 13:33:50 INFO:Loading the extension "web_search"...
2023-09-01 13:33:50 ERROR:Failed to load the extension "web_search".
Traceback (most recent call last):
File "D:\oobabooga_windows\text-generation-webui\modules\extensions.py", line 35, in load_extensions
exec(f"import extensions.{name}.script")
File "", line 1, in
File "D:\oobabooga_windows\text-generation-webui\extensions\web_search\script.py", line 7, in
from requests_html import HTML
ModuleNotFoundError: No module named 'requests_html'

Output generated in 2.47 seconds (3.25 tokens/s, 8 tokens, context 28, seed 295305407)
Traceback (most recent call last):
File "D:\oobabooga_windows\installer_files\env\lib\site-packages\gradio\routes.py", line 427, in run_predict
output = await app.get_blocks().process_api(
File "D:\oobabooga_windows\installer_files\env\lib\site-packages\gradio\blocks.py", line 1323, in process_api
result = await self.call_function(
File "D:\oobabooga_windows\installer_files\env\lib\site-packages\gradio\blocks.py", line 1067, in call_function
prediction = await utils.async_iteration(iterator)
File "D:\oobabooga_windows\installer_files\env\lib\site-packages\gradio\utils.py", line 336, in async_iteration
return await iterator.anext()
File "D:\oobabooga_windows\installer_files\env\lib\site-packages\gradio\utils.py", line 329, in anext
return await anyio.to_thread.run_sync(
File "D:\oobabooga_windows\installer_files\env\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "D:\oobabooga_windows\installer_files\env\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "D:\oobabooga_windows\installer_files\env\lib\site-packages\anyio_backends_asyncio.py", line 867, in run
result = context.run(func, *args)
File "D:\oobabooga_windows\installer_files\env\lib\site-packages\gradio\utils.py", line 312, in run_sync_iterator_async
return next(iterator)
File "D:\oobabooga_windows\text-generation-webui\modules\chat.py", line 328, in generate_chat_reply_wrapper
for i, history in enumerate(generate_chat_reply(text, state, regenerate, _continue, loading_message=True)):
File "D:\oobabooga_windows\text-generation-webui\modules\chat.py", line 313, in generate_chat_reply
for history in chatbot_wrapper(text, state, regenerate=regenerate, _continue=_continue, loading_message=loading_message):
File "D:\oobabooga_windows\text-generation-webui\modules\chat.py", line 205, in chatbot_wrapper
text = apply_extensions('input', text, state)
File "D:\oobabooga_windows\text-generation-webui\modules\extensions.py", line 207, in apply_extensions
return EXTENSION_MAP[typ](*args, **kwargs)
File "D:\oobabooga_windows\text-generation-webui\modules\extensions.py", line 63, in _apply_string_extensions
text = func(text)
File "D:\oobabooga_windows\text-generation-webui\extensions\web_search\script.py", line 38, in input_modifier
if internet:
NameError: name 'internet' is not defined

Any idea how to fix? :D

I have installed the googlesearch package and a small test script I did is able to access google search and receive search results. So googlesearch should work on my end. I can execute the test script in various directories, the path to the googlesearch package should be set and work ok.

But when I start the extension I get this error:
File "\mypath...\oobaboogawindows\text-generation-webui\extensions\web_search\script.py", line 3, in
from googlesearch import search
ModuleNotFoundError: No module named 'googlesearch'

this is on Windows 11

some modifications to version 1.0

Improved Handling of Online Search Results
In this version, we improved the handling of online search results by using search_data[0] to access relevant data. This simplifies the manipulation of search results, ensuring that we can extract the needed information more directly and efficiently.

import gradio as gr
import modules.shared as shared
from googlesearch import search
from bs4 import BeautifulSoup
import re
import requests

# Global variable to enable/disable Google search
search_access = False

# Function to create user interface
def ui():
    global search_access
    checkbox = gr.Checkbox(value=search_access, label="Activate Google Search.")
    checkbox.change(fn=update_search_access, inputs=checkbox)
    return checkbox, search_access

# Function to update access to Google search.
def update_search_access(checkbox_value):
    global search_access
    search_access = checkbox_value
    return search_access, checkbox_value

# User input modification function
def input_modifier(user_input, state):
    global search_access
    if search_access:
        if user_input.lower().startswith("search"):
            shared.processing_message = "*Search online...*"
            try:
                query = user_input.replace("search", "").strip()
                search_results = search(query, num_results=1)
                search_data = [(get_text(result), result) for result in search_results]
                state["context"] = "Summarize the results online and answer the question correctly, providing all the information in as much detail as possible, with links if necessary."
                user_prompt = f"{user_input}\n results online: {search_data[0]}"
                return user_prompt
            except Exception as e:
                print(type(e), e)
                state["context_instruct"] = "Notify user of connection problems"
                return ""
    shared.processing_message = "*Typing...*"
    return user_input

# Output modification function
def output_modifier(output):
    return output

# Bot prefix modification function
def bot_prefix_modifier(prefix):
    return prefix

# Function for displaying data (not used in this code)
def print_data(data):
    return data

# Function to extract text from a URL
def get_text(url):
    response = requests.get(url)
    soup = "\n".join([p.text for p in BeautifulSoup(response.text, "html.parser").find_all(["section", "p"])])
    soup = soup[:2044]
    text = re.sub(r"\s+", " ", soup.strip())
    output = "\n".join([text, f" -Source: {response.url}"])
    return output

Won't work at all, tons of errors.

tons.

Hi Simbake,

The moment I run a search in ooba, I receive a KeyError: 'context_instruct'. Related to line 96 in the script.py file.
Do you have any idea what might be the cause to this? Or better yet, what might be the solution to it ;).

Thanks,

I have noticed that the extension overrides the context supplied by the user, which breaks character interactions. I would like to suggest that the extension adds the search results and additional instructions to the end of the current context instead of replacing it with the relevant search results.

I have observed the "overriding" behavior of the extension with the verbose flag enabled.

After installing this extension and running with the --extension web_search flag, it gives me this error:

Traceback (most recent call last):
  File "G:\LLM\oobabooga_windows\text-generation-webui\modules\extensions.py", line 35, in load_extensions
    exec(f"import extensions.{name}.script")
  File "<string>", line 1, in <module>
  File "G:\LLM\oobabooga_windows\text-generation-webui\extensions\web_search\script.py", line 3, in <module>
    from googlesearch import search
ModuleNotFoundError: No module named 'googlesearch'

hi , first time user , everything is up to date , everything is running well ,
when asking plain questions (for expl: what is the plot of interstellar ?) i can see the phrase "searching online for a " and it s giving back an accurate response
but it doesnt take the search keyword , if i start my prompt with search ... the prompt get erased only , i tried the prompts from the front page :
search the weather in Nairobi, Kenya today.
or
search "the weather in Amsterdam" and tell me what should I wear.
the prompts just get erased and nothing is sent to the model

i tried just the prompt "search" and i got "I'm sorry, but as an AI developed by OpenAI, my purpose is not to perform web searches directly. However, if you provide me with a specific question or topic, I can certainly help you find relevant information from what I was trained on. Just let me know!"

it s not an issue i think , but the prompts from the front page not working was confusing . or am i doing something wrong ?

thank you for this plugin