farama-foundation / chatarena Goto Github PK

ChatArena (or Chat Arena) is a Multi-Agent Language Game Environments for LLMs. The goal is to develop communication and collaboration capabilities of AIs.

Home Page: https://www.chatarena.org/

License: Apache License 2.0

Python 98.92% Jupyter Notebook 1.08%

large-language-models multi-agent natural-language-processing multi-agent-reinforcement-learning multi-agent-simulation ai python artificial-intelligence chatgpt gpt-4

chatarena's People

Contributors

Stargazers

Watchers

Forkers

atlas3dss tokenbender johndpope maddyonline billschumacher techthiyanes mmizutani hospitaltapes ekryski twilwa ai-jie01 soulhighwing prithvimp zwq2018 peter65374 ningding97 hamiltoniangraph francyjglisboa cherifsy mdmmn378 machinelearningrifche albertoual readmecode caphadoop ourmony deniska83 ogdao332 serenkim eunjiinkim vishalbelsare pminervini system1system2 13331112522 plurigrid zxcayumi drdudebroman aetheriaxai graphgrailai crazyofapple kgourgou nnayan aengusl sanlalex jorbecalona seshakiran itsharex aidandos elliottower stefania11 my-basement cicimmmmm franghelfi avmb arthur20035 iq-scm josephdfox dci-ai-focus-group dexhunter omarelsebaey001 orderofmagnitude1 webhope-ltd while-basic davidcincotta andrewyu0 apollohuang1 yuanleirl yeq0 adambear aifylabs dapper-magician jinho-kim-osd 1ittlesnow jmanhype cool-rr zeranamu eses-lab stevegyutyan dqxiu krrishdholakia inf-ling joeyzenghuan joshkyh dcoinhub devon-research sijial430 kyuilisworking ethansystem multi-agent-story-generation andrewtanjs azure-arc-0 taoxiaoxie antonpolishko jeromyjsmith albertbj inkedyogi bxck75 ahren09 gabology mutualismai fengyz666

chatarena's Issues

The web display is incomplete, and can't submit.

Conflicting dependencies

When doing
pip install -e .[all] on the dev branch (it's the same on main iirc) I get

ERROR: Cannot install chatarena[all]==0.1.12.6 and pettingzoo[classic]==1.23.1 because these package versions have conflicting dependencies.

The conflict is caused by:
    chatarena[all] 0.1.12.6 depends on chess==1.9.4; extra == "all"
    pettingzoo[classic] 1.23.1 depends on chess==1.7.0; extra == "classic"

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

Upcoming Features and Future Directions of Chat Arena

Firstly, we would like to express our gratitude to all the contributors and users of Chat Arena. Over the last month, our community has grown significantly and we have received numerous feature requests, bug reports, and contributions. It is exciting to see the community thriving and passionate about our project.

Given the influx of feature requests, we believe it would be beneficial to list the most common ones we've received and share some insight about our future directions.

Common Feature Requests

Here are some of the most frequently requested features that we are considering:

OpenAI and Anthropic async query API support
Batch experiment support to run multiple games concurrently
Langchain backend support
Bard backend support

Future Directions

We're focusing on the following directions:

More game environments
Improved stability and reliability
Extensibility to more backends
Benchmarking different LLMs with in game environments
Improved documentation (homepage, notebook example)
Improved Web UI

We encourage you to share your thoughts on these upcoming features and future directions. It's through your feedback and contributions that Chat Arena can continue to grow and evolve.

Thank you again for your support and for being part of our growing community.

Best,
Chat Arena Team

Last updated: May 2023

Postprocessing of agent name removal

Hi, thanks for your excellent work! I am really inspired a lot by this repo. I noticed that in chatarena/backends/openai.py, there is a postprocessing step of removing the agent name if the response starts with it. However, I observed that sometimes gpt-3.5-turbo still yields responses starting with "${agent_name}: {response}". I suggest adding one more processing step as below:

# Remove the agent name if the response starts with it
response = re.sub(rf"^\s*\[.*]:", "", response).strip()
response = re.sub(rf"^\s*{re.escape(agent_name)}\s*:", "", response).strip()

Thanks.

Outline

got a "TypeError: Tab.init() got an unexpected keyword argument 'visible'" when perform 'gradio app'

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/jason/codes/chatarena/app.py:124 in │
│ │
│ 121 │ │ │
│ 122 │ │ with gr.Row(): │
│ 123 │ │ │ with gr.Column(elem_id="col-chatbox"): │
│ ❱ 124 │ │ │ │ with gr.Tab("All", visible=True): │
│ 125 │ │ │ │ │ chatbot = gr.Chatbot(elem_id="chatbox", visible=Tr │
│ 126 │ │ │ │ │
│ 127 │ │ │ │ player_chatbots = [] │
│ │
│ /home/jason/anaconda3/lib/python3.11/site-packages/gradio/component_meta.py: │
│ 146 in wrapper │
│ │
│ 143 │ │ if in_event_listener(): │
│ 144 │ │ │ return None │
│ 145 │ │ else: │
│ ❱ 146 │ │ │ return fn(self, **kwargs) │
│ 147 │ │
│ 148 │ return wrapper │
│ 149 │
╰──────────────────────────────────────────────────────────────────────────────╯
TypeError: Tab.init() got an unexpected keyword argument 'visible'

Does it mean a incorrect version of gradio which is 4.1.1 was installed?

Future Direction and Requests

Nice to meet you, ChatArena is an exciting project. I have great expectations for it.
There are several things I would like to achieve with ChatArena.

hold a tournament in which AI, prompted by users, competes to win or lose in logical thinking and debates.
the AI becomes the GM of a tabletop RPG, and multiple human players enjoy the scenario.
create an AI version of "THE SIMS." with transplanted character behaviors.

For 1 and 2, multiple humans can participate remotely and a public viewing mechanism is required.
For 3, we would like to recreate life on stage by storing data on the tone of voice, settings, and worldview of multiple characters.
(e.g. https://arxiv.org/abs/2304.03442)

All of these need to be expanded, but will ChatArena evolve in this direction as a way of thinking?

Always get same error: ERROR: [Errno 10048] error while attempting to bind on address ('127.0.0.1', 7860)

my env is win11,
(chatarena) E:\code\ai\chatarena>conda list

packages in environment at C:\Users\zphu.conda\envs\chatarena:

Update Anthropic Client

Anthropic changed their python sdk - making this code line outdated.

https://github.com/chatarena/chatarena/blob/fa6b374bb62fa7070454962eec6a9c88bc584d63/chatarena/backends/anthropic.py#L41

Would love to know if this might help - https://github.com/BerriAI/litellm

~100 lines of code, that standardizes all the llm api calls to the OpenAI call

from litellm import completion

## set ENV variables
# ENV variables can be set in .env file, too. Example in .env.example
os.environ["OPENAI_API_KEY"] = "openai key"
os.environ["ANTHROPIC_API_KEY"] = "anthropic key"

messages = [{ "content": "Hello, how are you?","role": "user"}]

# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)

# anthropic call
response = completion("claude-v-2", messages)

orjson package rust requirement

I'm not familiar with this package but it might be worth looking into other alternatives which don't require rust, as it makes it harder to install on something like google colab (or for example, I use fish prompt on mac which had some manual installation steps required to get rust to properly work). Otherwise it should probably be in the documentation.

Feature request: RL libraries integration

Hi, I was wondering if it would be possible to integrate chat arena with something like gymnasium or pettingzoo (multi agent version of gym). Would be very interesting to see LLMs be used both for language envs and for regular envs like poker, connect four etc.

OpenAI latency in gradio app (5+ seconds to generate a response)

I'm not sure what this is a problem with, but I'm going to post it here in case anyone else has the same issue or any idea how to fix it.

When testing locally, I was able to get a response in maybe 10-20 seconds, which is slow but acceptable at least.

When testing in HuggingFace spaces, I found it didn't generate a response at all and gave me this information in the logs:

2023-11-22 23:23:14,237:INFO - HTTP Request: POST http://localhost:7860/reset "HTTP/1.1 200 OK"
2023-11-22 23:23:16,315:INFO - HTTP Request: POST http://localhost:7860/api/predict "HTTP/1.1 200 OK"
2023-11-22 23:23:45,772:INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2023-11-22 23:24:21,570:INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2023-11-22 23:24:52,416:INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2023-11-22 23:25:30,880:INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2023-11-22 23:26:14,191:INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2023-11-22 23:26:49,628:INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2023-11-22 23:26:49,629:WARNING - Agent Lex Fridman failed to generate a response. Error: 'Choice' object is not subscriptable. Sending signal to end the conversation.

It may be due to the huggingface space using a different version of gradio, for some reason it runs this (note the version specified by our pyproject is gradio 3.34)

--> RUN pip install --no-cache-dir 	gradio[oauth]==3.23.0 	"uvicorn>=0.14.0" 	spaces==0.18.0 gradio_client==0.0.2

I have no idea how to change this part of the huggingface build section, it looks like a dockerfile but I don't see one anywhere, maybe I'm missing something.

Add disclaimer

Disclaimer: The architecture as described in ChatArena may only be used to
create a benevolent AGI. Any implementation must comply with Asimov’s rules
and the United Nations Charter.

AssertionError: openai package is not installed or the API key is not set

I am running into an error: AssertionError: openai package is not installed or the API key is not set.

I am certain that both the openai package is installed and I have the API set as environment variable OPENAI_API_KEY.

openai version 1.11.1
chatarena version 0.1.18

When I run the following code in the same environment 'is_openai_available' resolves to True.

try:
import openai
except ImportError:
is_openai_available = False
# logging.warning("openai package is not installed")
else:
try:
client = openai.OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
is_openai_available = True
except openai.OpenAIError:
# logging.warning("OpenAI API key is not set. Please set the environment variable OPENAI_API_KEY")
is_openai_available = False

Any ideas what could be causing this?

Message Pool

Design a message pool to manage the messages. This allows a unified treatment of the visibility of the messages.
Draft design:
The message pool is a list of (named) tuples, where each tuple has (turn, role, message).

There should be two potential configurations for step definition: multiple players can act in the same turn (rock-paper-scissors).
The agent can only see the messages

before the current turn
visible to the current role

Method format message:
Takes message pool and self_role, enviornment_roles as inputs, returns a formatted ChatGPT API input list.

Optional dependencies required even for environments which do not use them

I believe this is a problem with the init.py importing all of the sub-environments at the same time, meaning that to import one environment you need the modules to import all of them. For example, for the Umshini environments all you need is PettingZoo with no optional dependencies, but for the PettingZoo chess example you need pettingzoo[classic].

Simplest solution would be to have PettingZoo be a core requirement, if all environments importing will require that same init file which import them (https://github.com/chatarena/chatarena/blob/fa6b374bb62fa7070454962eec6a9c88bc584d63/chatarena/environments/__init__.py#L4), otherwise the init could be changed and they could be put in separate subfolders perhaps? I put the umshini environments in a separate folder as I wanted it to be clear that they are separate and don't require the same dependencies.

An alternative is to put the existing environments into a new directory called core or something like that? That would break backwards compatibility though, as it would require people to do from chatarena.environments.core import PettingZooChess or something. But it's probably good to categorize the environments in the long run if there are going to be others added.

Support Langchain Agents

Hi, will love to try out chatarena with the feature to simply import any existing Langchain agents

Installation issues/circular imports

I have installed locally (pip install -e .) and via the requirements.txt, and manually installed the chardet package, but still get this error.

ModuleNotFoundError Traceback (most recent call last)
File ~/anaconda3/envs/umshini/lib/python3.9/site-packages/requests/compat.py:11
10 try:
---> 11 import chardet
12 except ImportError:

ModuleNotFoundError: No module named 'chardet'

During handling of the above exception, another exception occurred:

AttributeError Traceback (most recent call last)
Cell In[10], line 1
----> 1 from chatarena.arena import Arena
2 arena = Arena.from_config("examples/nlp-classroom-3players.json")

File ~/Documents/GitHub/chatarena/chatarena/arena.py:7
4 import csv
5 import logging
----> 7 from .agent import Player
8 from .environments import Environment, TimeStep, load_environment
9 from .backends import Human

File ~/Documents/GitHub/chatarena/chatarena/agent.py:8
5 import uuid
6 from abc import abstractmethod
----> 8 from .backends import IntelligenceBackend, load_backend
9 from .message import Message, SYSTEM_NAME
10 from .config import AgentConfig, Configurable, BackendConfig

File ~/Documents/GitHub/chatarena/chatarena/backends/init.py:5
3 from .base import IntelligenceBackend
4 from .openai import OpenAIChat
----> 5 from .cohere import CohereAIChat
6 from .human import Human
7 from .hf_transformers import TransformersConversational

File ~/Documents/GitHub/chatarena/chatarena/backends/cohere.py:10
8 # Try to import the cohere package and check whether the API key is set
9 try:
---> 10 import cohere
11 except ImportError:
12 is_cohere_available = False

File ~/anaconda3/envs/umshini/lib/python3.9/site-packages/cohere/init.py:1
----> 1 from cohere.client import Client
2 from cohere.client_async import AsyncClient
3 from cohere.error import CohereAPIError, CohereConnectionError, CohereError

File ~/anaconda3/envs/umshini/lib/python3.9/site-packages/cohere/client.py:8
5 from concurrent.futures import ThreadPoolExecutor
6 from typing import Any, Dict, List, Optional, Union
----> 8 import requests
9 from requests.adapters import HTTPAdapter
10 from urllib3 import Retry

File ~/anaconda3/envs/umshini/lib/python3.9/site-packages/requests/init.py:45
41 import warnings
43 import urllib3
---> 45 from .exceptions import RequestsDependencyWarning
47 try:
48 from charset_normalizer import version as charset_normalizer_version

File ~/anaconda3/envs/umshini/lib/python3.9/site-packages/requests/exceptions.py:9
1 """
2 requests.exceptions
3 ~~~~~~~~~~~~~~~~~~~
4
5 This module contains the set of Requests' exceptions.
6 """
7 from urllib3.exceptions import HTTPError as BaseHTTPError
----> 9 from .compat import JSONDecodeError as CompatJSONDecodeError
12 class RequestException(IOError):
13 """There was an ambiguous exception that occurred while handling your
14 request.
15 """

File ~/anaconda3/envs/umshini/lib/python3.9/site-packages/requests/compat.py:13
11 import chardet
12 except ImportError:
---> 13 import charset_normalizer as chardet
15 import sys
17 # -------
18 # Pythons
19 # -------
20
21 # Syntax sugar.

File ~/anaconda3/envs/umshini/lib/python3.9/site-packages/charset_normalizer/init.py:23
1 """
2 Charset-Normalizer
3 ~~~~~~~~~~~~~~
(...)
21 :license: MIT, see LICENSE for more details.
22 """
---> 23 from charset_normalizer.api import from_fp, from_path, from_bytes, normalize
24 from charset_normalizer.legacy import detect
25 from charset_normalizer.version import version, VERSION

File ~/anaconda3/envs/umshini/lib/python3.9/site-packages/charset_normalizer/api.py:10
7 PathLike = Union[str, 'os.PathLike[str]'] # type: ignore
9 from charset_normalizer.constant import TOO_SMALL_SEQUENCE, TOO_BIG_SEQUENCE, IANA_SUPPORTED
---> 10 from charset_normalizer.md import mess_ratio
11 from charset_normalizer.models import CharsetMatches, CharsetMatch
12 from warnings import warn

AttributeError: partially initialized module 'charset_normalizer' has no attribute 'md__mypyc' (most likely due to a circular import)

Fix OpenAI errors

Hi @yuxiang-wu,

Tried to do something similar and hit a ton of OpenAI errors (Context window limitations, rate limiting with gpt-4, etc.).

Would recommend using something like reliableGPT for error handling - it'll do retries/model switching/etc. for you

from reliablegpt import reliableGPT
openai.ChatCompletion.create = reliableGPT(openai.ChatCompletion.create, ...)

Source: https://github.com/BerriAI/reliableGPT

HF demo does not work and has an error

See https://huggingface.co/spaces/chatarena/chatarena-demo