sotopia-lab / sotopia-pi Goto Github PK
View Code? Open in Web Editor NEWSotopia-π: Interactive Learning of Socially Intelligent Language Agents (ACL 2024)
Home Page: https://pi.sotopia.world/
License: Apache License 2.0
Sotopia-π: Interactive Learning of Socially Intelligent Language Agents (ACL 2024)
Home Page: https://pi.sotopia.world/
License: Apache License 2.0
No response
No response
No response
No response
No response
No response
No response
No response
No response
No response
No response
We need to align with together ai finetuning results
No response
After having a model checkpoint from #7 , we need to use vLLM to deploy it on the GCP server.
No response
We use:
to make the prompt shorter and avoid any >2048 cases.
No response
Current infernce and deploy pipeline lacks interface for hf access token, which needs update to use llama-2-13b. Need to update quantization config for inference to work too.
No response
Have a relatively formal report for our project.
No response
No response
No response
Since there is a published pip install sotopia, we can directly utilize that to include the data processing pipeline in this repo.
No response
We need to support sotopia code to support vLLM api.
No response
A low-level Q: I am wondering to which extent we need to directly work on FastChat library?
As mentioned in Llama-recipe, finetuning only requires a line of code if the input file is in the right form. So is inheriting the whole FastChat repo an overkill? In this way the CI pipeline is broken, and code structure is not very clear.
No response
According to together AI doc, the INST format support should support multi-turn dialogue directly, and there is no need to split the fine-tuning data points to (previous dialogue, current response). The goal of this Feature is to experiment with 1) the original (previous dialogue, current response) as the fine-tuning dataset entry and 2) putting the entire dialogue as one entry.
No response
No response
No response
No response
No response
No response
Migrate and replicate the training script of llama-2-13b from fastchat to llama factory.
No response
No response
No response
No response
No response
Adding data processing file to convert sotopia data into together AI readable format.
No response
All fine-tuning processes still stop at step 10. According to @lwaekfjlk, no early stopping code is found in the source code. And the curves shown in the below image stop at different losses. Need further experiments.
No response
Here is a comparison between two different fine-tuning settings:
epoch = 2, batch=4 -> epoch 1 stopped at step 5, epoch 2 stopped at step 10
epoch = 4, batch=32 -> epoch 1 stopped at step 3, epoch 2 stopped at step 6, epoch 3 stopped at step 9, epoch 4 stopped at step 10.
For each set of scenario split by easy and hard, we previously discuss if we want to use only dialogues that reach high performance / high quality. If we want to add this filtering, how we are going to define "high quality" should be clarified. Currently, the most simplified approach we adopt is use overall reward score, but this might be too high level.
No response
No response
No response
No response
No response
No response
No response
We want to add our finetuned LLM deployment code.
No response
We need to test whether different learning rate or batch size matters or not.
No response
We transfer the PR and issue template from sotopia to this repo.
No response
No response
Becuase of the way FastChat templates are formed, HF autograd may calculate gradients for the prompts. The prompts are masked so the gradients are zero but it may be wasting memory.
No response
No response
Currently we use plain dialogue (passing the other agent’s direct response instead of passing the prompt from environment agent) to fine-tune Llama 2 on Together AI. A qualitative analysis indicates that the fine-tuned llama-2-13b-chat performs well on selected difficult scenarios.
In order to call the fine-tuned model’s API and evaluate its performance based on Sotopia’s evaluation metrics, we may have two solutions:
Keep the same fine-tuning data and revise the prompt template for the fine-tuned model during inference. A potential problem with this solution would be the unfair(?) comparison between the fine-tuned model and the original Llama 2 model - their prompts during inference are different.
Modify the fine-tuning data to be consistent with the prompts for all other models. Fine-tune Llama 2 again with QA format instead of multi-turn INST format.
Before implementing Solution 2., we still have two key points to discuss and solve:
Issue 1. Currently we only have episode logs in the database. We could either 1) Re-run (GPT-4/GPT-4) dialogues and
collect the environment prompts in the process, or 2) Reverse the episode logs in the database to the environment prompts following a specific template.
Issue 2. Currently we have no idea about the performances of multi-turn INST fine-tuning and QA fine-tuning of Together AI. Once Issue 1 is resolved, we will experiment on the difference in performances between multi-turn INST fine-tuning and QA fine-tuning (#24).
We expect the evaluation result of the fine-tuned model trained on the bable servers to align perfectly with the fine-tuned model on Together AI.
On a side note, we are curious about the motivation of using environment prompts from the third-party GPT-4 (e.g., why is prompting with previous dialogue history important? what goals can’t multi-turn prompting format do?).
No response
First, we need to confirm which platform to use for finetuning.
Secondly, we need to run 13b-llama on this platform.
Thirdly, we need to run Mistral7b on this platfom.
The output of this issue is a model checkpoint.
No response
No response
No response
FastChat training and inferences uses Vicuna prompt template, which is in accurate since our prompt is already well-formed. We need a clean template.
No response
No response
No response
No response
Finish the instruction tuning warm up stage of sotopia model. The model gains some improvement in social ability.
No response
Based on the wandb visualization, it always stops at the 10th steps.
Have tried:
All not working.
No response
No response
No response
No response
We need to switch to llama factory.
No response
No response
No response
llama-factory contains methods for reinforced training of LLMs which we will use for the next stage of self-training.
No response
Adding filtering layer to select dialogue/scenario that produces high overall score.
No response
In the following example, the model outputs "[INST] What is the best way to eat a snowman?" after answering the question. The sentence is not found in the fine-tuning dataset and it looks like the model's hallucination.
Use either <s>[INST] question [/INST] answer</s>
or <s>[INST] <<SYS>> <</SYS>> question [/INST] answer</s>
on single-turn dialogues.
No response
The current split of fine-tuning data is based on task difficulty/variance:
All GPT-4/GPT-4 dialogues (with tag "gpt-4_gpt-4_v0.0.1_clean"
): 90 * 5 = 450
Train set: 76 (easy scenarios) * 5 = 380
Test set: 14 (hard scenarios) * 5 = 70
Choosing hard scenarios based on EnvironmentList.find(EnvironmentList.name == "hard_env_set").all()
An agreement on this archived issue should resolve issues #20 #5 #6
No response
No response
No response
No response
No response
No response
No response
No response
To prepare for submission, we can finish the 1-page naacl paper draft.
No response
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.