This Google Summer of Code (GSoC) 2024 project "Frame Blending by LLMs" is contributed by Zhongheng Cheng with Red Hen Lab.

My personal progress blog can be found here

• Installation
• Frame Blender
• Frame Hierarchy Analyzer
• FrameNet XML Parser
• RAG for Llama2 (Huggingface)
• Llama2 (Meta)
• Future Work

Follow the instructions to setup environment.

Modules to load on CWRU HPC:

• Python/3.11.3
• PyTorch/2.1.2-foss-2023a-CUDA-12.1.1
• PyYAML/6.0-GCCcore-12.3.0

# Cloning the repository
git clone https://github.com/Zhongheng-Cheng/gsoc2024-frame-blending
cd gsoc2024-frame-blending

# [Optional] Creating virtual environment
python -m venv venv
source venv/bin/activate

# Download dependencies
pip install -r requirements.txt

# Setup Huggingface API key
touch .env
# Enter your Huggingface API key in ".env" like this:
# HUGGINGFACE_API_KEY="..."

This is a terminal application built for CWRU HPC for generating frame blending examples. Users can enter multiple frames, select their hierarchically related frames, and generate frame blending results with options including zero/one/few-shot, CoT and rhetorical devices.

1. Basic installation mentioned in Installation.

2. Setup FrameNet dataset.

   • Copy the FrameNet dataset folder frame/ to the work directory.
   • Create JSON-format FrameNet dataset folder frame_json/ using FrameNet XML Parser.

   python framenet_xml_parser.py

3. Request an interactive job on the GPU node of CWRU HPC.

The program can be run using the ./frame_blender command and accepts the following flags:

• --encoding: Used to specify the encoding method。 Acceptable values include utf-8 and ascii, with utf-8 as the default.
• --debug: Used to enable debug mode, which directly outputs the prompt instead of the generation result from LLM. Acceptable values are True or False, with False as the default.

Note: If the box-drawing characters do not display well on HPC terminal, please set --encoding=ascii when running Frame Blender.

When in Frame Blender interface:

• Esc: Quit
• Tab: Move to next window
• +/-: Add/Remove input window
• In Settings window:
  • Arrow keys: Switch settings and change setting
• In Input window:
  • Enter characters in Input windows and get search result automatically in Hierarchy window when available
  • Enter: Enter Hierarchy window
  • Backspace: Cancel confirmed frame
• In Hierarchy window:
  • \: Quit Hierarchy window
  • Arrow keys: Switch different frames/frame relations
  • Enter: Confirm word
• When needed frames are all confirmed:
  • /: Start generating result
• In Result Window / Evaluation Window:
  • \: Quit Result/Evaluation window
  • Tab: Switch to Evaluation window
  • Arrow keys/Text input: Enter evaluation
  • Enter: Submit evaluation form (stored to /data/evaluation.json)

Please check out this link for a demonstration video:

https://zhongheng-cheng.github.io/2024/08/16/Week-12.html#guidance-to-run-frame-blender-on-cwru-hpc

This section of the project focuses on analyzing linguistic frame hierarchies. It involves constructing a tree-like structure to represent frame relations, searching within this structure, and performing other relevant analyses.

from frame_hierarchy_analyzer import analyze_hierarchy, save_hierarchy_to_file

# Example of building a hierarchy
frames = ['Event', 'Action', ...]
frame_relation = 'Inheritance'
reverse_order = False # False: In direction of "Is Inherited by"; True: In direction of "Inherits from"
root = analyze_hierarchy(frames, frame_relation, reverse_order) # Returns the root node of the tree hierarchy

# Finding a specific frame node
node = root.find('Event')

# Print the visualized hierarchy of any node with its subnodes
print(node)

# Counts the total number of nodes in the subtree including this node
total_number = node.count()

# Get the list of immediate child nodes of this node
children = node.children()

# Saving the hierarchy to a file
save_hierarchy_to_file(root, 'output_hierarchy.txt')

This code transforms the original FrameNet data in XML format to JSON format, leaving out unimportant information for frame analysis, such as frame ID and created data. Mainly developed by Rohan. Minor modifications are made to accommodate the FrameNet data input in JSON format for Frame Hierarchy Analyzer.

from framenet_xml_parser import parse

# Example of parsing a directory of .xml files
xml_folder_path = "frame"
json_folder_path = "frame_json"
parse(xml_folder_path, json_folder_path)

This code utilizes Llama2-7b-chat with Huggingface API, and achieves Retrieval Augmented Generation (RAG) leveraging Llama-index. Specifically, a JSON parser is used to read all the JSON-format FrameNet frame data, and create a query engine with vector store index for querying.

When using get_query_engine(), the index created upon reading data files would be saved to ./query_engine.index/, and be automatically loaded when getting query engine next time. To avoid saving index data locally, you can specify save_index=False as a parameter for get_query_engine().

from rag import get_query_engine, generate_response

prompt = "..."
query_engine = get_query_engine()
response = generate_response(query_engine, prompt)

Referring to Meta - 5 Steps to Getting Started with Llama 2

0. Create a virtual environment

python -m venv venv
source venv/bin/activate   # enter the virtual environment

1. Download dependencies

pip install -r requirements.txt

2. Download the model

Request download access to Llama 2 here

git clone https://github.com/facebookresearch/llama
cd llama
./download.sh # requires the pre-signed URL from Meta License

3. Convert the model weights to run with Hugging Face

# in the llama/ directory

# create a link to the tokenizer
ln -h ./tokenizer.model ./llama-2-7b-chat/tokenizer.model

# convert to hugging face format
TRANSFORM=`python -c "import transformers;print('/'.join(transformers.__file__.split('/')[:-1])+'/models/llama/convert_llama_weights_to_hf.py')"`
pip install protobuf && python $TRANSFORM --input_dir ./llama-2-7b-chat --model_size 7B --output_dir ./llama-2-7b-chat-hf

4. Write Python scripts and run the model

python main.py

Enhance the generation performance, letting the LLM really "blends" the frames rather than "mixes/uses" the frames.

• Improve the model. Newer and more powerful models (e.g. Llama 3) and more parameters(e.g. Llama2-13b, llama2-70b).
• Training. With the frame blender workflow and evaluation data might be collected in the future, feed these generation result along with their evaluation data back into the model to train the LLM to understand how to construct better frame blending examples.