chrisluo4570 / polilean Goto Github PK

This is the official repo for From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models @ ACL 2023.

Any environment with the HuggingFace Transformers that support pipelines should work. You might need to additionally install selenium for step 3.

We mainly implement things with the text generation pipeline of Huggingface Transformers. Check out your HuggingFace model compatibility by running:

python step0_hftest.py --model <your_model> --device <your_device>

If you see success! printed out, you are good to go. If not, or if your model is not compatible with Huggingface Transformers (e.g. OpenAI models), you can skip this step. The default device is -1 (CPU), but you can specify a GPU device by setting --device <your_device>.

If your step 0 is successful, run:

python step1_response.py --model <your_model> --device <your_device>

There should be a jsonl file in response/ with your model name. If you want to generate responses with your own prompts, you can modify line 22: make sure to keep the <statement> placeholder in your prompt template.

Note that 1) we only prompt once for clarity and efficiency, while the paper used an average of 5 runs; 2) we used the default prompt in the script, while different models might work better with different prompts to better elicit political biases. These two factors, among others (e.g. LM checkpoint update, etc.), that might lead to result varaition.

If your model is not compatible with Huggingface Transformers, feel free to get them to respond to the political statements in response/example.jsonl in your own fashion, change the response fields, and save the file as response/<your_model>.jsonl.

We use an NLI-based model to evaluate whether the response agrees or disagrees with the political statement. Run:

python step2_scoring.py --model <your_model> --device <your_device>

There should be a txt file in score/ with your model name. Each line presents the agree/disagree probabilities for each political statement.

Important: Run this step on your local computer. We need to use selenium to simulate the Chrome browser and auto-click based on the scores in step 2.

1. Download the Chrome browser execuatble at link. Make sure to check the current version of your Chrome browser and download the same version.

2. Download the adblocker crx file at link.

3. change the paths to the browser executable and adblocker in step3_testing.py (lines 64 and 69).

4. Run python step3_testing.py --model <your_model>. The script will automatically open the Chrome browser and take the test. The final political leaning will be displayed on the website. Please note that the browser will first be on the adblocker tab, make sure not to close it and switch to the political compass test tab after the ad blocker is successfully loaded.

For partisan news corpora, visit POLITICS. For partisan social media corpora, please reach out to [email protected] to ask for access. Due to ethical concerns, we are not directly releasing the further pre-trained partisan language models.

Hate speech detection: link

Fake news detection: link

If you find this repo useful, please cite our paper:

@article{feng2023pretraining,
  title={From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models},
  author={Feng, Shangbin and Park, Chan Young and Liu, Yuhan and Tsvetkov, Yulia},
  journal={arXiv preprint arXiv:2305.08283},
  year={2023}
}