Giter VIP home page Giter VIP logo

xmd's Introduction

XMD: An End-to-End Framework for Interactive Explanation-Based Debugging of NLP Models

PRs Welcome arXiv

This repo provides the model, code & data of our paper: XMD: An End-to-End Framework for Interactive Explanation-Based Debugging of NLP Models. [PDF]

Overview

NLP models are susceptible to learning spurious biases (i.e., bugs) that work on some datasets but do not properly reflect the underlying task. Explanation-based model debugging aims to resolve spurious biases by showing human users explanations of model behavior, asking users to give feedback on the behavior, then using the feedback to update the model. While existing model debugging methods have shown promise, their prototype-level implementations provide limited practical utility.

Project website: https://inklab.usc.edu/xmd

Getting Started:

  • This project is intended to be deployed on modern linux environments.
  • Install Python 3, verify install status by using python3 --version to check. python may be installed under different aliases, use the appropriate one on your system.
Setup Environment

  1. Create a virtual environment using:

    • anaconda:
      1. conda create -n xmd python=3.8
    • virtualenv:
      1. python3 -m pip install virtualenv
      2. python3 -m venv xmd
  2. Activate your environment:

    • anaconda: conda activate hilt-demo
    • virtualenv: source hilt-demo/bin/activate
  3. Install prerequisites:

    • pip install -r requirements.txt
    • Note that requirements.txt may contain package version not compatible with your installation. Please adjust accordingly.


Setup Frontend

  • Follow Vue.js frontend installation instructions here


Setup Backend

  1. Setup PostgreSQL

    1. Follow postgres's instruction to install postgres 12 on your local system
    2. Make sure postgres is started and enabled by sudo systemctl start postgresql-12 and sudo systemctl enable postgresql-12
    3. Verify postgres status by running sudo systemctl status postgresql-12
    4. Create a postgrees user by running sudo -u postgres createuser hilt-user
    5. Modify /var/lib/pgsql/12/data/pg_hba.conf so that the identification method use md5. See here in detail You can use any text editor you want. I would recommend micro or nano. As a backup,vi should be installed by default on any modern linux systems.
    6. Open annotation_backend/create_empty_db.sql, you will need to execute these sql commands as the postgres user. This can be done by:
      1. cat annotation_backend/create_empty_db.sql to print the command to the current console.
      2. sudo -u postgres psql to switch to postgres user and enter the postgres environment. If prompt for password, enter one you configured during the setup process. Try 123.
      3. Copy and paste commands in annotation_backend/create_empty_db.sql into the command window to execute these sql commands to initialize and configure the database.
      4. Exit postgres environment by typing \q
  2. Setup Django

    1. Use python3 annotation_backend/manage.py migrate to setup postgres database for access
    2. Use python annotation_backend/manage.py createsuperuser to setup django super user for website login
  3. Start the backend by using python annotation_backend/manage.py runserver 0.0.0.0:8000 Note 8000 here is hard coded in, you need to move other services that may be running on 8000 before hand.

  4. After verifying that the backend can be access via a web browser and can be accessed normally, it is recommended to use systemd or tmux to start a headless instance to allow server to serve after logout of current instance.

  5. It is recommended to use chmod and chown to set appropriate permissions for the database and source code files. Note that postgresql db need to be accessed as the postgres user.

(Old) Follow Django annotation backend installation instructions here


Setup Model API

  • Follow Vue.js frontend installation instructions here


Directory overview

  • annotation_backend/ - Django application
  • frontend/ - Vue.js frontend project directory
  • model_api/ - Model API


Citation

If you find our work helpful, please cite the following:

@article{lee2022xmd,
  title={XMD: An End-to-End Framework for Interactive Explanation-Based Debugging of NLP Models},
  author={Lee, Dong-Ho and Kadakia, Akshen and Joshi, Brihi and Chan, Aaron and Liu, Ziyi and Narahari, Kiran and Shibuya, Takashi and Mitani, Ryosuke and Sekiya, Toshiyuki and Pujara, Jay and Ren, Xiang},
  journal={arXiv preprint arXiv:2210.16978},
  year={2022}
}

xmd's People

Contributors

danny911kr avatar kirannvs avatar akshenk8 avatar yufei-pan avatar

Stargazers

 avatar  avatar Ruixin (Ray) Yang avatar Jeff Carpenter avatar  avatar  avatar  avatar

Watchers

 avatar

Forkers

kirannvs

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.