Angler is an interactive visualization system that helps machine translation (MT) engineers and researchers explore and curate challenge sets to improve their models and data. Challenge sets (sometimes called "golden sets" or "aggressor tests") are often small, curated sets of important data samples, which ML practitioners use to validate and monitor an ML model's behavior. We used Angler to understand how ML practitioners prioritize model improvements when the input space is infinite and obtaining reliable signals of model quality is expensive.

This code accompanies the research paper:

Angler: Helping Machine Translation Practitioners Prioritize Model Improvements
Samantha Robertson*, Zijie J. Wang*, Dominik Moritz, Mary Beth Kery, Fred Hohman
ACM Conference on Human Factors in Computing Systems (CHI), 2023.
Paper, Interactive demo, Code, *Contributed equally

• Visually explore machine translation data over time
• Compare translation datasets (e.g., usage log data versus training data)
• Surface potentially interesting and critical data samples with two sources:
  • Model's unfamiliar topics
  • Failure cases of model unit tests

Each row in the table represents a challenge set. Each set contains English-to-Chinese translation pairs from a translation dataset composed from 4 open source datasets: scientific_papers (corpus of full-text scientific articles), tatoeba (open-source translation data), umass_global (English language tweets from 2014-2016), and wmt_chat (customer service chat histories). These sets are generated because they contain unfamiliar topics or failed certain unit tests.

To sort the challenge sets, click any metric in the table header. The metrics include:

To preview a challenge set, click any row in teh Table View. The preview includes 100 sentences and the most representative keywords from the challenge set.

To see more details about a particular challenge set, click the Show Details button to open the Detail View.

The Detail View provides multiple visualizations to help users explore a particular challenge set. The visualizations include:

To focus on sentences with interesting attributes (e.g., from a particular time, with low familiarity, or from a specific dataset), users can create filters by brushing or clicking throughout the visualizations.

To build and develop Angler locally:

# Install dependencies
npm install

# Start a localhost server
npm run dev

# Navigate to localhost:5173 in any browser

When making contributions, refer to the CONTRIBUTING guidelines and read the CODE OF CONDUCT.

To cite our paper, please use:

@inproceedings{robertson2023angler,
  title={Angler: Helping Machine Translation Practitioners Prioritize Model Improvements},
  author={Robertson, Samantha and Wang, Zijie J. and Moritz, Dominik and Kery, Mary Beth and Hohman, Fred},
  booktitle={Proceedings of the SIGCHI Conference on Human Factors in Computing Systems},
  year={2023},
  organization={ACM},
  doi={10.1145/3544548.3580790}
}

This code is released under the LICENSE terms.