This repo accompanies the Cloudera Fast Forward report Few-Shot Text Classification. It provides small library to fascilitate text classification using latent text embeddings with Sentence-BERT as well as a simple application to explore text classification in several limited-labeled-data regimes.
The primary output of this repository is the Few-Shot Text Classification application, a prototype user interface for the latent text embedding classification method. It includes the ability to apply various models for both on-the-fly and few-shot classification on the AG News dataset.
Instructions are given both for general use (on a laptop, say), and for Cloudera CML and CDSW. We'll first describe what's here, then go through how to run everything.
.
├── apps # Small Streamlit application.
├── cml # This folder contains scripts that facilitate the project launch on CML.
├── data # This folder contains starter data, and is where text embeddings will live.
├── scripts # This is where all the code that does something lives.
├── notebooks # This contains several Jupyter/Colab notebooks that accompany the report and demonstrate basic usage.
└── fewshot # A small library of useful functions.
There are also images
and tests
directories that are unimportant and can be ignored. Let's examine each of the important folders in turn.
The application accompanying this project comes with a launcher script to assist launching an Application with CDSW/CML.
To launch the applications in another environment, run the code inside the launcher file, with the prefixed !
removed.
You may need to specify different ports.
This script facilitates the automated project setup on CML and is triggered by the declarative pipeline as defined in the .project-metadata.yaml
file found in the project's root directory.
fewshot
├── data
│ ├── loaders.py
│ └── utils.py
├── embeddings
│ ├── sentence_embeddings.py
│ └── word_embeddings.py
├── models
│ ├── few_shot.py
│ └── on_the_fly.py
├── eval.py
└── utils.py
The data/loaders.py
is used in all scripts and notebooks, containing code that returns a specialized Dataset
object that makes it easier to handle the original text, embeddings, and labels simultaneously.
The latent text embedding method relies on first embedding text with Sentence-BERT before performing any other steps. This code is found under embeddings/sentence_embeddings.py
. More sophisticated methods incorporate word embeddings to augment the Sentence-BERT embeddings, and this code is under embeddings/word_embeddings.py
.
There are two regimes in which we perform text classification and we include a model for each. models/few_shot.py
contains code to train a model that incorporates some labeled data, while models/on_the_fly.py
computes a model that performs classification with no labeled data at all.
We also provide helper functions for generating predictions and computing metrics such as basic accuracy in eval.py
. utils.py
contains additional helper functions for I/O and serializing data.
scripts
├── few-shot_text_classification.py
└── on-the-fly_text_classification.py
These scripts perform basic text classification for the various classification regimes and generate outputs that are used in the app. These outputs have been pre-trained and included with this repo.
notebooks
├── CreatingRedditDataset.ipynb
├── Wmap_Experiments.ipynb
└── Zmap_Experiments.ipynb
The Wmap_Experiments
and Zmap_Experiments
notebooks walk through analysis discussed in the accompanying Few-Shot Text Classification report and are intended for interactive learning purposes. These work best when run as a Colab (rather than as a Jupyter Notebook) to take advantage of free GPUs.
The CreatingRedditDataset
notebook is included purely to document the steps taken when creating the Reddit dataset that we include with this module (located in the data/reddit
directory.) It is not intended to be run directly.
To go from a fresh clone of the repo to the final state, follow these instructions in order.
The code and applications within were developed against Python 3.6.9, and are likely also to function with more recent versions of Python. We relied on GPUs for much of the analysis and use a version of PyTorch optimized for CUDA 10.2.
To install dependencies, first create and activate a new virtual environment through your preferred means, then pip install from the requirements file. I recommend:
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
In CML or CDSW, no virtual env is necessary. Instead, inside a Python 3 session (with at least 2 vCPU / 4 GiB Memory), simply run
!pip3 install -r requirements.txt # notice `pip3`, not `pip`
The requirements.txt
file installs a GPU-compatible version of PyTorch. If GPUs are not in your future, it might be prudent to instead install a CPU-only version of torch (which is more lightweight) with the following command:
!pip3 install torch==1.6.0+cpu torchvision==0.7.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
(Remove the !
for non-CML/CDSW environments.)
We use two datasets in several of the scripts and notebooks, although the few-shot application currently only allows interaction with the AG News dataset.
AG News
This is a collection of 127,600 news articles in four categories. The dataset is pulled from the open-source Datasets repository maintained by HuggingFace. The load_or_cache_data
function in the data/loaders.py
file manages downloading this dataset on its first call, after which the dataset is cached and does not need to be downloaded again.
This dataset contains nearly four million preprocessed submissions and comments from Reddit, collected between 2006 and 2016. Like AG News, it is also available on the HuggingFace Datasets repository, but it is extremely large and we do not recommend that you download it yourself. Instead, we provide curated subsamples of this dataset in the data/reddit
directory, as well as a notebook (CreatingRedditDataset.ipynb
) detailing how we performed the sampling.
To fit models and perform text classification experiments, one can either call the scripts in the scripts
directory, or walk through a more detailed process in the notebooks. Their functionality is as follows:
-
scripts/on-the-fly_text_classification.py
performs on-the-fly (zero-shot) text classification; that is, text classification with no labeled training examples. This script generates a simple model (called aZmap
, stored in thedata/maps
directory) that is used in the next script as well as in the app.Zmaps
are data-agnostic because they do not rely specifically on training data. We have already performed this analysis and include this output for you. Similar procedures are echoed in thenotebooks/Zmap_Experiments
notebook. -
scripts/few-shot_text_classification.py
performs few-shot text classification; that is, text classification with only a few labeled training examples. This script generates a model known as aWmap
.Wmaps
rely on training data and are thus specific to a given dataset. In thedata/maps
directory we include aWmap
trained on the AG News dataset, which is also used in the app. Similar procedures are echoed in thenotebooks/Wmap_Experiments
notebook.
To run scripts, follow this procedure in the terminal or a Session:
!python3 scripts/on-the-fly_text_classification.py
!python3 scripts/few-shot_text_classification.py
(Remove the !
for non-CML/CDSW environments.)
NOTE: The scripts are intended to be run on a GPU-enabled machine in order to complete in a timely fashion. On a CPU-only machine, these scripts can take from 3-6 hours to complete. With GPUs enabled, this reduces to a few minutes.
There are three ways to launch this project on CML:
- From Prototype Catalog - Navigate to the Prototype Catalog on a CML workspace, select the "Deep Learning for Question Answering" tile, click "Launch as Project", click "Configure Project"
- As ML Prototype - In a CML workspace, click "New Project", add a Project Name, select "ML Prototype" as the Initial Setup option, copy in the repo URL, click "Create Project", click "Configure Project"
- Manual Setup - In a CML workspace, click "New Project", add a Project Name, select "Git" as the Initial Setup option, copy in the repo URL, click "Create Project". Then, follow the installation instructions above.
The fewshot
module logic is partly covered by unittests. To run all tests, use:
python -m unittest discover
We recommend running tests before committing any major changes.
The end-to-end test (test_e2e.py
) will not work if files generated by on-the-fly_text_classification.py
have not been generated. (These files are checked in.)
We use black library to format code. It is required that new changes to the library conform to this style. To auto-format code, you can call:
./format_code.sh