label-sleuth Goto Github PK

  Quick Start   |   Documentation   |   Join Slack  

Label Sleuth is an open source no-code system for text annotation and building text classifers. With Label Sleuth, domain experts (e.g., physicians, lawyers, psychologists) can quickly create custom NLP models by themselves, with no dependency on NLP experts.

Creating real-world NLP models typically requires a combination of two expertise - deep knowledge of the target domain, provided by domain experts, and machine learning knowledge, provided by NLP experts. Thus, domain experts are dependent on NLP experts. Label Sleuth comes to eliminate this dependency. With an intuitive UX, it escorts domain experts in the process of labeling the data and building NLP models which are tailored to their specific needs. As domain experts label examples within the system, machine learning models are being automatically trained in the background, make predictions on new examples, and provide suggestions for the users on the examples they should label next.

Label Sleuth is a no-code system, no knowledge in machine learning is need, and - it is fast to obtain a model – from task definition to a working model in just a few hours!

Table of contents

Installation for end users

Setting up a development environment

Project structure

Using the system

Customizing the system

• System configuration
• Implementing new components

Reference

Follow the instructions on our website.

The system requires Python 3.8 or 3.9 (other versions are currently not supported and may cause issues).

1. Clone the repository:

   git clone [email protected]:label-sleuth/label-sleuth.git

2. cd to the cloned directory: cd label-sleuth

3. Install the project dependencies using conda (recommended) or pip:

# Create and activate a virtual environment:
conda create --yes -n label-sleuth python=3.9
conda activate label-sleuth
# Install requirements
pip install -r requirements.txt

pip install -r requirements.txt

4. Start the Label Sleuth server: run python -m label_sleuth.start_label_sleuth.

   By default all project files are written to <home_directory>/label-sleuth, to change the directory add --output_path <your_output_path>.

   You can add --load_sample_corpus wiki_animals_2000_pages to load a sample corpus into the system at startup. This fetches a collection of Wikipedia documents from the data-examples repository.

   By default, the host will be localhost to expose the server only on the host machine. If you wish to expose the server to external communication, add --host <IP> for example, --host 0.0.0.0 to listen to all IPs.

   Default port is 8000, to change the port add --port <port_number> to the command.

   The system can then be accessed by browsing to http://localhost:8000 (or http://localhost:<port_number>)

The repository consists of a backend library, written in Python, and a frontend that uses React. A compiled version of the frontend can be found under label_sleuth/build.

See our website for a simple tutorial that illustrates how to use the system with a sample dataset of Wikipedia pages. Before starting the tutorial, make sure you pre-load the sample dataset by running:

python -m label_sleuth.start_label_sleuth --load_sample_corpus wiki_animals_2000_pages.

The configurable parameters of the system are specified in a json file. The default configuration file is label_sleuth/config.json.

A custom configuration can be applied by passing the --config_path parameter to the "start_label_sleuth" command, e.g., python -m label_sleuth.start_label_sleuth --config_path <path_to_my_configuration_json>

Alternatively, it is possible to override specific configuration parameters at startup by adding them to the run command, e.g., python -m label_sleuth.start_label_sleuth --changed_element_threshold 100

Configurable parameters:

"users":[
 {
   "username": "<predefined_username1>",
   "token":"<randomly_generated_token1>",
   "password":"<predefined_user1_password>"
 }
] 

Label Sleuth is a modular system. We welcome the contribution of additional implementations for the various modules, aiming to support a wider range of user needs and to harness efficient and innovative machine learning algorithms.

Below are instructions for implementing new models and active learning strategies:

def _train(self, model_id: str, train_data: Sequence[Mapping], model_params: Mapping):

def load_model(self, model_path: str):

def infer(self, model_components, items_to_infer) -> Sequence[Prediction]:

 def get_per_element_score(self, candidate_text_elements: Sequence[TextElement],
                           candidate_text_element_predictions: Sequence[Prediction], workspace_id: str,
                           dataset_name: str, category_name: str) -> Sequence[float]:    

Eyal Shnarch, Alon Halfon, Ariel Gera, Marina Danilevsky, Yannis Katsis, Leshem Choshen, Martin Santillan Cooper, Dina Epelboim, Zheng Zhang, Dakuo Wang, Lucy Yip, Liat Ein-Dor, Lena Dankin, Ilya Shnayderman, Ranit Aharonov, Yunyao Li, Naftali Liberman, Philip Levin Slesarev, Gwilym Newton, Shila Ofek-Koifman, Noam Slonim and Yoav Katz (EMNLP 2022). Label Sleuth: From Unlabeled Text to a Classifier in a Few Hours.

Please cite:

@inproceedings{shnarch2022labelsleuth,
	title={{L}abel {S}leuth: From Unlabeled Text to a Classifier in a Few Hours},
	author={Shnarch, Eyal and Halfon, Alon and Gera, Ariel and Danilevsky, Marina and Katsis, Yannis and Choshen, Leshem and Cooper, Martin Santillan and Epelboim, Dina and Zhang, Zheng and Wang, Dakuo and Yip, Lucy and Ein-Dor, Liat and Dankin, Lena and Shnayderman, Ilya and Aharonov, Ranit and Li, Yunyao and Liberman, Naftali and Slesarev, Philip Levin and Newton, Gwilym and Ofek-Koifman, Shila and Slonim, Noam and Katz, Yoav},
	booktitle={Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing ({EMNLP}): System Demonstrations},
    	month={dec},
    	year={2022},
    	address={Abu Dhabi, UAE},
	publisher={Association for Computational Linguistics},
	url={https://aclanthology.org/2022.emnlp-demos.16},
    	pages={159--168}
}