Giter VIP home page Giter VIP logo

lexsubgen's Introduction

LexSubGen

Lexical Substitution Framework

This repository contains the code to reproduce the results from the paper:

Arefyev Nikolay, Sheludko Boris, Podolskiy Alexander, Panchenko Alexander, "Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution", Proceedings of the 28th International Conference on Computational Linguistics, 2020

Installation

Clone LexSubGen repository from github.com.

git clone https://github.com/Samsung/LexSubGen
cd LexSubGen

Setup anaconda environment

  1. Download and install conda
  2. Create new conda environment
    conda create -n lexsubgen python=3.7.4
  3. Activate conda environment
    conda activate lexsubgen
  4. Install requirements
    pip install -r requirements.txt
  5. Download spacy resources and install context2vec and word_forms from github repositories
    ./init.sh

Setup Web Application

If you do not plan to use the Web Application, skip this section and go to the next!

  1. Download and install NodeJS and npm.
  2. Run script for install dependencies and create build files.
bash web_app_setup.sh

Install lexsubgen library

python setup.py install

Results

Results of the lexical substitution task are presented in the following table. To reproduce them, follow the instructions above to install the correct dependencies.

Model SemEval COINCO
GAP P@1 P@3 R@10 GAP P@1 P@3 R@10
OOC 44.65 16.82 12.83 18.36 46.3 19.58 15.03 12.99
C2V 55.82 7.79 5.92 11.03 48.32 8.01 6.63 7.54
C2V+embs 53.39 28.01 21.72 33.52 50.73 29.64 24.0 21.97
ELMo 53.66 11.58 8.55 13.88 49.47 13.58 10.86 11.35
ELMo+embs 54.16 32.0 22.2 31.82 52.22 35.96 26.62 23.8
BERT 54.42 38.39 27.73 39.57 50.5 42.56 32.64 28.73
BERT+embs 53.87 41.64 30.59 43.88 50.85 46.05 35.63 31.67
RoBERTa 56.74 32.25 24.26 36.65 50.82 35.12 27.35 25.41
RoBERTa+embs 58.74 43.19 31.19 44.61 54.6 46.54 36.17 32.1
XLNet 59.12 31.75 22.83 34.95 53.39 38.16 28.58 26.47
XLNet+embs 59.62 49.53 34.9 47.51 55.63 51.5 39.92 35.12

Results reproduction

Here we list XLNet reproduction commands that correspond to the results presented in the table above. Reproduction commands for all models you can find in scripts/lexsub-all-models.sh Besides saving to the 'run-directory' all results are saved using mlflow. To check them you can run mlflow ui in LexSubGen directory and then open the web page in a browser.

Also you can use pytest to check the reproducibility. But it may take a long time:

pytest tests/results_reproduction
  • XLNet:

XLNet Semeval07:

python lexsubgen/evaluations/lexsub.py solve --substgen-config-path configs/subst_generators/lexsub/xlnet.jsonnet --dataset-config-path configs/dataset_readers/lexsub/semeval_all.jsonnet --run-dir='debug/lexsub-all-models/semeval_all_xlnet' --force --experiment-name='lexsub-all-models' --run-name='semeval_all_xlnet'

XLNet CoInCo:

python lexsubgen/evaluations/lexsub.py solve --substgen-config-path configs/subst_generators/lexsub/xlnet.jsonnet --dataset-config-path configs/dataset_readers/lexsub/coinco.jsonnet --run-dir='debug/lexsub-all-models/coinco_xlnet' --force --experiment-name='lexsub-all-models' --run-name='coinco_xlnet'

XLNet with embeddings similarity Semeval07:

python lexsubgen/evaluations/lexsub.py solve --substgen-config-path configs/subst_generators/lexsub/xlnet_embs.jsonnet --dataset-config-path configs/dataset_readers/lexsub/semeval_all.jsonnet --run-dir='debug/lexsub-all-models/semeval_all_xlnet_embs' --force --experiment-name='lexsub-all-models' --run-name='semeval_all_xlnet_embs'

XLNet with embeddings similarity CoInCo:

python lexsubgen/evaluations/lexsub.py solve --substgen-config-path configs/subst_generators/lexsub/xlnet_embs.jsonnet --dataset-config-path configs/dataset_readers/lexsub/coinco.jsonnet --run-dir='debug/lexsub-all-models/coinco_xlnet_embs' --force --experiment-name='lexsub-all-models' --run-name='coinco_xlnet_embs'

Word Sense Induction Results

Model SemEval 2013 SemEval 2010
AVG AVG
XLNet 33.4 52.1
XLNet+embs 37.3 54.1

To reproduce these results use 2.3.0 version of transformers and the following command:

bash scripts/wsi.sh

Web application

You could use command line interface to run Web application.

# Run main server
lexsubgen-app run --host HOST 
                  --port PORT 
                  [--model-configs CONFIGS] 
                  [--start-ids START-IDS] 
                  [--start-all] 
                  [--restore-session]

Example:

# Run server and serve models BERT and XLNet. 
# For BERT create server for serving model and substitute generator instantly (load resources in memory).
# For XLNet create only server.
lexsubgen-app run --host '0.0.0.0' 
                  --port 5000 
                  --model-configs '["my_cool_configs/bert.jsonnet", "my_awesome_configs/xlnet.jsonnet"]' 
                  --start-ids '[0]'

# After shutting down server JSON file with session dumps in the '~/.cache/lexsubgen/app_session.json'.
# The content of this file looks like:
# [
#     'my_cool_configs/bert.jsonnet',
#     'my_awesome_configs/xlnet.jsonnet',
# ]
# You can restore it with flag 'restore-session'
lexsubgen-app run --host '0.0.0.0' 
                  --port 5000 
                  --restore-session
# BERT and XLNet restored now
Arguments:
Argument Default Description
--help Show this help message and exit
--host IP address of running server host
--port 5000 Port for starting the server
--model-configs [] List of file paths to the model configs.
--start-ids [] Zero-based indices of served models for which substitute generators will be created
--start-all False Whether to create substitute generators for all served models
--restore-session False Whether to restore session from previous Web application run

FAQ

  1. How to use gpu? - You can use environment variable CUDA_VISIBLE_DEVICES to use gpu for inference: export CUDA_VISIBLE_DEVICES='1' or CUDA_VISIBLE_DEVICES='1' before your command.
  2. How to run tests? - You can use pytest: pytest tests

lexsubgen's People

Contributors

agoel00 avatar bsheludko avatar bykovdmitrii avatar nvanva avatar somang-park avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

lexsubgen's Issues

Errors in setting up the code base

Thanks for the great work and making it open source. I was trying to set up the code and follow this example, however, while setting it up, I am encountering this error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-12-2ebbdaa05245> in <module>()
      1 # Loading substitute generator
      2 sg = SubstituteGenerator.from_config(
----> 3     str(CONFIGS_PATH / "subst_generators" / "lexsub" / "xlnet_embs.jsonnet")
      4 )

19 frames
/content/LexSubGen/lexsubgen/subst_generator.py in from_config(cls, config_path)
     96             object of the SubstituteGenerator class.
     97         """
---> 98         subst_generator, _ = build_from_config_path(config_path)
     99         return subst_generator
    100 

/content/LexSubGen/lexsubgen/utils/params.py in build_from_config_path(config_path, config)
    194     params = Params(config)
    195 
--> 196     return build_from_params(params), config
    197 
    198 

/content/LexSubGen/lexsubgen/utils/params.py in build_from_params(params)
    236                 item = build_from_params(item_params)
    237             elif isinstance(item_params, list):
--> 238                 item = [build_from_params(elem_params) for elem_params in item_params]
    239             else:
    240                 item = item_params

/content/LexSubGen/lexsubgen/utils/params.py in <listcomp>(.0)
    236                 item = build_from_params(item_params)
    237             elif isinstance(item_params, list):
--> 238                 item = [build_from_params(elem_params) for elem_params in item_params]
    239             else:
    240                 item = item_params

/content/LexSubGen/lexsubgen/utils/params.py in build_from_params(params)
    226     if "class_name" in params:
    227         cls_name = params.pop("class_name")
--> 228         cls = clsname2cls(cls_name)
    229 
    230         # init params acquisition

/content/LexSubGen/lexsubgen/utils/params.py in clsname2cls(clsname)
    203         import_path += "." + module_path
    204     # try:
--> 205     module = importlib.import_module(import_path)
    206     cls = getattr(module, clsname)
    207     # except Exception as e:

/usr/lib/python3.7/importlib/__init__.py in import_module(name, package)
    125                 break
    126             level += 1
--> 127     return _bootstrap._gcd_import(name[level:], package, level)
    128 
    129 

/usr/lib/python3.7/importlib/_bootstrap.py in _gcd_import(name, package, level)

/usr/lib/python3.7/importlib/_bootstrap.py in _find_and_load(name, import_)

/usr/lib/python3.7/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_)

/usr/lib/python3.7/importlib/_bootstrap.py in _load_unlocked(spec)

/usr/lib/python3.7/importlib/_bootstrap_external.py in exec_module(self, module)

/usr/lib/python3.7/importlib/_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)

/content/LexSubGen/lexsubgen/post_processors/target_excluder.py in <module>()
      8 
      9 
---> 10 class TargetExcluder(PostProcessor):
     11     def __init__(self, lemmatizer: Optional[str] = None, use_pos_tag: bool = True):
     12         """

/content/LexSubGen/lexsubgen/post_processors/target_excluder.py in TargetExcluder()
     30         target_words: List[str],
     31         target_pos: Optional[List[str]] = None,
---> 32     ) -> Tuple[np.ndarray, Dict[str, int]]:
     33         """
     34         Abstract method that transforms prob estimator predictions.

/usr/local/lib/python3.7/dist-packages/overrides/overrides.py in overrides(method, check_signature, check_at_runtime)
     86     """
     87     if method is not None:
---> 88         return _overrides(method, check_signature, check_at_runtime)
     89     else:
     90         return functools.partial(

/usr/local/lib/python3.7/dist-packages/overrides/overrides.py in _overrides(method, check_signature, check_at_runtime)
    112                 return wrapper  # type: ignore
    113             else:
--> 114                 _validate_method(method, super_class, check_signature)
    115                 return method
    116     raise TypeError(f"{method.__qualname__}: No super class method found")

/usr/local/lib/python3.7/dist-packages/overrides/overrides.py in _validate_method(method, super_class, check_signature)
    133         and not isinstance(super_method, property)
    134     ):
--> 135         ensure_signature_is_compatible(super_method, method, is_static)
    136 
    137 

/usr/local/lib/python3.7/dist-packages/overrides/signature.py in ensure_signature_is_compatible(super_callable, sub_callable, is_static)
     93         ensure_return_type_compatibility(super_type_hints, sub_type_hints, method_name)
     94         ensure_all_kwargs_defined_in_sub(
---> 95             super_sig, sub_sig, super_type_hints, sub_type_hints, is_static, method_name
     96         )
     97         ensure_all_positional_args_defined_in_sub(

/usr/local/lib/python3.7/dist-packages/overrides/signature.py in ensure_all_kwargs_defined_in_sub(super_sig, sub_sig, super_type_hints, sub_type_hints, check_first_parameter, method_name)
    154             ):
    155                 raise TypeError(
--> 156                     f"`{method_name}: {name} must be a supertype of `{super_param.annotation}` but is `{sub_param.annotation}`"
    157                 )
    158 

TypeError: `TargetExcluder.transform: target_words must be a supertype of `typing.Union[typing.List[str], NoneType]` but is `typing.List[str]`

I made sure that the environment and versions are the same as suggested in the repo. Any help would be appreciated. Thanks

Adding support for other huggingface models

In the example listed here, is it possible to use other models from the huggingface models hub to generate lexical substitutes? I am happy to contribute more models to the repo once I understand the pipeline of adding new models.

Also, which approach from the paper does this example correspond to? Is it the XLNet+embs approach listed in bold in this table from the paper?
Screenshot 2021-10-12 at 11 20 40 AM

Scores order in GAP

cumsum = np.cumsum(list(gold_mapping.values()))

Could you please explain why we don't need to sort values here to calculate GAP?
I believe we need, but it assumed that we give an input dict with the right order of insertion and it works properly. I think it's better to mention this fact in the annotation or to sort values.

Anyway, thanks for the great results and the interesting article!

There appear to be 1 leaked semaphore objects to clean up at shutdown

I am getting this message UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' when I try to run my code:

def predictor_pt6(left, right):
    LEXSUBGEN_ROOT = str(Path().resolve().parent)
   
 if LEXSUBGEN_ROOT not in sys.path:
        sys.path.insert(0, LEXSUBGEN_ROOT)
    
    CONFIGS_PATH = Path().resolve().parent / "configs"
    os.environ["CUDA_VISIBLE_DEVICES"]="1,2" 
    sg = SubstituteGenerator.from_config(
    str(CONFIGS_PATH / "subst_generators" / "lexsub" / "xlnet_embs.jsonnet")
    )

 
    mask = "target"
    input = f"{left} {mask} {right}"
    index_mask = input.split().index(mask)
    substitutes, w2id = sg.generate_substitutes(input, [index_mask], target_pos=["v"]
    
    return substitutes

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.