Giter VIP home page Giter VIP logo

street-reasoning's Introduction

STREET: a Multi-Task Structured Reasoning and Explanation Benchmark

Running Geneartion Script

This repository contains the annotations and script to generate the data for the STREET dataset.

Requirements:

  • Python 3

Execute the following python script:

python run_street_data_generation.py

The script will download some of the original datasets in which STREET is built and merge them with our reasoning and explanation annotations.

The final data will be contained within the ./data/$STREET_TASK$/reasoning_annotated_$SPLIT$.jsonl files in JSONL format.

Citation

@inproceedings{
    ribeiro2023street,
    title={{STREET}: A {MULTI}-{TASK} {STRUCTURED} {REASONING} {AND} {EXPLANATION} {BENCHMARK}},
    author={Danilo Neves Ribeiro and Shen Wang and Xiaofei Ma and Henghui Zhu and Rui Dong and Deguang Kong and Juliette Burger and Anjelica Ramos and zhiheng huang and William Yang Wang and George Karypis and Bing Xiang and Dan Roth},
    booktitle={International Conference on Learning Representations},
    year={2023},
    url={https://openreview.net/forum?id=1C_kSW1-k0}
}

street-reasoning's People

Contributors

amazon-auto avatar henghui-zhu-amazon avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar

Forkers

henghui-zhu-amazon xiang-li-oss csshali

street-reasoning's Issues

Cannot download annotation

Hi,

Thanks for the wonderful work! However, it seems that the download link for annotations doesn't work. Here is the error message:

Downloading original AQUA-RAT...
Downloading original AR-LSAT...
Merging original AQUA-RAT with reasoning graph annotations...
data/aqua_rat/aqua_rat_graph_annotation_train.jsonl
Traceback (most recent call last):
  File "/cluster/work/sachan/yifan/data/cg_reasoning/street/street-reasoning-main/run_street_data_generation.py", line 79, in <module>
    main()
  File "/cluster/work/sachan/yifan/data/cg_reasoning/street/street-reasoning-main/run_street_data_generation.py", line 76, in main
    merge_with_reasoning_graph_data()
  File "/cluster/work/sachan/yifan/data/cg_reasoning/street/street-reasoning-main/run_street_data_generation.py", line 68, in merge_with_reasoning_graph_data
    aqua_rat_data = dataset_utils.AquaRatData()
  File "/cluster/work/sachan/yifan/data/cg_reasoning/street/street-reasoning-main/utils/dataset_utils.py", line 112, in __init__
    self.load_graph_annotations()
  File "/cluster/work/sachan/yifan/data/cg_reasoning/street/street-reasoning-main/utils/dataset_utils.py", line 77, in load_graph_annotations
    self.graph_annotations[split] = utils.load_from_jsonl_file(path)
  File "/cluster/work/sachan/yifan/data/cg_reasoning/street/street-reasoning-main/utils/utils.py", line 36, in load_from_jsonl_file
    return [json.loads(line) for line in fh.readlines() if line]
  File "/cluster/work/sachan/yifan/data/cg_reasoning/street/street-reasoning-main/utils/utils.py", line 36, in <listcomp>
    return [json.loads(line) for line in fh.readlines() if line]
  File "/cluster/project/sachan/yifan/softwares/anaconda/anaconda3/envs/llama/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cluster/project/sachan/yifan/softwares/anaconda/anaconda3/envs/llama/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cluster/project/sachan/yifan/softwares/anaconda/anaconda3/envs/llama/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

When I checked the file (e.g., data/aqua_rat/aqua_rat_graph_annotation_train.jsonl), it contains some invalid oid of Git-LFS.

version https://git-lfs.github.com/spec/v1
oid sha256:42de0109f6f0efdd813f18a018b8eba747483f0b07874f6b9b11fbae8db985cd
size 245085

Could you take a look?

Best,
Yifan

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.