cogcomp / multirc Goto Github PK

1、

those bundle can't find!
2、

this dev predicted file style is steal not the same as given example!

hi,

i train your LR model without changing anything, but i only get f1 of 0.61, how to run to get 0.66?

I think some "sentences_used" field is inaccurate or wrong. For example:

The article is:
"paragraph": {"text": "<b>Sent 0: </b>timothy likes to play sports .<br><b>Sent 1: </b>he spends his time after school playing basketball and baseball .<br><b>Sent 2: </b>sometimes timothy pretends he is a famous baseball pitcher for his favorite team with his friends .<br><b>Sent 3: </b>he plays with his friends mandy and andrew .<br><b>Sent 4: </b>timothy also plays pretend when he is alone .<br><b>Sent 5: </b>he has an imaginary friend named sean .<br><b>Sent 6: </b>sean is an elephant who watches television with timothy .<br><b>Sent 7: </b>mandy likes playing baseball but she also likes to paint .<br><b>Sent 8: </b>mandy 's favorite class at school is art .<br><b>Sent 9: </b>she likes making pictures of flowers .<br><b>Sent 10: </b>her teacher says she is a good artist .<br><b>Sent 11: </b>she painted a picture of a tree for her teacher .<br><b>Sent 12: </b>there were red and yellow leaves on it .<br><b>Sent 13: </b>it had apples on it .<br><b>Sent 14: </b>when andrew goes home after baseball , he likes to eat a snack .<br><b>Sent 15: </b>he eats carrots and bananas .<br><b>Sent 16: </b>if he is a good boy his mom , mrs. smith , sometimes gives him milk and cookies .<br><b>Sent 17: </b>afterwards , andrew finishes his homework .<br><b>Sent 18: </b>"

The question and answer is:

{"multisent": false, "question": "who does timothy play with ?", "sentences_used": [0, 1], "answers": [{"text": "timothy plays with mandy and andrew . timothy also plays with his imaginary friend sean", "isAnswer": true, "scores": {}}, {"text": "the famous baseball pitcher", "isAnswer": false, "scores": {}}, {"text": "mrs smith", "isAnswer": false, "scores": {}}, {"text": "timothy likes to play sports", "isAnswer": false, "scores": {}}, {"text": "basketball and baseball", "isAnswer": false, "scores": {}}], "idx": "18"}

We can see clearly that we cannot answer this question or ask this question just base on sentence 0 and sentence 1.

{"pid":"News/CNN/cnn-3b5bbf3ba31e4775140f05a8b59db55b22ee3e63.txt","qid":"0","scores":[1,0,0,1,0]},

different with below

{
"WH_dev_1639": "best",
"WH_dev_3121": "united kingdom",
"WH_dev_844": "horror",
"WH_dev_612": "canada",
"WH_dev_2211": "13",
"WH_dev_1956": "kuwait",
"WH_dev_2779": "rome",
"WH_dev_3432": "edwards air force base",
"WH_dev_2655": "rock",
"WH_dev_4437": "santa cruz",

What does the multisent property of a question mean exactly?

According to the paper, it has been verified that every question is multi-sentence, so shouldn't multisent always be true? In fact, the following question from the dev set clearly relies on multiple sentences, but is annotated as "multisent": false.

          {
            "question": "How many years after he entered the army did Cavour become prime minister?",
            "sentences_used": [0, 7],
            "multisent": false
          },

Context:
Cavour was a younger son of a noble Piedmontese family, and entered the army in 1826, serving in the engineers.
[...]
In 1850 he became minister of commerce; in 1852, prime minister. After that, his history is the history of Italy itself.

when i run run.sh, got this error

i run scripts of sun_surfaceLR.sh

@microth
Would be great if we can change the script a little bit such that it is more compatible with the setting we have in the CodaLab.

I think we need the following settings:

• training script: take the local train file and train on it.
• evaluation script: given the pre-trained model, load it and evaluate on a given input file. The following format would be ideal:

> script_name  <input-data-json-file> <output-prediction-json-path>