An error is reported when running eval about groundinglmm HOT 5 CLOSED

{'testlen': 0, 'reflen': 0, 'guess': [0, 0, 0, 0], 'correct': [0, 0, 0, 0]}
ratio: 1e-06
Bleu_1: 0.000
Bleu_2: 0.000
Bleu_3: 0.000
Bleu_4: 0.000
computing METEOR score...
Traceback (most recent call last):
File "/home/zhangc/groundingLMM/eval/gcg/evaluate.py", line 285, in
main()
File "/home/zhangc/groundingLMM/eval/gcg/evaluate.py", line 269, in main
coco_eval.evaluate()
File "/home/zhangc/miniconda3/envs/glamm/lib/python3.10/site-packages/pycocoevalcap/eval.py", line 57, in evaluate
score, scores = scorer.compute_score(gts, res)
File "/home/zhangc/miniconda3/envs/glamm/lib/python3.10/site-packages/pycocoevalcap/meteor/meteor.py", line 41, in compute_score
self.meteor_p.stdin.flush()
BrokenPipeError: [Errno 32] Broken pipe

from groundinglmm.

The value returned in the tokenizer is none
/home/zhangc/miniconda3/envs/glamm/lib/python3.10/site-packages/pycocoevalcap/tokenizer/ptbtokenizer.py
p_tokenizer = subprocess.Popen(cmd, cwd=path_to_jar_dirname,
stdout=subprocess.PIPE)
print("111",p_tokenizer.communicate(input=sentences.rstrip()))
token_lines = p_tokenizer.communicate(input=sentences.rstrip())[0]
print("token_lines",token_lines)
token_lines = token_lines.decode()
lines = token_lines.split('\n')

from groundinglmm.

Hi @clevercaicai

Thank you for your interest in our work. I tried running the GCG evaluation again, however can't reproduce the error you are getting. I suspect there might something went wrong during inference. Can you please try to repeat the evaluation process following the official docs at run_evaluation.sh and see if the problem persists. Further,

1. After running inference using eval/gcg/infer.py, please make sure that the results directory has 8177 json files corresponding to each video.
2. It is good idea to analyze the contents of a few predictions to verify if the model is generating reasonable outputs. If not, there may be some issues with the checkpoints or the transformers version.

I hope it will help. Thank You.

from groundinglmm.

Thanks again for your answer, the problem may be with the subprocess library, which I solved by using os.system. Now all the indicators are calculated correctly.

from groundinglmm.

Thanks @clevercaicai for the update, do let me know if you have any further question or face any other issue. Thanks

from groundinglmm.