[encoding,Â] top1000.dev.tsv about msmarco HOT 6 CLOSED

This has been fixed and uploaded for the top1000 dev and eval. The top1000 train and the train triples will follow shortly.

for context I used https://ftfy.readthedocs.io/en/latest/

import ftfy
import sys

filename = sys.argv[1]
with open(filename, 'r') as f:
with open('cleaned' + filename, 'w') as w:
for l in f:
w.write(ftfy.fix_text(l))

from msmarco.

@dfcf93

from msmarco.

Hey,

You are correct. We seem to have a bug in the topfiles. I will go ahead and update the files at some point but in the meantime each line also includes the QID and PID(Question ID) and (Passage ID) which you can join with the collection.tsv and the query files. Sorry for the delay.

from msmarco.

Hi,
And also found this problem in train.triples file, but not in collection. I think it's a encoding problem while parsing web document.
Thanks for the reply.

from msmarco.

Just not to leave you in the dark I am going to update all the documents to fix this issue I am just currently away on paternity leave. I should be able to get to it mid may

from msmarco.

Just not to leave you in the dark I am going to update all the documents to fix this issue I am just currently away on paternity leave. I should be able to get to it mid may

That's very kind of you.

from msmarco.