Use trust_remote_code for dataset load about bigcode-evaluation-harness HOT 3 CLOSED

if I'm not mistaken, the only dataset that has arbitrary code in the harness now and could need this flag in the future is MultiPL-E?

It's a bit different from models where users can pass any model they want, whereas we have a limited number of carefully selected tasks so I'm not sure we need the flag for all tasks (we can add it just for MultiPL-E when it's a requirement if the authors don't update their loading script by then).

from bigcode-evaluation-harness.

I haven't checked myself. And moving all datasets to parquet export and no builder script is the better idea anyway. Perhaps adding a notice to the new task template might be an easier "fix".

If there is just a single case it can by bypassed using the env flag much easier.

I additionally came across this with the task I am developing. But moving to parquet is on my to-do list either way.
Feel free to close this issue and reject the draft PR.

from bigcode-evaluation-harness.

sounds good, I also talked to MultiPL-E authors and they will remove the python code in the dataset so it wouldn't require the flag.

from bigcode-evaluation-harness.