Computation of the accuracy scores when there are compilation and runtime errors about apps HOT 7 CLOSED

Thanks for catching the bug. Think we forgot to add a "> 0" on line 29.
I'll push the big fix soon.

As for your other question. I thought it was a good metric that ultimately wasn't used to be able to measure how often does the model even produce code that runs compared to gibberish. So compile errors were worse in my opinion than runtime errors. It can also be seen as measuring grammar vs semantic errors in a loose way.

from apps.

BTW APPS is now available on Hugging Face hub https://huggingface.co/datasets/codeparrot/apps and we're currently adding the evaluation metric

from apps.

I'll make the changes you suggested but also feel free to make a pull requests too. Thanks for looking through it and adding it to hugging face!

from apps.

Great I'll open a PR! I saw that you already changed it thanks!

from apps.

Okay I think now with the examples and documentation it is working correctly and as intended. So I think this issue is good to close now. Feel free to reopen if there's something that was missed.

from apps.

Also for the comment regarding the expressions. The following should work provided they're numpy arrays:

import numpy as np
a = [-2, -1, 0, 1, -2]
a[a==-2]  # outputs  -2 which is not what we expect

b = np.asarray(a)
b==-2  # outputs array([ True, False, False, False,  True])
# Then the line below returns the following, which is what we expect. The length of which is 2.
b[b==-2]   # outputs array([-2, -2]) 

from apps.

Thank you for your reply and for the fix! Regarding the comment above tmp_results is defined as a list in the function, maybe we could add res.extend(np.array(results[index])) here

from apps.