Java/Javascript split questions about gorilla HOT 2 OPEN

Hi @vandyxiaowei,

Regarding your first question:
Good point. We will display the detailed score breakdown of the AST Simple category in the next website release (likely next week on May 16/17).

Regarding your second question:
Yes, we are expecting values in the format of new ArrayList; it needs to be valid Java syntax, and a JSON list [] would not be correct.
For Java (and JavaScript as well), before querying the model, we do some pre-processing on the prompt and function document. Specifically, at the end of the prompt, we will explicitly state that the provided function is in Java 8/JavaScript/Python syntax. And for parameter types that are not native to JSON, we will change their type to String (since String is JSON compatible) and add in the parameter description that "This is Java/Javascript" + {original_type} + " in string representation."
So in the example you provided above, when expecting type ArrayList, model will get the instruction that this is a String type parameter with the parameter description containing "This is Java ArrayList in string representation.", and thus the model should output the value in String format (eg, "new ArrayList<>(Arrays.asList(10, 20, 30))"), which is JSON compatible.
The relevant code that handles the pre-processing is here.

Let me know if you have more questions!

from gorilla.

Hi~，can we show the detailed scores for Java/Javascript splits on the leaderboard (instead of the averaged Simple function)?
Thanks!

from gorilla.