thoward27 / grade Goto Github PK

Blocked on: actions/setup-python#30

Describe the bug
The leaderboard's default score is None, it should be whatever value ensures it shows up in the last place.

To Reproduce
Steps to reproduce the behavior:

1. Create a leaderboard question
2. Student's code doesn't complete the test
3. Default value of leaderboardScore may place it at the top of the leaderboard

Expected behavior
Leaderboard default values should put students in last place.

Is your feature request related to a problem? Please describe.
When a test case fails currently, the reason for the assertion failure is printed in full, along with other potentially confusing information for students.

Describe the solution you'd like
It would be great to have a way to specify an error message for each assert, and have that error message be the only thing that gets output if the assert fails.

Describe alternatives you've considered
A function-wide error message decorator could work, but would lack the nuance of being able to give the student what their output was, or having different messages for asserts for different incorrect outputs.

Blocked on: readthedocs/readthedocs.org#6324

Have to skip valgrind assertions until it is resolved.

Blocked on: actions/runner-images#59

Need to update sphinx to include the new directory structure.

Describe the bug
The page for the grade pipeline documentation is empty.

To Reproduce
Steps to reproduce the behavior:

1. Go to https://grade.readthedocs.io/en/latest/source/pipeline.html
2. See error

Expected behavior
The documentation for the pipeline functions should be there.

Environment (please complete the following information):

• OS: [e.g. iOS] Windows 10

Describe the bug
When using the '@visibility' decorator for an individual test case function, nothing is actually changed. The visibility for the whole assignment is still 'visible', while the visibility value for the individual test cases is not output as a part of the JSON file. The attribute appears to be set correctly, but just not included in the JSON output.

To Reproduce
Steps to reproduce the behavior:

1. Decorate a test function with '@visibility('after_due_date)'
2. Call grade run and grade report
3. Look at JSON output for individual test cases
4. See error

Expected behavior
The visibility of an individual test case should show up in the list of properties for that test case in the JSON file.

Describe the bug
When using PartialCredit on a list of pipelines and a list of values, it raises an error upon calling the 'score' function. This appears to be due to this code from partialcredit.py:

line 31: self.max_score = value
and
line 41: return min(self.max_score, round(self._score, 2))

Since value can be an int or a list inside init, max_score can be set to a list, and an error pops up when trying to compare that list to the proper score.

Is your feature request related to a problem? Please describe.
The names of each test case in grade's output is just the class name, function name, and function docstring, instead of being user-defined.

Describe the solution you'd like
Potentially a name decorator that will override the default name if one exists for a test function.

Describe alternatives you've considered
The docstring for each test function could also be used by itself, but this wouldn't allow more descriptive test docstrings, forcing them to be solely names.

Describe the bug
The json output from grade report is formatted without any newlines.

To Reproduce
Steps to reproduce the behavior:

1. Run an autograder
2. Run grade report --format json

Expected behavior
The json output has newlines to easily read the output if needed.

The documentation leaves a lot to be desired. Here are some things that need to be done.

• Sphinx Integration
• README updates
• Github Pages to serve Sphinx Docs

Currently, unlike the standard Gradescope autograder, when a student's code fails to compile, nothing informative is shown (not even to the instructor). Ideally, the compilation failure (stderr) could be captured and shown (possibly even to the student) as the test output for the compilation check. This would cut down on TA support time, as often it's something as simple as the student forgot their compile script.

The Results class should be able to produce valid Markdown grade reports. Similarly to JSON, it should return a string, and be a property of the results class.

I have some work I can dredge up on this from a project a while back, I'll work to integrate it in and we can move from there.