gradescope / gradescope-utils Goto Github PK
View Code? Open in Web Editor NEWPython package for building Gradescope autograders
Home Page: https://gradescope-utils.readthedocs.io/en/latest/
License: MIT License
Python package for building Gradescope autograders
Home Page: https://gradescope-utils.readthedocs.io/en/latest/
License: MIT License
I added the leaderboard decorator to a function and used your sample code. But it always sets in to none
The code:
@leaderboard("high score1")
def test_leaderboard(self):
"""Sets a leaderboard value"""
set_leaderboard_value(random.randint(0, 10))
and the output is
"leaderboard": [
{
"name": "time",
"value": null,
"order": "asc"
},
{
"name": "high score1",
"value": null,
"order": "desc"
},
I am getting this error when I use partial_credit decorator
@partial_credit(10)\nNameError: name 'partial_credit' is not defined\n\n"
I use it like this:
@partial_credit(10)
def test_exercise_1(self, set_score=None):
"""Exercise 1 answers"""
flag = 0
...
if(flag == 0):
set_score((8-flag)*/8)
return 1
Where am I going wrong?
I realize this could be submitted as a pull request.
FILE: gradescope-utils/gradescope_utils/autograder_utils/files.py:
ISSUE: SUBMISSION_BASE should be passed in as an default argument to allow local testing.
SUBMISSION_BASE = '/autograder/submission'
def check_submitted_files(paths, base=SUBMISSION_BASE):
...
target_path = os.path.join(base, path)
...
https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests
Makes it easier to script parametrized tests.
One of the issues with Python testing, is that it's easy for students to read directories and figure out what files to potentially read. Although one could argue that a student who has figured this out probably should deserve to pass the tests. However, this kind of information then spreads.
There's many ways we mitigate this but all of them involve a lot of jumping through hoops (dynamically renaming the tests, removing the test files after they are loaded but before the students code is loaded), etc (which only work if in a sandboxed environment).
A simple and effective way to prevent this would be to allow the initialization of JSONTestRunner to take a parameter(s) to limit how much stdout and stderr is written out in the results. That's the most likely way for students to get information back. If both stdout and stderr are truncated to a specified maximum amount, then it allows both the framework to print relevant messages, but also keeps students from reading the test files.
Another option would be allow tests cases to write to a logfile that is then sent back to stdout first and anything else truncated.
Would there be a way to set a default visibility other than visible
for all tests?
Right now I didn't find any, and adding all the @visibility("after_published")
decorators is quite annoying.
Thank you very much !
When a student submits code that breaks a test but it's not due to an assertion error, the feedback is either confusing or misleading.
For example, if a student has a simple KeyError mistake
def my_code(dataframe):
return dataframe['SalesPerson'] # assume SalesPerson is not a valid key
The test will fail but with the message:
"Test Failed: 'SalesPerson'
That really offers no help
Describe the solution you'd like
An easy solution is to provide the class of the exception or error:
"Test Failed: <class 'KeyError'> 'SalesPerson'\n"
The fix is simply in json_test_runner.py (buildResult)
line 77
if err:
if err[0] is AssertionError:
output += "Test Failed: {0}\n".format(err[1])
else:
output += "Test Failed: {0} {1}\n".format(err[0], err[1])
# note that isinstance will NOT work here
Describe alternatives you've considered
A similar pull request
#18
However with that solution you get the following:
"Test Failed: 'SalesPerson'\nhashtable_class_helper.pxi:pandas._libs.hashtable.PyObjectHashTable.get_item line 1627: \n"
In this context, the backtrace is not helpful to the student. Perhaps a config option?
Also, in that pull request it should not use isinstance()
to test the error class.
(tested in python3.6).
Additional context
Describe the bug
Raising an Assertion Error in unittest's setUpClass method causes exception in jsontestrunner's getLeaderboardData
To Reproduce
import unittest
from gradescope_utils.autograder_utils.decorators import (
weight,
number,
leaderboard,
)
class TestAutograder(unittest.TestCase):
@classmethod
def setUpClass(cls) -> None:
raise AssertionError("compile for all tests failed")
return super().setUpClass()
@number(1.1)
@weight(1)
def test_01(self):
"Test 01"
pass
Expected behavior
Raised error would have populated results.json for all test cases of that class.
Instead, received following error:
The autograder produced the following output:
Traceback (most recent call last):
File "/usr/lib/python3.10/unittest/suite.py", line 166, in _handleClassSetUp
setUpClass()
File "/autograder/source/tests/test.py", line 14, in setUpClass
raise AssertionError("compile for all tests failed")
AssertionError: compile for all tests failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/autograder/source/run_tests.py", line 8, in <module>
JSONTestRunner(visibility='visible',stdout_visibility='visible').run(suite)
File "/usr/local/lib/python3.10/dist-packages/gradescope_utils/autograder_utils/json_test_runner.py", line 196, in run
test(result)
File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__
return self.run(*args, **kwds)
File "/usr/lib/python3.10/unittest/suite.py", line 122, in run
test(result)
File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__
return self.run(*args, **kwds)
File "/usr/lib/python3.10/unittest/suite.py", line 122, in run
test(result)
File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__
return self.run(*args, **kwds)
File "/usr/lib/python3.10/unittest/suite.py", line 114, in run
self._handleClassSetUp(test, result)
File "/usr/lib/python3.10/unittest/suite.py", line 176, in _handleClassSetUp
self._createClassOrModuleLevelException(result, e,
File "/usr/lib/python3.10/unittest/suite.py", line 236, in _createClassOrModuleLevelException
self._addClassOrModuleLevelException(result, exc, errorName, info)
File "/usr/lib/python3.10/unittest/suite.py", line 246, in _addClassOrModuleLevelException
result.addError(error, sys.exc_info())
File "/usr/local/lib/python3.10/dist-packages/gradescope_utils/autograder_utils/json_test_runner.py", line 136, in addError
self.processResult(test, err)
File "/usr/local/lib/python3.10/dist-packages/gradescope_utils/autograder_utils/json_test_runner.py", line 123, in processResult
if self.getLeaderboardData(test)[0]:
File "/usr/local/lib/python3.10/dist-packages/gradescope_utils/autograder_utils/json_test_runner.py", line 51, in getLeaderboardData
column_name = getattr(getattr(test, test._testMethodName), '__leaderboard_column__', None)
AttributeError: '_ErrorHolder' object has no attribute '_testMethodName'
Describe the bug
When using the JSONTestRunner
class to obtain the test results in a JSON format, it throws an error when the input parameter buffer=False
if any of the test cases fails.
To Reproduce
We create a new testcase that fails.
import unittest
from gradescope_utils.autograder_utils.decorators import weight, visibility
class TestAutograder(unittest.TestCase):
@weight(1)
@visibility("visible")
def test_auto(self):
"""
A test that throws an error
"""
self.assertTrue(False)
Running the above with the JSONTestRunner
provided in gradescope_utils produces the following error:
TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str'
Expected behavior
As mentioned in line 132
of the file json_test_runner.py
, we Set buffer to True to include test output in JSON
. So ideally, when it is False
, there should be an empty output for the test.
Potential causes/fix
When buffer
is set to False
, the getOutput
method in JSONTestResult
returns None
. As a result, in the method buildResult
, the variable output
gets set to None
as well. In line 77
, if there is an error, the method tries to append the message Test Failed
to the output, but since output
is None
, it is unable to execute the line output += "Test Failed: {0}\n".format(err[1])
and hence throws an error. To fix this, a check needs to be put on output
(to see if it is not None
) before appending the Test failed
message onto it.
This occurs when it is used alongside the weight decorator.
It would be awesome if y'all would be willing to put a license on this code. Preferably MIT or Apache, but any clarity would be helpful.
Description
Naming test cases with a new-line character on the first line causes the name of the function to be placed in the results.json
file instead of the human-readable name in the docstring of the testcase.
To Reproduce
Create a testcase and name it with a docstring that has a new-line character on the first line. This file is called tests.py
.
import unittest
from gradescope_utils.autograder_utils.decorators import weight, number
class TestAutograder(unittest.TestCase):
@weight(1.0)
@number("1")
def test_prob1(self):
"""
A human-readable name.
"""
pass
Running the above with the JSONTestRunner
provided in gradescope_utils
produces the following results.json:
{
"tests": [
{
"name": "test_prob1 (tests.TestAutograder)",
"score": 1.0,
"max_score": 1.0,
"number": "1"
},
],
"leaderboard": [],
"execution_time": "0.00",
"score": 10.0
}
Expected behavior
Should have produced the following results.json
file:
{
"tests": [
{
"name": "A human-readable name.",
"score": 1.0,
"max_score": 1.0,
"number": "1"
},
],
"leaderboard": [],
"execution_time": "0.00",
"score": 10.0
}
Additional context
Removing the new-line character seems to fix the issue. I.e. the following produces the expected result.
import unittest
from gradescope_utils.autograder_utils.decorators import weight, number
class TestAutograder(unittest.TestCase):
@weight(1.0)
@number("1")
def test_prob1(self):
"""A human-readable name."""
pass
This also seems to only be the case inside of Gradescope's docker containers used for actually grading student submissions. This did not happen when I ran the code on my macbook.
Is your feature request related to a problem? Please describe.
gradescope-utils
does not produce output for subtests
(for definition of subtest, see https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests)
Describe the solution you'd like
Output a separate json object for each subtest, so that each subtest appears in the json output just like each test.
Describe alternatives you've considered
Additional context
gradescope-utils is a nice, useful package, thanks for developing it :)
Hi, I tried to run python setup.sh and I got this :
Could not open requirements file: [Errno 2] No such file or directory: '/autograder/source/requirements.txt'
any ideas?
In json_test_runner.py
, the buffer
argument is not being passed to the TestResult
class because it is not an argument in JSONTestResult
and therefore not passed to the super().__init__
. This means that the buffer
value for JSONTestRunner
is not respected.
The correct change is to add buffer
to the arguments of JSONTestResult.__init__
and pass it to TestResult
in the super().__init__
function
Is your feature request related to a problem? Please describe.
Currently there is no way to have a test without an assigned score. A weight of 0 is assigned by default.
Describe the solution you'd like
No score should be assigned if the weight is not provided.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.