cm090 / gradescope-autograder Goto Github PK

Students end up running into similar problems with the autograder every quarter. Instead of a one-off post on Piazza, we can create a document with these common issues and ways to resolve them. This can be coupled with an autograder FAQ for students.

The autograder still fails to catch certain edge cases. One way to resolve this is by ensuring a results.json file is always generated in the correct directory. When this does not happen, use a template results file to inform the student of the error.

Right now, manual autograders check for file existence by looking at the exact filename. This does not allow for wildcards, causing problems if a student names their file different from what the instructions recommend. Students receive false negatives on the missing files check even though they submitted all the required files. For this feature, you will update both manual templates to support wildcards in the format *.java, *.png, and so on. You can expect each wildcard begins with a single star, followed by a dot and the file extension. When this is encountered, the autograder will search for files in the submission directory that end with the specified file extension. If at least one is found, the check passes. Otherwise, alert the student of a missing file.

This feature is partially implemented on the HWQuizQuestions autograder, which you can download here. The goal is to add support for any wildcard provided in the config.json file list.

If you have time, have the wildcard search ignore other files from the configuration. For example, if the files list includes MyClass.java and *.java, the autograder will ignore MyClass.java when searching for matches.

All of the autograder templates have a config.json file that provides a list of options for the creator. We want to update the builder tool to dynamically read these configuration options, infer datatypes from default values, and display a list of inputs to the user. You can either expand the current JSwing project or rewrite the tool in a Java GUI library of your choice.

A few requirements:

• Must be designed with cross-platform capabilities in mind. Bundle the final codebase into a runnable .jar file.
• The user will first provide the folder containing one of the four templates. For now, you can assume they'll come from the existing repo structure.
• If the template runs unit tests, the user will then provide the Java project folder. They will then select which test packages to include (assuming the package name includes test) and the grading weights.
• Look into a way for the user to add non-test files to the autograder. The 220 LinkedList homework comes to mind, where we don't expect students to upload the SLL interface file themselves.
• After selecting directories and files, the user should be prompted to enter additional configuration information based on the template's config.json file. We can pull the list of packages from the previous screen, so don't prompt the user for those.
• Once everything is submitted, the program will copy the template and selected Java project files to a new user-specified folder. The configuration file will be updated and everything will be compressed into an autograder.zip file. A new file explorer window will open to the export directory.
• Keep basic documentation in mind while working. This will make it easier for us to create tutorials in the future. Think high-level steps for running the program.

Right now, students must wait for the autograder to finish before receiving feedback on their submission. We want to find a way to notify them of failed builds if they leave the page early. Our idea is as follows:

• If the autograder fails (output.json includes ERROR: Grading Failed) send an email to the student.
• The autograder includes a metadata file with the student's email, dropbox name, project ID, and submission ID.
• Automatically send an email to the student informing them of the failed submission. Provide the name of the dropbox and a link to resubmit.
• We do not want to use a personal email address (for security reasons), so look into a program or library that allows us to send emails without credentials or from a repurposed account (specific to this use case).

The manual exam autograder includes a feature that tells instructors and TAs the percentage of students who completed the exam. We want to expand these statistics to include these categories:

• Submitted the exam with all files
• Submitted the exam with a failure (missing files)
• Started the exam but did not submit

This can be accomplished with hidden grading items, where students receive a 1/1 if the condition is met and a 0/1 otherwise. We would also like to expand this functionality to the autograded exam template, tracking the same conditions.

The autograder only looks for files in the root submission directory. This works well in most instances but causes lookups to fail if a student uploads a directory with all their files inside. We want to recursively search for files in all the autograder templates, performing the following actions based on template:

• For manual grading, leave the files as-is. Make comparisons on the files themselves and recurse inside of directories if found.
• For automated grading, bring all .java files to the root of the submissions directory. Then use the existing code to move appropriate files into the test runner.

Documentation already exists for creating/uploading autograders and grading submissions, but we can compile the most common questions into a single document. We can also provide links to other resources, such as videos.

Manual autograder templates currently expect a predefined list of files to be uploaded where their names match exactly. This is a problem if we expect MyClass.java and a student uploads myclass.JAVA, for example. We want to compare files strictly in lowercase to prevent false negatives. Your implementation should be robust enough to eventually handle converting wildcards to lowercase without much modification.

A video exists, but we don't yet have a written guide for the file tool. Walk through exporting an assignment from Gradescope and the tool itself.