Automatic Programming Assignments Grader

Usage of main.go:
  -schema string
        Marking scheme to follow when grading the assignment (required)

Testing & Benchmarking

To run the test suite, simply execute the following command:

go test -v -cover ./...

To run the benchmarks, simply execute the following command:

go test -v -run="none" -bench=. -benchmem ./... -benchtime="5s"

Installation

First off, you'll need to Install Go and make sure enviornment variables are set up properly. Following that, you'll need to Install Docker. To be able to run Java Style checking test tasks, you'll need to install Checkstyle as well.

Running the following two commands install two Go-based projects that run the grading.

go get -u github.com/docker-exec/dexec
go get -u github.com/johnhany97/grader

Note: We use a modified version of dexec which supports timing out executions. Changes should've been merged, however, ensure you have version 1.0.9-SNAPSHOT of dexec installed before continuing.

That customized version could also be found at https://github.com/johnhany97/dexec

Supported languages

Following is a list of supported languages with the required docker image.

C: dexec/lang-c
Clojure: dexec/lang-clojure
CoffeeScript: dexec/lang-coffee
C++: dexec/lang-cpp
C#: dexec/lang-csharp
D: dexec/lang-d
Erlang: dexec/lang-erlang
F#: dexec/lang-fsharp
Go: dexec/lang-go
Groovy: dexec/lang-groovy
Haskell: dexec/lang-haskell
Java: dexec/lang-java AND johnhany97/grader-junit
Lisp: dexec/lang-lisp
Lua: dexec/lang-lua
JavaScript: dexec/lang-node
Nim: dexec/lang-nim
Objective C: dexec/lang-objc
OCaml: dexec/lang-ocaml
Perl 6: dexec/lang-perl6
Perl: dexec/lang-perl
PHP: dexec/lang-php
Python: dexec/lang-python
R: dexec/lang-r
Racket: dexec/lang-racket
Ruby: dexec/lang-ruby
Rust: dexec/lang-rust
Scala: dexec/lang-scala
Bash: dexec/lang-bash

Example usage

The following commands are to be ran from the project's root directory.

grader -schema="examples/inputOutput/javaSchema.json"

grader -schema="examples/python/pySchema.json"

grader -schema="examples/inputOutput/goSchema.json"

Schema Properties

file - Name of the file being graded, should contain it's extension - required
language - Name of the programming language in which the file being graded is written - optional
className - Class name of the file provided, usually is the name without the extension - required by junit and pyunit tests
folder - Folder within which the file provided is located - required
tests - Array of test tasks - required (even if just empty array)
outfile - Whether to store results in a file or just output it in the terminal - default = ''

Test Tasks

Currently supported are 4 different types of test tasks. They should confirm to the following format:

{
  "type": "TYPE_OF_TEST", // io - junit - pyunit - output
 
  // io --- start
  
  "input": [ " " ], // Array of strings where each string is a line in the input. They are joined by \n
  "expectedOutput": [ " " ], // Array of strings where each string is a line in the expected output. They are joined by \n
  
  // io --- end
 
  // junit/pyunit --- start
  
  "unitTest": "", // The test to be injected in the shell file, after being escaped
  
  // junit/pyunit --- end
  
  // output --- start
  
  "expectedOutput": [ " " ], // Array of strings where each string is a line in the expected output. They are joined by \n
  
  // output --- end
  
}

Example Schema

{
  "file": "Solution.java",
  "language": "java",
  "className": "Solution",
  "folder": "examples/inputOutput/",
  "tests": [
    {
      "type": "javaStyle"
    },
    {
      "type": "io",
      "description": "Testing normal boundaries",
      "input": [
        "3",
        "1",
        "2",
        "3"
      ],
      "expectedOutput": [
        "1",
        "2",
        "3"
      ]
    },
    {
      "type": "io",
      "input": [
        "2",
        "2",
        "5"
      ],
      "expectedOutput": [
        "2",
        "5"
      ]
    },
    {
      "type": "junit",
      "unitTest": "@Test\n  public void adderWorksWithZero() {\n    Solution s = new Solution();\n    int actual = s.adder(0, 3);\n    assertEquals(3, actual);\n  }"
    }
  ],
  "outfile": ""
}

Output: (Order may vary as tasks are finished by the job worker)

[  
   {  
      "test":{  
         "type":"javaStyle",
         "description":"",
         "input":null,
         "expectedOutput":null,
         "unitTest":""
      },
      "description":"Results generated by Checkstyle. A static analyzer built for analyzing Java Programs.",
      "stdOut":"Starting audit...\n[WARN] /Users/john/projects/dissertation/grader/examples/inputOutput/Solution.java:1: Using the '.*' form of import should be avoided - java.util.*. [AvoidStarImport]\n[WARN] /Users/john/projects/dissertation/grader/examples/inputOutput/Solution.java:4:3: Missing a Javadoc comment. [JavadocMethod]\n[WARN] /Users/john/projects/dissertation/grader/examples/inputOutput/Solution.java:16: Comment has incorrect indentation level 0, expected is 2, indentation should be the same level as line 18. [CommentsIndentation]\n[WARN] /Users/john/projects/dissertation/grader/examples/inputOutput/Solution.java:16: Line is longer than 100 characters (found 211). [LineLength]\nAudit done.",
      "stdErr":"",
      "successful":false,
      "similarity":0,
      "timedOut":false
   },
   {  
      "test":{  
         "type":"io",
         "description":"Testing normal boundaries",
         "input":[  
            "3",
            "1",
            "2",
            "3"
         ],
         "expectedOutput":[  
            "1",
            "2",
            "3"
         ],
         "unitTest":""
      },
      "description":"Expected:\n1\n2\n3\nGot:\n1\n2\n3\nTest passed: true",
      "stdOut":"1\n2\n3",
      "stdErr":"",
      "successful":true,
      "similarity":1,
      "timedOut":false
   },
   {  
      "test":{  
         "type":"io",
         "description":"",
         "input":[  
            "2",
            "2",
            "5"
         ],
         "expectedOutput":[  
            "2",
            "5"
         ],
         "unitTest":""
      },
      "description":"Expected:\n2\n5\nGot:\n2\n5\nTest passed: true",
      "stdOut":"2\n5",
      "stdErr":"",
      "successful":true,
      "similarity":1,
      "timedOut":false
   },
   {  
      "test":{  
         "type":"junit",
         "description":"",
         "input":null,
         "expectedOutput":null,
         "unitTest":"@Test\n  public void adderWorksWithZero() {\n    Solution s = new Solution();\n    int actual = s.adder(0, 3);\n    assertEquals(3, actual);\n  }"
      },
      "description":"",
      "stdOut":"JUnit version 4.10\r\n.\r\nTime: 0.009\r\n\r\nOK (1 test)",
      "stdErr":"",
      "successful":true,
      "similarity":0,
      "timedOut":false
   }
]

Test and Benchmark runs

Last Test run:

=== RUN   TestExecute
--- PASS: TestExecute (8.65s)
=== RUN   TestExecuteWithInput
--- PASS: TestExecuteWithInput (8.89s)
=== RUN   TestExecuteJUnitTests
--- PASS: TestExecuteJUnitTests (4.23s)
=== RUN   TestExecutePyUnitTests
--- PASS: TestExecutePyUnitTests (2.38s)
=== RUN   TestExecuteJavaStyle
--- PASS: TestExecuteJavaStyle (0.66s)
PASS
coverage: 77.9% of statements
ok      github.com/johnhany97/grader/processors 24.823s coverage: 77.9% of statements
=== RUN   TestRunTestInputOutputTestHandler
--- PASS: TestRunTestInputOutputTestHandler (28.10s)
    input_output_test.go:98: Successfully graded the solution for .java language
    input_output_test.go:98: Successfully graded the solution for .py language
    input_output_test.go:98: Successfully graded the solution for .cpp language
    input_output_test.go:98: Successfully graded the solution for .cs language
    input_output_test.go:98: Successfully graded the solution for .java language
    input_output_test.go:98: Successfully graded the solution for .py language
    input_output_test.go:98: Successfully graded the solution for .cpp language
    input_output_test.go:98: Successfully graded the solution for .cs language
    input_output_test.go:98: Successfully graded the solution for .java language
    input_output_test.go:98: Successfully graded the solution for .py language
    input_output_test.go:98: Successfully graded the solution for .cpp language
    input_output_test.go:98: Successfully graded the solution for .cs language
=== RUN   TestNewResultInputOutputTestHandler
--- PASS: TestNewResultInputOutputTestHandler (0.00s)
    input_output_test.go:187: Successfully obtained the expected result
    input_output_test.go:187: Successfully obtained the expected result
=== RUN   TestHandleErrInputOutputTestHandler
--- PASS: TestHandleErrInputOutputTestHandler (0.00s)
=== RUN   TestRunTestJavaStyleTestHandler
--- PASS: TestRunTestJavaStyleTestHandler (0.62s)
    java_style_test.go:49: Successfully style checked the solution for .java language
=== RUN   TestNewResultJavaStyleTestHandler
--- PASS: TestNewResultJavaStyleTestHandler (0.00s)
    java_style_test.go:119: Successfully obtained the expected result
    java_style_test.go:119: Successfully obtained the expected result
=== RUN   TestHandleErrJavaStyleTestHandler
--- PASS: TestHandleErrJavaStyleTestHandler (0.00s)
=== RUN   TestRunTestJUnitTestHandler
--- PASS: TestRunTestJUnitTestHandler (8.23s)
    junit_test.go:65: Successfully graded the solution for .java language
    junit_test.go:65: Successfully graded the solution for .java language
=== RUN   TestNewResultJUnitTestHandler
--- PASS: TestNewResultJUnitTestHandler (0.00s)
    junit_test.go:138: Successfully obtained the expected result
    junit_test.go:138: Successfully obtained the expected result
=== RUN   TestHandleErrJUnitTestHandler
--- PASS: TestHandleErrJUnitTestHandler (0.00s)
=== RUN   TestRunTestOutputTestHandler
--- PASS: TestRunTestOutputTestHandler (8.67s)
    output_test.go:66: Successfully graded the solution for .java language
    output_test.go:66: Successfully graded the solution for .py language
    output_test.go:66: Successfully graded the solution for .cpp language
    output_test.go:66: Successfully graded the solution for .cs language
=== RUN   TestNewResultOutputTestHandler
--- PASS: TestNewResultOutputTestHandler (0.00s)
    output_test.go:145: Successfully obtained the expected result
    output_test.go:145: Successfully obtained the expected result
=== RUN   TestHandleErrOutputTestHandler
--- PASS: TestHandleErrOutputTestHandler (0.00s)
=== RUN   TestRunTestPyUnitTestHandler
--- PASS: TestRunTestPyUnitTestHandler (3.36s)
    pyunit_test.go:54: Successfully graded the solution for .py language
    pyunit_test.go:54: Successfully graded the solution for .py language
=== RUN   TestNewResultPyUnitTestHandler
--- PASS: TestNewResultPyUnitTestHandler (0.00s)
    pyunit_test.go:109: Successfully obtained the expected result
    pyunit_test.go:109: Successfully obtained the expected result
=== RUN   TestHandleErrPyUnitTestHandler
--- PASS: TestHandleErrPyUnitTestHandler (0.00s)
PASS
coverage: 93.3% of statements
ok      github.com/johnhany97/grader/test       48.986s coverage: 93.3% of statements

Last Benchmark Run:

goos: darwin
goarch: amd64
pkg: github.com/johnhany97/grader/processors
BenchmarkExecute-12                            2        2687419688 ns/op           18880 B/op        177 allocs/op
BenchmarkExecuteWithInput-12                   2        2746329358 ns/op           19224 B/op        183 allocs/op
BenchmarkExecuteJUnitTests-12                  2        4290938234 ns/op           12016 B/op        100 allocs/op
BenchmarkExecutePyUnitTests-12                 5        1649888741 ns/op           18478 B/op        182 allocs/op
BenchmarkExecuteJavaStyle-12                  10         622639507 ns/op           13876 B/op        115 allocs/op
PASS
ok      github.com/johnhany97/grader/processors 53.897s
goos: darwin
goarch: amd64
pkg: github.com/johnhany97/grader/test
BenchmarkRunTestInputOutputTestHandler-12              2        2855917014 ns/op           20500 B/op        197 allocs/op
BenchmarkNewResultInputOutputTestHandler-12     20000000               430 ns/op              96 B/op          7 allocs/op
BenchmarkHandleErrInputOutputTestHandler-12     200000000               45.9 ns/op             0 B/op          0 allocs/op
BenchmarkRunTestJavaStyleTestHandler-12               10         664239069 ns/op           14096 B/op        117 allocs/op
BenchmarkNewResultJavaStyleTestHandler-12       200000000               44.0 ns/op             0 B/op          0 allocs/op
BenchmarkHandleErrJavaStyleTestHandler-12       200000000               46.9 ns/op             0 B/op          0 allocs/op
BenchmarkRunTestJUnitTestHandler-12                    2        4260429266 ns/op           13772 B/op        105 allocs/op
BenchmarkNewResultJUnitTestHandler-12           100000000               57.2 ns/op             0 B/op          0 allocs/op
BenchmarkHandleErrJUnitTestHandler-12           200000000               37.6 ns/op             0 B/op          0 allocs/op
BenchmarkRunTestOutputTestHandler-12                   2        2676039517 ns/op           20284 B/op        188 allocs/op
BenchmarkNewResultOutputTestHandler-12          20000000               399 ns/op              92 B/op          6 allocs/op
BenchmarkHandleErrOutputTestHandler-12          200000000               45.6 ns/op             0 B/op          0 allocs/op
BenchmarkNewResultPyUnitTestHandler-12          100000000               56.4 ns/op             0 B/op          0 allocs/op
BenchmarkHandleErrPyUnitTestHandler-12          200000000               36.7 ns/op             0 B/op          0 allocs/op
PASS
ok      github.com/johnhany97/grader/test       141.407s

johnhany97 / grader Goto Github PK

grader's Introduction

Automatic Programming Assignments Grader

Testing & Benchmarking

Installation

Supported languages

Example usage

Schema Properties

Test Tasks

Example Schema

Test and Benchmark runs

grader's People

Contributors

Stargazers

Watchers

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent