martijndwars / grading-tools Goto Github PK

View Code? Open in Web Editor NEW

0.0 0.0 0.0 167 KB

tool support for automatic grading

JavaScript 15.86% Java 84.14%

grading-tools's People

Contributors

Watchers

grading-tools's Issues

Empty .spt files

The test runner has a problem with empty .spt files. The parsing probably Stack trace:

Checkout student project in ./in4303-2265944356522279983
running reference language implementation
group Testing Syntax Analysis
running grammars/MiniJava-correct/MiniJava-correct.packed.esv
Resetting the directory monitor for full analysis
invalid test mainClass simple main class with print in suite /home/ubuntu/in4303-2265944356522279983/MiniJava-tests-syntax/syntax/mainClasses.spt
invalid test comment simple comment in suite /home/ubuntu/in4303-2265944356522279983/MiniJava-tests-syntax/syntax/comments.spt
invalid test comment simple comment no whitespace in suite /home/ubuntu/in4303-2265944356522279983/MiniJava-tests-syntax/syntax/comments.spt
invalid test comment comment block in suite /home/ubuntu/in4303-2265944356522279983/MiniJava-tests-syntax/syntax/comments.spt
invalid test comment comment block no whitespace in suite /home/ubuntu/in4303-2265944356522279983/MiniJava-tests-syntax/syntax/comments.spt
invalid test comment comment block with linebreaks in suite /home/ubuntu/in4303-2265944356522279983/MiniJava-tests-syntax/syntax/comments.spt
invalid test comment comment block with nested comments in suite /home/ubuntu/in4303-2265944356522279983/MiniJava-tests-syntax/syntax/comments.spt
invalid test comment comment block commented end in suite /home/ubuntu/in4303-2265944356522279983/MiniJava-tests-syntax/syntax/comments.spt
invalid test comment empty comment in suite /home/ubuntu/in4303-2265944356522279983/MiniJava-tests-syntax/syntax/comments.spt
invalid test comment comment block in between assignment in suite /home/ubuntu/in4303-2265944356522279983/MiniJava-tests-syntax/syntax/comments.spt
Cannot get Stratego interpreter, or Stratego invocation failed
org.metaborg.spoofax.core.SpoofaxException: Invoking Stratego function failed
    at org.metaborg.spoofax.core.stratego.StrategoRuntimeUtils.invoke(StrategoRuntimeUtils.java:21) ~[reporter-1.5.0-SNAPSHOT.jar:na]
    at org.metaborg.sunshine.services.pipelined.builders.BuilderSink.invoke(BuilderSink.java:105) [reporter-1.5.0-SNAPSHOT.jar:na]
    at org.metaborg.sunshine.services.pipelined.builders.BuilderSink.sink(BuilderSink.java:89) [reporter-1.5.0-SNAPSHOT.jar:na]
    at org.metaborg.sunshine.services.pipelined.builders.BuilderInputTermFactoryLink.sink(BuilderInputTermFactoryLink.java:77) [reporter-1.5.0-SNAPSHOT.jar:na]
    at org.metaborg.sunshine.pipeline.connectors.ALinkManyToMany.sink(ALinkManyToMany.java:29) [reporter-1.5.0-SNAPSHOT.jar:na]
    at org.metaborg.sunshine.pipeline.connectors.LinkMapperOneToOne.sink(LinkMapperOneToOne.java:46) [reporter-1.5.0-SNAPSHOT.jar:na]
    at org.metaborg.sunshine.pipeline.connectors.ALinkManyToMany.sink(ALinkManyToMany.java:29) [reporter-1.5.0-SNAPSHOT.jar:na]
    at org.metaborg.sunshine.services.filesource.FileSource.poke(FileSource.java:65) [reporter-1.5.0-SNAPSHOT.jar:na]
    at org.metaborg.sunshine.drivers.SunshineMainDriver.run(SunshineMainDriver.java:200) [reporter-1.5.0-SNAPSHOT.jar:na]
    at org.metaborg.spoofax.testrunner.core.TestRunner.run(TestRunner.java:55) [reporter-1.5.0-SNAPSHOT.jar:na]
    at nl.tudelft.in4303.grading.Grader.runTests(Grader.java:59) [reporter-1.5.0-SNAPSHOT.jar:na]
    at nl.tudelft.in4303.grading.tests.TestsGrader.runLanguages(TestsGrader.java:79) [reporter-1.5.0-SNAPSHOT.jar:na]
    at nl.tudelft.in4303.grading.tests.TestsGrader.grade(TestsGrader.java:43) [reporter-1.5.0-SNAPSHOT.jar:na]
    at nl.tudelft.in4303.grading.Grader.grade(Grader.java:53) [reporter-1.5.0-SNAPSHOT.jar:na]
    at nl.tudelft.in4303.grading.github.GitHubGrader.grade(GitHubGrader.java:146) [reporter-1.5.0-SNAPSHOT.jar:na]
    at nl.tudelft.in4303.grading.github.GitHubGrader.feedback(GitHubGrader.java:98) [reporter-1.5.0-SNAPSHOT.jar:na]
    at nl.tudelft.in4303.grading.Main.main(Main.java:32) [reporter-1.5.0-SNAPSHOT.jar:na]
Caused by: org.spoofax.interpreter.core.InterpreterException: Exception during evaluation
    at org.spoofax.interpreter.core.Interpreter.evaluate(Interpreter.java:117) ~[reporter-1.5.0-SNAPSHOT.jar:na]
    at org.spoofax.interpreter.core.Interpreter.invoke(Interpreter.java:82) ~[reporter-1.5.0-SNAPSHOT.jar:na]
    at org.strategoxt.HybridInterpreter.invoke(HybridInterpreter.java:442) ~[reporter-1.5.0-SNAPSHOT.jar:na]
    at org.metaborg.spoofax.core.stratego.StrategoRuntimeUtils.invoke(StrategoRuntimeUtils.java:15) ~[reporter-1.5.0-SNAPSHOT.jar:na]
    ... 16 common frames omitted
Caused by: org.spoofax.interpreter.core.InterpreterErrorExit: Internal error: with clause failed unexpectedly in 'make-dynamic-rules'
    at org.strategoxt.lang.InteropSDefT.evaluate(InteropSDefT.java:194) ~[reporter-1.5.0-SNAPSHOT.jar:na]
    at org.strategoxt.lang.InteropSDefT.evaluate(InteropSDefT.java:183) ~[reporter-1.5.0-SNAPSHOT.jar:na]
    at org.strategoxt.lang.InteropSDefT$StrategyBody.evaluate(InteropSDefT.java:245) ~[reporter-1.5.0-SNAPSHOT.jar:na]
    at org.spoofax.interpreter.core.Interpreter.evaluate(Interpreter.java:109) ~[reporter-1.5.0-SNAPSHOT.jar:na]
    ... 19 common frames omitted
Caused by: org.strategoxt.lang.StrategoErrorExit: Internal error: with clause failed unexpectedly in 'make-dynamic-rules'
    at org.strategoxt.lang.SRTS_EXT_fatal_err_0_3.invoke(SRTS_EXT_fatal_err_0_3.java:28) ~[reporter-1.5.0-SNAPSHOT.jar:na]
    at org.strategoxt.lang.compat.override.java_integration.SRTS_EXT_fatal_err_0_2.invoke(SRTS_EXT_fatal_err_0_2.java:33) ~[reporter-1.5.0-SNAPSHOT.jar:na]
    at org.strategoxt.lang.compat.override.java_integration.report_failure_0_2_override.invoke(report_failure_0_2_override.java:27) ~[reporter-1.5.0-SNAPSHOT.jar:na]
    at org.strategoxt.stratego_lib.report_with_failure_0_1.invoke(report_with_failure_0_1.java:32) ~[reporter-1.5.0-SNAPSHOT.jar:na]
    at trans.make_dynamic_rules_0_0.invoke(make_dynamic_rules_0_0.java:61) ~[na:na]
    at trans.test_runner_file_0_0.invoke(test_runner_file_0_0.java:50) ~[na:na]
    at org.strategoxt.lang.Strategy.invokeDynamic(Strategy.java:30) ~[reporter-1.5.0-SNAPSHOT.jar:na]
    at org.strategoxt.lang.InteropSDefT.evaluate(InteropSDefT.java:192) ~[reporter-1.5.0-SNAPSHOT.jar:na]
    ... 22 common frames omitted
language

Re-using same TestRunner for multiple languages fails

Currently, the same TestRunner instance is used for running multiple languages. That is, each language is registered with the same runner.

However, I created a minimal example with one test and two erroneous languages A, B. The test is meant to catch an error in language A. Now:

If I only run the reporter against language A, it gets full points.
If I first run it against A, then against B, I get both the points for A and B.
If I first run it against B, then against A, I get no points.

Jasmin dependency

Labs 10-12 rely on Jasmin (foreign call). Figure out a way to dynamically include Jasmin for these labs. E.g. put it in grading-releng and ship it by default.

Incorrect use of passed languages

The TestResult#finishedLanguage is called for every finished language. If the test detected the language, variable passed is incremented by 1. In the report, this variable is used in the following sentence:

of your valid tests detect erroneous language definitions

However, the finishedLanguage method is called for the correct implementation as well. If I have 1 test which passes the correct implementation but does not catch an erroneous language I get:

0 of your valid tests detect 1 erroneous language definitions

which does not make sense.

Proposed solution: check if description == null or points == 0 in #finishedLanguage as an indicator that this is a 'pass' for the reference implementation and should not increment the passed variable. I'm not sure how this will affect the rest of the system..

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.