dansteren / cs236-test-runner Goto Github PK

The $LAB variable is reported wrong when the user updates the script. Just print out the input instead of the actual variable. It can't update until the command has finished running.

The Problem

Running ./test -u on macs results in the following error:

Updating File Paths (leave field empty to skip)
    Lab Number: 2
sed: 1: "test": undefined label 'est'
    The variable $LAB now contains "2". Feel free to use it below...
    Path to input files (i.e. $LAB/input):
    Path to actual outputs (i.e. $LAB/actual):
    Path to expected outputs (i.e. $LAB/expect):
    Path to project files (i.e. ../):
    UPDATE COMPLETE!

Since I don't have a mac I haven't been able to fix this completely but there is a simple fix.

The Fix

Simply edit the test script itself. Line 2 of the file reads:

LAB="lab-1"

This is setting a variable that is referenced throughout the script. All you need to do is change the number. For example:

LAB="lab-2"

That's it! Save the file and you're good to go.

The script should check for pmccabe before trying to test code complexity. If it isn't installed, inform the user and possibly try to install it (after getting confirmation from the user).

Right now the user receives no feedback as to which lab it will test. It defaults to 3 but it doesn't inform the user at all. I think it should tell the user which test cases it is running.

So your test driver when it does the diff of the files it compares all whitespace - which because of Windows/unix and different ways that c++ outputs newlines can end up being different.

You can add the flag -Z on the diff to ignore whitespace at the end of the line. I've manually done this edit every time I download your test driver because I was getting this sort of misunderstanding.

Would you be willing to include the source code for this project so that I could fork and make some modifications?

BTW great idea for a project, and mostly great implementation :).

I've noticed a couple bugs that are kind of related to #5.

First of all, old files in actual should probably be deleted every time the test driver is run. If I have a working program, pass all the tests, then change something that stops my program from creating output files, the test driver will still tell me I passed everything, since the actual folder still has correct files from the previous run.

After encountering the first problem, I manually deleted all the files in the actual folder, then ran the test driver. It tells me I passed everything, despite creating no actual output files. [I'm guessing diff doesn't report any differences if it doesn't find one of the files] Here's the output:

Compiling project...

lab-1 test results:

diff: lab-1/actual/10: No such file or directory
Test 10......PASSED
diff: lab-1/actual/11: No such file or directory
Test 11......PASSED
diff: lab-1/actual/12: No such file or directory
Test 12......PASSED
diff: lab-1/actual/13: No such file or directory
Test 13......PASSED
diff: lab-1/actual/14: No such file or directory
Test 14......PASSED
diff: lab-1/actual/15: No such file or directory
Test 15......PASSED
diff: lab-1/actual/61: No such file or directory
Test 61......PASSED
diff: lab-1/actual/62: No such file or directory

Test 62......PASSED

pmccabe......PASSED

If you could check that output files exist and delete them before each run, it'd be awesome.

Problem: When compiling the program it can encounter errors and not fully compile. However, regardless of whether the program compiles or not it still tries to run the executable. The result is either a crashing script (if the user has never compiled before) or the program continues to run the test on the old executable.

Solution: The script should check for errors in the compilation and terminate the script without trying to run the executable.

Other Considerations: If the script only outputs warnings then it should be fine to continue.

Questions:

• Should there be an option to continue even if the compilation failed?
• Should there be a verbose option that prints out all compilation output to the screen?
• Should there be a silent option that turns off all output?
• After compilation should we print the results of the compilation?
• Should we ask the user if they want to proceed with the execution?

When I run the program it works, but gives the following message before each test runs:

diff: invalid option -- Z
diff: Try diff --help' for more information.`

Because it actually appears to be working, I'm guessing the problem is somewhere in the program, not in my code.

[update] because diff can't run, the program actually isn't working, but because diff doesn't return anything, it thinks it's working.

Allow user to update paths to the different directories.