Exit status can’t be used to distinguish between tests not passing and other errors about smalltalkci HOT 3 OPEN

Good suggestion! Are you aware of any testing framework encoding failure kinds with exit codes? It'd be nice if we could just follow what others have worked out already 🙂

from smalltalkci.

I’m not aware of any convention for testing frameworks in particular. More generally, one convention is to use 1 through n for warnings, and anything above n for errors, another is to use 1 for an unspecified failure and anything above for specific failures. Following the first, the status of ‘smalltalkci’ could be 1 to warn that the tests did not pass, and following the second, the status could be 2 to specify its failure is due to the tests not passing.

I’m not sure whether it’d be useful for the status to also distinguish between the tests not passing due to failures, errors, unexpected passes or some combination of these, and whether it would make sense to also use a non-zero status to warn that the tests passed but that some of them were skipped or declared as an expected failure. The latter would probably have to be introduced as an option to ‘smalltalkci’ to not break existing uses. The status could be say 6 or greater if there was an error in ‘smalltalkci’ itself, else 5 if some tests had an error, else 4 if some tests failed, else 3 if some unexpectedly passed, else 2 if some were declared as expected failures, else 1 if some were skipped, else 0.

from smalltalkci.

Yes, something like this sounds reasonable. Maybe we leave some sort of gap between the codes, so that we can add more in the future, but that's something we can discuss in the actual implementation. I'm afraid I don't have any time to work on it myself, but I can help review if someone is willing to take a look.

Fun fact: AmigaOS uses these four exit code:

OK 0
WARN 5
ERROR 10
FAILURE 20

from smalltalkci.