unit tests sometimes have sporadic failures about radamsa HOT 4 CLOSED

Hi,

Great! Build issues are very welcome. I'll fix soonish when I have spare time. My *BSD buildbots are currently offline, so might be that I haven't noticed some issue on BSDish platforms.

There have been many issues with unit tests on OSX due to minor differences in arguments etc. Might make sense to use the version of owl used for building also for sorting and echoing in tests.

from radamsa.

So the behavior is intended, because radamsa is supposed to pad the input with a low probability with random data, if the input is very short. This is done to improve coverage of test with tiny fixed inputs. The only issues is with OSX sort getting confused by non-textual data. This does not matter, because the tests are probabilistic and are expected to fail on occasion, in which case they are tried again many times before tests/run considers them to fail. Sort stderr is now piped to /dev/null, so it doesn't get in the way.

Does the build work otherwise on OSX?

from radamsa.

The only issues is with OSX sort getting confused by non-textual data.

Oh, I did not realize that the output was not representative of a failure. Sorry for the false alarm on those tests.

I've modified ab.sh as follows to determine why it is sometimes failing:

# check bad string insertion happens as intended (more likely within quoted area)
mkdir -p tmp

echo '-----------------------------------------------------------------""---------------------------------------------------------------------------' \
   | $@ -m ab -p od -n 20 > tmp/ab.sh.tmp
cat tmp/ab.sh.tmp | grep -q '^-*\".*%.*\"-*$'
rc=$?
if [ $rc -ne 0 ]; then
   echo "Unexpected output:"
   cat tmp/ab.sh.tmp
   exit 1
fi

Attached is one the output from one such test run which failed.

ab.sh.tmp

This does not matter, because the tests are probabilistic and are expected to fail on occasion, in which case they are tried again many times before tests/run considers them to fail.

When building Radamsa the unit tests automatically run. As unit tests may sporadically fail, could the unit tests be sectioned into their own make target (e.g. "make check") so that they do not run automatically? Otherwise, one may build Radamsa, see a failure, and wonder if the sporadic failure indicates an issue with Radamsa on one's platform.

Sure, I can understand that Radamsa, being probabilistic, may not always fuzz as expected. However, is it possible to modify the unit tests which may fail with some probability so that the tests will be more deterministic?

Does the build work otherwise on OSX?

Seems to work so far. I've not tried the TCP stuff yet, which looks interesting.

from radamsa.

Makes sense. There is now a separate test build target, which runs the tests if necessary. I also added a thanks-section to readme.md.

from radamsa.