Flaky Generator and Statistics Tests about mathtools HOT 2 CLOSED

This Test assertion failed recently: mathtools.generators.elements.ints.IntRangeElementGeneratedValuesTest.testRangeSize16(IntRangeElementGeneratedValuesTest.java:63)

The "Range Size 16" Test Case creates an IntRangeElement with an independent startValue, and dependent endValue that is 15 greater than startValue, such that there are 16 possible outcomes.

The input that fails is the one with input Parameter of Integer.MAX_VALUE (2147483632).

Aside Note: The duration of this test appears to vary greatly with input startValue, for this particular run, there is a range of 2 - 220 milliseconds.

The failed assertion is on the standard deviation of the generated values. There is an expected value of 10, and a tolerance of 4.5. These values were not determined mathematically, but through observation of many test runs.

This assertion has produced a high rate of failure following upgrade to Kotlin 1.8.0.

This is an indication that the underlying implementation of Random has changed.

from mathtools.

Consider that as long as a random number source is used in these tests, the standard deviation will fluctuate.

Recently, in #135 some tests were modified to check that the standard deviation is within some expected range without using an assertion. These tests will print to standard output when the standard deviation is outside of the expected limits. This should reduce the amount of interruption caused by these failing tests, without removing the ability to monitor changes in frequency of random outliers.

from mathtools.