test runs with optional "real-time" flag enabled are failing intermittently about single-market-robot-simulator HOT 2 CLOSED

Dec 1 2019 Test Data

In December, 2019 I ran some repeated tests via the "npm test" command; about 7000 repetitions total spread across locally available and Google Cloud machines.

For the docker-container versions of single-market-robot-simulator, the total error rate from npm test is around 1.5% and the error rates on v4.3.0 and v5.6.0 are similar. On the n1-highcpu-2 Google instances, the "real time" error is almost all (90+%) of errors.

An error rate of 4.3% (22/511 cases) where 16 errors were related to the immediate deadline issue and only 6 were related to this issue. That was on an HP i5-3470S; a somewhat older, slower machine.

An error rate of under 0.5% (5/1252 cases and 0/262 cases) were observed with node 12.13.1 and node 12.13.0. These errors were also almost exclusively the "real time" issue.

Failure mode of real time issue

In the real time test, a market with one buyer and one seller, each using the ZI strategy, is set up for 1.5 seconds of real-time with each agent having a Poisson arrival rate of 10/sec (ideally).

The test requires that within ~15 bids and ~15 asks, the two agents trade, which should produce an unknown division of a known total profit and efficiency is always 100%. The tests check the number of trades, the division of profit, and the efficiency, among others... for 10 periods.

The pattern of failure is always that one (and only one) of the ten periods does not have a trade. but the tests check that there must be a trade.

back-of-the-envelope guess at random effects

In each test we run 10 periods three times. Only one of these has to fail, and typically only one does.

Randomly, it is possible that no trade occurs. If we assume that the probability of no trade in 15 bid ask pairs is on the order of 2^-15, then the probability of 1 in 30 failing is on the order of 0.09%, which is about an order of magnitude from what we are seeing. This calculation is wrong because with Poisson arrival, we are not guaranteed pairs (it could be bid,bid,bid,bid,ask,ask,...) and we are not guaranteed 15 bids and 15 asks.

What can make the probability of a period without trades increase?

• Anything that keeps ~15 bids and 15 asks from being delivered in the 1.5 sec allotted for the test.
• Strings of bids and asks instead of bid,ask,bid,ask.
• Note: the bid-ask improvement rule, is off for this test. so in a string of bids without an ask a lower bid can replace (unaccepted) a higher bid that might have triggered a trade

The code uses setInterval to create a "jump in simulation time" every 40 ms, these may or may not actually occur every 40 ms (see src/index.js function onRealTimeWake and surrounding code).
Whenever they occur, the events (agent bids, asks, etc) that should have occurred during the brief sleep interval are executed. If the time allotted for the period has elapsed, the code should still
run the events through the end of the period.

Simple fix: Increase the test to 5.0 seconds.

Increasing the test time to 5 sec shows promise. In 500 tests, this real time issue has not resurfaced.

from single-market-robot-simulator.

This test fix was first published in version 5.6.17

from single-market-robot-simulator.