At the moment, we place user.conf
under measurements/
e.g. closed/Intel/measurements/clx_9282-2s_openvino-linux/ssd-small/Server
. It can be argued, however, that it belongs under results
e.g. closed/Intel/results/clx_9282-2s_openvino-linux/ssd-small/Server
.
I don't think we should strictly require all performance and accuracy runs to happen with exactly the same LoadGen parameters. For example, the submitter may decide to do an accuracy Server run under a higher QPS than any of the corresponding performance runs to increase the throughput. (We don't measure the latency for accuracy runs, therefore latency constraints don't have to be obeyed.)
As another example, the submitter may have a bunch of VALID performance Server runs at slightly different QPS values e.g. [ 99.1, 99.2, 99.5, 99.3, 99.1 ]. In principle, they should have no problem to obtain 3 more VALID runs at QPS=99.1, but it would be just contributing to global warming. Instead, they should be allowed to submit these 5 runs and claim QPS=99.1 as achieved.
In such cases, the user.conf
files may be slightly different. It's probably undecidable which one should be stored under measurements
then. However, we won't be loosing any information if we allow to store them next to the LoadGen logs for each run.