Strong machine dependence of spey results (simplified likelihoods) about spey HOT 11 OPEN

Thanks @WolfgangWaltenberger; in the future, can you also report the machines that you observe the difference so that I can try to find the exact machine configuration? Note that if I don't have access to the machine I won't be able to replicate the issue :/ This particular issue seems to be related to machine precision and optimiser performance tagged to it. Is this solved for other Scipy versions?

Also, this might be due to Python 3.11, I haven't tested Spey with 3.11 yet.

The ultimate solution may be adding an iminuit optimiser, but that will take some time to implement.

from spey.

Hey Jack, I can try and give you all that info. That said, statements "if floating_variable != specific_value:" are clearly, undoubtedly, horribly wrong. In this case, this is the culprit. It seems a given that it is not python 3.11, or a specific version of scipy or any of this. I can give you all the version numbers, but are you sure this is helping in any way? All digits of 64-bit floats after the 16th are a lottery. There will be other machine dependent results coming up that require all these versions, don't you worry ;)

from spey.

Hi @WolfgangWaltenberger, if your solution solves it for all the machines I can definitely add it or maybe add a clip on the mu value? Let me know if there is a one-fits-all type of solution and I'll update the release ASAP.

from spey.

All that said:
*) ubuntu 23.10, python3.11, scipy 1.10.0, numpy 1.26.2, it so happens that right now i get: oUL=0, eUL=inf. Could be different next time, on an ARM computer, etc etc etc.
*) on a centos 7, python3.10, scipy version 1.10.0, numpy 1.26.1, i currently get oUL=0.2842, eUL=0.3516, the right answers.
I do recall even different answers, but thats enough for now i guess.

from spey.

How about

sigma_mu = self.sigma_mu(muhat, expected=expected) if not np.isclose(muhat, 0.0) else 1.0

from spey.

Hi @WolfgangWaltenberger, I also implemented the update within the issue23 branch and attached a PR to it. On my machine, your example works perfectly without any interruption, and I get the following result:

spey oUL(mu)=0.0016
spey eUL(mu)=13.3741

Can you please confirm this on your side as well?

from spey.

This is a realistic example, the data are from CMS-SUS-20-004, testing a very average simplified model point. So, eUL being 4 orders of magnitude bigger than oUL is absolutely impossible. The values I gave above were realistic. I will try to
illustrate with a plot.

from spey.

This is a realistic example, the data are from CMS-SUS-20-004, testing a very average simplified model point. So, eUL being 4 orders of magnitude bigger than oUL is absolutely impossible. The values I gave above were realistic. I will try to illustrate with a plot.

Maybe the issue is originating from an earlier step? perhaps, $\hat{\mu}$, max llhd or asimov data?

from spey.

Now I am the one who cannot reproduce. Using your patch I get on one machine
oUL(mu)=0.2842, eUL(mu)=0.3516 (thats sensible)
on another i get
oUL(mu)=0, eUL(mu)=0.9062.
pretty random. I tried to visualise nll(mu) in a sensible range, e.g. (-.5, 2.) but i get incompatible nonsense

from spey.

HI @WolfgangWaltenberger, I need a more specific issue. I compared the tool with the previous application (including Bill's version) and got the same result. If you can point out what needs to be changed, I can implement it accordingly. Otherwise, I'm afraid I don't know how to help here.

from spey.

@WolfgangWaltenberger is there any update on this? or is it solved?

from spey.