Adding dynesty support (especially to retrievals) about species HOT 8 CLOSED

I have restructured the AtmosphericRetrieval class in the last commit on the main branch (see b2adbcb).

In run_retrieval you can run the Dynesty sampler, which can use either prior_transform() and lnlike_func(), or lnprior_dynesty() and lnlike_dynesty(). The latter in case anything Dynesty-specific needs to be implemented with the prior/lnlike otherwise these can be removed.

Would this help with your Dynesty implementation?

I am not sure if lnlike_func/lnlike_dynesty can be pickled or if it needs to be moved outside of the class. In that last scenario it would be a bit more work to implement because the attributes should be converted into parameters. But hopefully it works like this 🤞.

from species.

I have set it up a bit differently! There is now the setup_retrieval method, and then separately run_multinest/run_dynesty. The latter will have only MultiNest/Dynesty parameters, so perhaps the kwargs_multinest is not needed. Sorry about that, but we can leave it for now and decide later if this new setup works.

from species.

Hi Tomas - I've implemented an initial run_dynesty on my fork's branch add_run_dynesty that is based on the backtracks implementation. That branch is up to date with commit 3d150aa.

I've tested the various permutations (dynamic bool, n_pool None/int, mpi_pool bool, resume bool) of the multiprocessing locally to confirm that they at least begin sampling and save the intermediate products. I have a few lines at the end that take the dynesty output results object and saves a retrieval_post_equal_weights.dat file that is equivalent to the output from pymultinest.

I'm going to try to test this on my cluster next, I couldn't do more than a few processes with mpi or a multiprocess Pool on my local machine due to memory constraints.

Let me know what you think, and if you'd like me to open a PR for that branch.

from species.

Awesome! Good to hear that it is working already!

Would be great if you could create a PR for that 👍

from species.

Thanks a lot William! Great that you are trying to implement this!

In FitModel, there was a similar issue from what I remember when adding run_ultranest. I had to make the embedded function for the likelihood an actual method of the class. Within run_multinest and run_ultranest, I then added a local function for the likelihood that calls the class method. This should indeed be possible here as well (I think), although I am not sure how straightforward. I can have a look so the code structure can be used for your implementation.

Would really help indeed if the opacity memory could be shared between cores, but I am not sufficiently experienced with multiprocessing to understand what is the best way to do that... Maybe Paul and Evert have put some thought into that?

from species.

Wow! Awesome stuff. I will pull soon and try out later this week ^_^ thanks for entertaining this implementation Tomas!

from species.

Thanks again for implementing the Dynesty support in AtmosphericRetrieval! Since I had been making some updates in FitModel, I have adapted run_dynesty in there, which was quite straightforward (see 1f02b5a).

from species.

Looks great! I had a commit I forgot to push to do something similar, though I hadn't yet documented it. I'll test on my end with some more AF Lep b fits. I think we can close the issue now :).

from species.