Run simulations on cluster whilst using orderly about naomi-aghq HOT 2 CLOSED

Howes, Adam T
Hi Cluster,
I'd like to run a simulation study (simulating data, fitting models, analysing results). I have done this before setting up saving objects to file paths and so on and it became a mess that was difficult to trace back where things came from (so for example I was resorting to naming files after the date, which I don't think is a good idea). I've recently been using orderly in less computationally intensive work and it has really helped my workflow there. My question is: is there anyone who has used orderly for more computationally demanding tasks (e.g. simulations, using the cluster). If so, I'd love to chat to you / see an example of how you've set things up!
Caveats: could be the case that orderly just isn't the right tool for this / it's not designed for this. If so, is there another best practise I could use? Another possibility is that orderly and the cluster don't work nicely together.
Thanks,
Adam

Edit: chatting to Ashton, Robert about this now suggesting I use bundles

Question we're talking about if anyone has input:
A) 1 orderly report for each set of parameter settings, OR
B) 1 orderly report for all parameter settings (controlled inside report)
Another way of phrasing, control parameters with orderly's .yaml or control parameters with standard R inside report?

If it's reasonable / possible, I think A) is the "ideal" in that the reports would have full provenance for each simulation run (allowing answering questions like "something went wrong with the model run #465, what was the input data for that one" EDIT: actually you can have this with B) as well just looking at e.g. input_data[[465]]) as well as the reports being searchable with orderly syntax. Downside of A): is it manageable to 1000s (however many simulations) of orderly reports in the archive?

EDIT: parallel across reports helped by A)?

from naomi-aghq.

Ok then in that case if you want to use spud it looks like this

spud <- spud::sharepoint$new("https://imperiallondon.sharepoint.com/")
folder <- sp$folder("HIVInferenceGroup-WP", "Shared Documents/Naomi", verify = TRUE)
folder$list()
folder$upload(...)
folder$download(...)
If you want to use MS365R it looks like

site <- Microsoft365R::get_sharepoint_site(site_url = "https://imperiallondon.sharepoint.com/sites/HIVInferenceGroup-WP", tenant = "imperiallondon",
app = "42dff8b4-7cc4-43de-82f1-3cddd3b8a47c", scopes = c("Group.ReadWrite.All", "Directory.Read.All", "Sites.Manage.All"))
drive <- site$get_drive()
drive$list_items()
drive$upload_file("local/path", "one_drive/path")
drive$download_file("one_drive/path", "dest/path")

from naomi-aghq.