A set of handy scripts and tools that helps you create, manage, and run your SAM project. Frodo supports `SLURM` workload manager and can parallelize your simulation over multiple nodes.
jsonnet add unnecessary decimal point to doubles and I don't know how to work around it. One way is to cast them as string and let SAMpp cast them back to its preferable type. But this is rather tedious.
I think I should include all the parameters in the file name then I don’t have to change it all the time, also probably I can just include them in the parameters pool as well.
The problem here is that my list is space delimitated and that's not a good thing, I think it'd be nice if I can just save the entire input script to the jsonnet instead of decomposing it and feeding it after. This will make it much easier to remove or add one parameter afterward.
Advantages
I don't need to introduce restriction in the grid file, like what I have in the nobs, and no space rule.
I think it'd be nicer if I can modulate each files to pieces like ResearcherParameters.jsonnet, JournalParamters.json, etc... This makes the process of creating a new config file much easier.
I think some of the parameters like hacking methods can also be modulated. For instance, if I have them into a module and I can read them one by one by their own parameters, then constructing a JSON is much easier.
I can add the simulation parameters to the Submission record and therefore add them to simulation files but then since meta files are aggregate of one study, then I cannot easily do this. I can see what are the parameters than I’m reading are and use those but I still need a better way of handling this.
For now, I can do it like this and see what’d I need later but I’m not happy with the file names and they way of handling parameters.
I think it’s a good idea to have an option to make a release ready SAM using the present Makefile in the SAMoo folder. This way, I can always make sure that I have the latest and best version, if I want to.
I can just replace the grid.R with a library to write a JSON and then either pass the JSON to SAM or extract variables from it via bash and pass it to the jsonnet.
Stopos understands space delimitated files, since I'm reading the file line by line, I need to check if my function to read the line of params.pool can handle space delimitated files.
I think it’d be nice if I save all the output from a simulation to a SQLite database at the end of the run, both meta and sim data. I stumbled upon this tool called csvkit which I think can do that. TERMSQL was another tools designed for this purpose.
I'm making all addresses fix and also would like to archive everything at the end of a run, including the executable. So, I think, I should call SAM from $PROJECT/build/ and the archive the build directory as well.
I think I need to change the order of loops in such a way that each group has relatively similar running time. Probably maxpubs should be the top level category.
This is an old problem. I'm still thinking how to include all the parameters in such a way that at the end I can query a CSV or SQLite file and have everything.
I can use csvkit to import everything into the SQlite apparently. I can also save everything into a dB folder. Cleanup the CSVs and move on to the next simulation.
This is quite challenging. I would like to keep a high level representation of the data, and also guarantee the flexibility. The performance isn't an issue here, I just need to be flexible. JSONnet is still nice because it takes care of loops and recursion and what not. But, it's quite high maintenance at this point where I'm changing the API so much. My Python solution is nicer to look at but it's not very elegant either. I'm not sure how to find the balance here.
The other thing is, I like to have this options. It's not just a topping, it helps a lot with the parallelization when I move to Lisa for instance.
The new method of creating config files are working nicely, but I don't have much control over which set of config files I submit together. I need a way of sorting the JSON files.
I think the fact that I'm copy-pasting everything rather aggressively, e.g. entire folder, it's very likely that I actually overwrite some of the files during the operation.
One way to get around the issue of none equal numbers of columns is to first read all the headers from the CSV files, create the biggest table possible and then delegate the work to the to_sql. In this way, I'd not face an issue.
This is still not the best way but it'll provide a way to add data to a table.