I do not have much experience with cluster computing, but I have access to a SLURM cluster.
Is there any straightforward way to execute your pipeline there?

if the previous is not possible, I still would like to make a comprehensive analysis of all the methods and datasets, is there any way to easily handle all datasets/methods dependencies by re-using as much of your code as possible to execute it in a regular Linux Server?

Hi, thanks again for a great job!

I ran into the following problem, not sure if I'm doing something wrong or it is an actual error. I also have a couple of questions regarding the overall structure of the benchmarking repository and how to execute it.

When running the dynbenchmark/scripts/06-benchmark/1-submit_jobs.R in line 23:

 methods <-
    dynwrap::get_ti_methods(method_ids, evaluate = FALSE) %>%
    mapdf(function(m) {
      l <- m$fun()
      l$fun <- m$fun
      l$type <- "function"
      l
    }) %>%
    list_as_tibble() %>%
    select(id, type, fun, everything())

As I see it after modifying the tibble obtained by dynwrap::get_ti_methods() with the mapdf() there is no column 'id'. Therefore, the following operation select() raises the following error:

Error: `id` must evaluate to column positions or names, not a function

This could be easily fixed by keeping the 'id' info in the mapdf(): l$id <- m$id

  methods <-
    dynwrap::get_ti_methods(method_ids, evaluate = FALSE) %>%
    mapdf(function(m) {
      l <- m$fun()
      l$fun <- m$fun
      l$type <- "function"
      l$id <- m$id
      l
    }) %>%
    list_as_tibble() %>%
    select(id, type, fun, everything())

Is this an actual error and proper fix, or I'm missing something?

• In general what would be the proper way to try to recreate your results?
  By sequentially executing your scripts, results are stored in the 'derived' folder, but it seems that sometimes the scripts are pointing to the 'results' folder (initially empty) instead of the 'derived'. I solved this by downloading the dynbenchmark_results repository into the 'results' folder. Now, I'm a little confused about how the scripts and the 'derived' and 'results' folder interact with each other. Could you please give a couple of words on that?

Thanks again!!

I used Ubuntu 22.04 LTS
My session info shows that I installed all other dyn* except dynbenchmark.

Hi,
can you please help clarify the key difference between the tree trajectory and the multifurcation? I was looking at supplementary fig S2(a) but not quite sure how you define the bifurcation, vs multifurcation vs. tree.
so for a multifurcation, at any node one can split into any number of branches? (0 or more)
for a tree, at any node one can split into??

Thanks a lot

evaluate_ti_method(dataset, method, parameters, metrics,
give_priors = NULL, output_model = TRUE, seed = function()
random_seed(), map_fun = map, verbose = FALSE)

it is hard to understand how to use.
can you provide more？

Hi,
It would be very helpful if you could share the code you used to create the synthetic datasets.
Specifically I am interested in creating a dataset with "golden standard" trajectory.
For my understanding, you used Splatter framework in combination with a trajectory backbone.
I can't understand how you were able to specify all of the "from" and "to" nodes by using Splatter path simulation.
Any help / code will be appreciated!

• Scalability
• Accuracy
• Stability
• Summary
• Guidelines
• Interpretation
• Example predictions
• Manuscript

Hello,

Does any one have an idea how to add a vector of cells to the model ?

Thanks

Thanks for the great work!
There are more single-cell trajectory inference methods published recently and surely in the future. Will you keep adding the new methods into your evaluation? Thanks!

Error message on Mac:
Installation failed: Does not appear to be an R package (no DESCRIPTION)

Error message on Linux:
HTTP error 404: Not found.

I got the following error when trying to install dynbenchmark:

> devtools::install_github("dynverse/dynbenchmark/package")
Downloading GitHub repo dynverse/dynbenchmark@master
✔  checking for file ‘/tmp/RtmpOkmvRG/remotes29e382fcc140/dynverse-dynbenchmark-74fe868/package/DESCRIPTION’ ...
─  preparing ‘dynbenchmark’:
✔  checking DESCRIPTION meta-information ...
─  checking for LF line-endings in source and make files and shell scripts
─  checking for empty or unneeded directories
─  looking to see if a ‘data/datalist’ file should be added
─  building ‘dynbenchmark_0.0.0.9000.tar.gz’ (766ms)
   
Installing package into ‘/home/pmonteagudo/R/x86_64-redhat-linux-gnu-library/3.5’
(as ‘lib’ is unspecified)
* installing *source* package ‘dynbenchmark’ ...
** R
** data
*** moving datasets to lazyload DB
** inst
** byte-compile and prepare package for lazy loading
Warning: replacing previous import ‘dyneval::calculate_harmonic_mean’ by ‘dynutils::calculate_harmonic_mean’ when loading ‘dynbenchmark’
Warning: replacing previous import ‘dyneval::calculate_geometric_mean’ by ‘dynutils::calculate_geometric_mean’ when loading ‘dynbenchmark’
Warning: replacing previous import ‘dyneval::calculate_arithmetic_mean’ by ‘dynutils::calculate_arithmetic_mean’ when loading ‘dynbenchmark’
Error : object ‘get_platform_from_counts’ is not exported by 'namespace:dyngen'
ERROR: lazy loading failed for package ‘dynbenchmark’
* removing ‘/home/pmonteagudo/R/x86_64-redhat-linux-gnu-library/3.5/dynbenchmark’
Error in i.p(...) : 
  (converted from warning) installation of package ‘/tmp/RtmpOkmvRG/file29e3879a6627c/dynbenchmark_0.0.0.9000.tar.gz’ had non-zero exit status

Any thoughts on that? Thanks in advance