ctu-bern / presize Goto Github PK

Review: manuscript

Part of review for JOSS.

Overall

Firstly, let me thank the authors for their efforts to prepare an excellent package. It's certainly of wide interest for the research community and particularly providing a Shiny App is a great way to make it accessible to those without programming expertise. As R is popular in statistics, life sciences and biomedical research, an R package would be widely used in these fields.

I agree that PR #64 significantly improves the manuscript as discussion in Issue #61. Overall the paper nicely summarises the functionality of the package and provides some examples of use cases. I have some minor feedback on structure and grammar. Generally, I believe the paper is worth of acceptance with minor editing.

Contributions

• Contribution and authorship: Has the submitting author (@aghaynes) made major contributions to the software Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

As shown here @aghaynes has prepared the manuscript and got feedback other authors. Authorship seems to be appropriate to me.

Structure

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?

These are both well covered but I'd recommend merging them into one "Summary" section to avoid repetition. For example lines 9, 19, and 22 are similar.

Introduction

• State of the field: Do the authors describe how this software compares to other commonly-used packages?

Normally I'd expect some discussion of alternatives but none seem to be available for "precision" sample sizes (or confidence intervals). I'd be willing to make an exception to this as under the circumstances but it could be helpful to provide some examples of "power" calculations for hypothesis testing. JOSS manuscripts are meant to be short but I think some comparisons here would be informative and related software (e.g., for power calculations in R) should be cited. As pointed out the paper this is complementary work.

Usage

As discussed in #61 I think this section is appropriate, especially as long-form vignette documentation (discussed here: #66) is not yet available.

Grammar

The writing is overall of high quality and easy to understand for non-experts. Some statistical jargon is unavoidable in this kind of work. There are a few sections that could be rephrased for clarity. These should be minor changes that will not take long. Please read the manuscript carefully for similar issues.

line 9, 20: we have therefore...

line 13: Sample size is a crucial step in for planning a clinical study.

line 13-14: A too small study that is too small leads to inconclusive results.

line 18: man research projects aim at to...

line 22: We programmed have developed an R package that...

Change "first" and "second" to "firstly" and "secondly".

line 35: we have also

line 42: can be installed and loaded into the R session...

line 53: we have developed

line 61: We often observe in our consulting activity that researchers try to implement a hypothesis-based approach

line 64 fascilitate -> facilitate, adequate -> appropriate

• correct spelling and grammar (above)

References

References are formatted correctly but a few notable citations are missing.

• Cite R packages for power calculations
• Cite the R core team

> citation()

To cite R in publications use:

  R Core Team (2020). R: A language and environment for statistical computing. R Foundation for
  Statistical Computing, Vienna, Austria. URL https://www.R-project.org/.

A BibTeX entry for LaTeX users is

  @Manual{,
    title = {R: A Language and Environment for Statistical Computing},
    author = {{R Core Team}},
    organization = {R Foundation for Statistical Computing},
    address = {Vienna, Austria},
    year = {2020},
    url = {https://www.R-project.org/},
  }

We have invested a lot of time and effort in creating R, please cite it when using it for data
analysis. See also ‘citation("pkgname")’ for citing R packages.

Questions

A limitation of the package is that it does not allow calculating the probability of a CI, i.e. the probability that a future confidence interval will have at least the desired precision.

It's great that the authors have acknowledged clearly and as it stands the packages is already very useful. Are there plans to add this in the future? How difficult would it be? Maintaining an R package is a commitment and I understand that time and funding to support this may be limited. As mentioned in the paper, investing in this will reduce the burden on consulting activities and aid researchers that do not have this expertise available.

This completes my review of the manuscript. I will test the latest version of the software and shiny app to complete the JOSS review.

Cronbach's alpha method is also defined in

Bonett, D. G. and Wright, T. A. (2015) Cronbach's alpha reliability: Interval estimation, hypothesis testing, and sample size planning, J. Organiz. Behav., 36, pages 3– 15. doi: 10.1002/job.1960.

Describe the solution you'd like
prec_alpha implementing functions 3 and 4 in the appendix. Maybe 2? Plus equivilents to get N given a CI width.

New function added 2022-11-22: prec_cronb
It still needs integrating into the shiny app

Submission: openjournals/joss-reviews#3118
Status: Conditional accept

Hey @aghaynes & @majensen,

I've finished my review and was able to check off most items but a few remain.
The submission is in very good shape overall and I think the changes are minor.

First off, kudos @aghaynes to you and your team for producing such a nicely put together package.
Your description of the background for this work and your accompanying documentation made it easy for me to get familiar with the technique and explore the functionality of the package.
The documentation looked very complete and the Shiny app is nice touch, especially in the way it automatically generates R code to match.

Below are the items I'd like to see addressed before I can change my review to an Accept:

Contribution and authorship

I see different sets of authors in a few places and wanted to make sure everything was complete and correct:

• Paper: Haynes, Lenz, Stalder, Limacher
• R package: Lenz [aut], Haynes [cre,aut], Limacher [aut], Stalder [ctb]
• git commit history: Haynes, Lenz, Stalder (by order of commits)
• pkgdown site: Lenz, Haynes, Limacher

First, could you clarify Limacher's association and contribution to the package? Second, is the above information (including ordering) correct?

Functionality

Everything worked great except the Shiny app.
When I install the package, load the package, and attempt to run the Shiny app, I get an error and the app does not launch:

Error: In sliderInput(): min, max, and value cannot be NULL, NA, or empty. 24.
stop(call. = FALSE, sprintf("In %s(): min, max, and value cannot be NULL, NA, or empty.", fun)) 23.

I had a hunch and checked to see if the Shiny app worked under the development version and it does.
I assume the authors are aware of this but I think it warrants ensuring JOSS archives a fixed version of the software and not 0.1.3.

Let me know if you can reproduce my issue. If so, I think generating a new release and updating this submission is warranted.

Community guidelines

Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

I wasn't able to find any of these three bits of information in the README file.
Please add them.

Paper

The paper is overall well-written and quite useful.
However, I believe it doesn't fit in with the JOSS guidelines in two ways:

1. It does not contain an initial, high-level summary paragraph entitled Summary
2. It contains too much detail about the package. That said, the information that falls outside the scope of the JOSS paper is very useful information and could be moved elsewhere

Other, non-review details

These are outside the JOSS guidelines so the review is not contingent on addressing them:

• You might link your pkg site, https://ctu-bern.github.io/presize from GitHub (and possibly other places)

• There's a typo at the bottom of the Example section in the README, "THe" instead of "The"

• The image at the bottom of your pkgdown site under the Funding header isn't working for me

• I noticed some scenarios where passing nonsense output produces cryptic errors both when calling functions directly or calling them indirectly using the Shiny app. For example, if I mistakenly specified the conf.width argument in percent points (e.g., 5) rather than as a proportion (e.g., 0.05),

  prec_prop(p = 0.5, conf.width = 5, conf.level = 0.95, method = 'wilson')
  Error in uniroot(function(n) eval(quo) - prec, c(1, 1e+07), ...) :
    f() values at end points not of opposite sign
  

  or in the Shiny app:

  Error: An error has occurred. Check your logs or contact the app author for clarification.

  I don't think these are deal breakers and I know writing tons of validation logic can be time-consuming but I thought I'd point out that I ran into it.

Please let me know if you have any questions of if I can clarify anything.

The function prec_meandiff(), in differences.R, has an expression called "md" where s is calculated with denominator n1 + n2 + 2 (line 98). Shouldn't that be n1 + n2 - 2?

Thanks,
Clay Ford

Describe the bug
warning re condition > 1 if passing multiple conf.widths to prec_prop

To Reproduce

> prec_prop(p = .36, conf.width = c(0.07, 0.08, 0.09, 0.1, 0.11))

     sample size for a proportion with Wilson confidence interval. 

     p      padj        n conf.width conf.level       lwr       upr
1 0.36 0.3607440 718.9930       0.07       0.95 0.3257440 0.3957440
2 0.36 0.3609716 549.6552       0.08       0.95 0.3209716 0.4009716
3 0.36 0.3612296 433.5577       0.09       0.95 0.3162296 0.4062296
4 0.36 0.3615177 350.5139       0.10       0.95 0.3115177 0.4115177
5 0.36 0.3618361 289.0708       0.11       0.95 0.3068361 0.4168361

NOTE: padj is the adjusted proportion, from which the ci is calculated.

Warning messages:
1: In prec_prop(p = 0.36, conf.width = c(0.07, 0.08, 0.09, 0.1, 0.11)) :
  more than one method was chosen, 'wilson' will be used
2: In if (conf.width < min(p)) { :
  the condition has length > 1 and only the first element will be used

Expected behavior
No warning

Review here

Code for badge:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.03118/status.svg)](https://doi.org/10.21105/joss.03118)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.03118">
  <img src="https://joss.theoj.org/papers/10.21105/joss.03118/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.03118/status.svg
   :target: https://doi.org/10.21105/joss.03118

Badge looks like this

Citation (bibtex):

@article{Haynes2021,
  doi = {10.21105/joss.03118},
  url = {https://doi.org/10.21105/joss.03118},
  year = {2021},
  publisher = {The Open Journal},
  volume = {6},
  number = {60},
  pages = {3118},
  author = {Alan G. Haynes and Armando Lenz and Odile Stalder and Andreas Limacher},
  title = {`presize`: An R-package for precision-based sample size calculation in clinical research},
  journal = {Journal of Open Source Software}
}

Review: Community guidelines

Part of review for JOSS.

Contributing guidelines

Regarding this criteria, the Contributing guidleines are available (as requested in #61).

1. Contribute to the software 2) Report issues or problems with the software 3) Seek support

However, efforts should be made to direct users to these and guide new users through this process (I note some have been made already).

• provide contributing guidelines
• direct users to them in the README
• direct users to the GitHub repository from CRAN (DESCRIPTION)
• direct users to the Issues for Bug Reports from CRAN (DESCRIPTION)
• GitHub issues template for bug reports and feature requests
• GitHub pull request template

Some of these are not strictly necessary for publication but are minor efforts that aid novice R users for bug reporting or seeking support via GitHub. See the community GitHub page for further recommendations.

The description of the ICC page should mention that it makes the assumption that no interaction effect exists between rater and subject.
This is mentioned in passing in the Bonett reference, but only under "model 3". Non-statisticians are unlikely to understand this from the formulas.

Calculating the sample size required for developing a clinical prediction model

Binary

• A note on estimating the Cox-Snell R2 from a reported C statistic (AUROC) to inform sample size calculations for developing a prediction model with a binary outcome
• Sample size for binary logistic prediction models: Beyond events per variable criteria
• No rationale for 1 variable per 10 events criterion for binary logistic regression analysis
• Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes (pmsampsize for stata and R)
• Minimum sample size for external validation of a clinical prediction model with a binary outcome

Multinomial

• Sample size considerations and predictive performance of multinomial logistic prediction models

Continuous

• Minimum sample size for developing a multivariable prediction model: Part I – Continuous outcomes
• Minimum sample size for external validation of a clinical prediction model with a continuous outcome

Adaptive sample size determination for the development of clinical prediction models

https://onlinelibrary.wiley.com/doi/10.1002/sim.9025?af=R

Hi I need to extract sample size and rr possibilities which return a lwr >2.
How can I set this lwr in prec_riskratio?

Thank for your support
Pia

The current arguments of prec_mean are mu and sigma, where statistically, it would make a lot of sense if they were mu and sigma, or mean and sd. I would opt for mu and sigma, since it describes the assumed true distribution.

Correcting this scores not so well on the tradeoff of correctness and backward compatibility. Maybe you could add a little comment in the argument descriptions.

Describe the bug
Shiny app () returns error as shown:

ntot' required when 'prev' is specified

To Reproduce
See screenshots. Hosted locally with:

launch_presize_app()

Development version installed with remotes::install_github("aghaynes/presize"). Commit: aghaynes@a917c32

Expected behavior
Prevalence input is required by ShinyApp. It is possible to use a button to disable it. Slider could be removed from UI with conditional input.

Estimating sample size from conf.width without specifying n is not currently supported in Shiny App. Cannot specify ntot from Shiny App.

Screenshots

Desktop (please complete the following information):

R version 4.0.3 (2020-10-10)
 os       macOS Catalina 10.15.7      
 system   x86_64, darwin17.0          

   presize        * 0.2.2      2021-04-07 [1] Github (aghaynes/presize@a917c32)

remotes upgrades warnings during installation to errors resulting in the following (or a similar) error message...

√ checking for file 'C:\Users\XXX\AppData\Local\Temp\Rtmp8EQc6a\remotes1f3864265efc\CTU-Bern-presize-03aeee1/DESCRIPTION' ...

• preparing 'presize':
  √ checking DESCRIPTION meta-information ...
• checking for LF line-endings in source and make files and shell scripts
• checking for empty or unneeded directories
• building 'presize_0.0.1.9005.tar.gz'

• installing source package 'presize' ...
  ** using staged installation
  ** R
  ** inst
  ** byte-compile and prepare package for lazy loading
  Fehler: (konvertiert von Warnung) Paket 'ggplot2' wurde unter R Version 3.6.3 erstellt
  Ausführung angehalten
  ERROR: lazy loading failed for package 'presize'
• removing 'C:/Program Files/R/R-3.6.0/library/presize'
  Error: Failed to install 'presize' from GitHub:
  (converted from warning) installation of package ‘C:/Users/XXX/AppData/Local/Temp/Rtmp8EQc6a/file1f387af6bc9/presize_0.0.1.9005.tar.gz’ had non-zero exit status

use Sys.setenv(R_REMOTES_NO_ERRORS_FROM_WARNINGS="true") to rectify.

prec_spec(spec=c(0.6, 0.7, 0.8, 0.9), n=56, method="wilson") 

     precision for a Specificity with wilson confidence interval. 

  sens   sensadj  n conf.width conf.level       lwr       upr
1  0.6 0.5935806 56  0.2485780       0.95 0.4692916 0.7178696

Issues:

• The resulting CIs do not coincide with the method in Stata (cii proportions)
• It should read: precision for a specificity with Wilson confidence interval
• The column names should read ‘spec’ and ‘specadj’
• It would be nice if the routine also runs if the total number (i.e. number of cases plus number of non-cases) and the prevalence (of cases) are given.

As a part of the JOSS review/acceptance procedure, the package had to be uploaded to an archive. It have been uploaded to Zenodo.

Is your feature request related to a problem? Please describe.
Not a problem, add a vignette as extra documentation.

mention in the JOSS review

Describe the solution you'd like
"real world" examples of how to use the package/functions.

Could also compare using classical power analysis...??

Review: documentation

Part of review for JOSS.

Examples

The following functions do not have an @examples or \examples field in their "man" or R files. Please add these so they are displayed in the R/RStudio help system.

• prec_cor
• prec_lim_agree
• prec_or

Ideally several examples should be given, e.g., for Pearson and Spearman correlation and all examples should be well commented to explain the context for understanding use cases.

• all examples commented

- [ ] usage for parameters without defaults explained in examples

Shiny App

The documentation for the shiny app needs more details. It is not clear before launching what the app does.

• launch_presize_app

Building dev version of documentation

Documentation is not available for the dev version 0.2.1 installed from GitHub. These comments above are based on CRAN release 0.1.3 because of this, although the GitHub source seems to have missing examples also. Building the docs with devtools or roxygen2 before pushing to GitHub is recommended (it is required for uploading to CRAN)

• build updated documentation
• check docs are available in R help system before pushing to GitHub

Suggestions

The review does not hinge on these but are recommended for benefit of users understanding the software.

• README: Providing a screenshot of the Shiny App would give a good preview on GitHub. Images can also be displayed in the R help system.
• The package itself does not have a documentation page that can be accessed by help(presize-package) or just help(precize). See the examples "help(igraph-package)" or help(ggplot2-package).

Optional:

• Vignette: A vignette is not required by CRAN but can be helpful to provide further context for examples.

Is your feature request related to a problem? Please describe.
Sample size calculation is often not transparently reported.

Describe the solution you'd like
This feature will help alleviate this issue by providing a paragraph to copy-paste/paraphrase into the publication/SAP/protocol. The paragraph could be displayed on each measures page below the replication code

Describe alternatives you've considered
None, I am all-in.

Please assign me the issue.

Power is currently ca 50% - half the trials will find a larger CI

Could abuse the closed formulae to estimate the power...

> mu <- 5
> sd <- 2
> n <- 20
> presize::prec_mean(mu, sd, n)

     precision for mean 

  mean sd  n conf.width conf.level      lwr      upr
  1    5  2 20   1.872058       0.95 4.063971 5.936029

> power.t.test(delta = 0.94, sd = 2, power = .5, sig.level = .05, type = "one.sample")

     One-sample t test power calculation 

              n = 19.3697
          delta = 0.94
             sd = 2
      sig.level = 0.05
          power = 0.5
    alternative = two.sided

> power.t.test(delta = 0.94, sd = 2, power = .8, sig.level = .05, type = "one.sample")

     One-sample t test power calculation 

              n = 37.49745
          delta = 0.94
             sd = 2
      sig.level = 0.05
          power = 0.8
    alternative = two.sided

Range checks for all params complete

• prec_auc
• prec_cor
• prec_icc
• prec_kappa
• prec_lim_agree
• prec_lr
• prec_mean
• prec_meandiff
• prec_neg_lr
• prec_or
• prec_pos_lr
• prec_prop
• prec_rate
• prec_rateratio
• prec_riskdiff
• prec_sens
• prec_spec

When a function is done, just check the box