declaredesign / designlibrary Goto Github PK

Create a function that lists all available designers and designs with corresponding citation and description

simple_two_arm_designer(N=200, code=TRUE) does not return a N=200 design with the 200 substituted in, but instead returns a block of code where N is still a free variable. code=TRUE essentially ignores all the other arguments

Having a flag that changes the return type is usually weak design - see eg https://martinfowler.com/bliki/FlagArgument.html - you may be better off just having users do get_design_code(my_design), instead of copying the code=TRUE boilerplate into every single design.

c. When design gets made by a designer add in a declare_citation (made by declare design team, date etc, with possibly a brief description taken in from attributes

Would be nice to know what the name of the Rmd file is and where it got put.

See, for example, the otherwise very shoddy contribute_design

That check:

• code option T or F
  • all shiny args in formals
  • code creates an object that
  • basic checks: can produce design, can produce rmd, can produce diagnoses of right dimensions
    • expand_designer_shiny_args_text
    • get_or_run_diagnosis
    • get_shiny_arguments
    • get_constants
    • designer_default_args_text
    • expand_designer_shiny_args_text
    • get_shiny_diagnosis
    • get_or_run_shiny_diagnosis
    • contribute_design
• vignettes + designers:
  • returned object should be same name as * in *_designer

https://help.github.com/articles/creating-a-pull-request-template-for-your-repository/

ie a vignette that lists every design ) designer in library with a citation and short summary (eg print get_analysis?). Citation especially important as new designs get added.

via DeclareDesign/DeclareDesign#162

@nfultz has outlined the problem very nicely here

Basically, construct_design_code() is using getSrcref() to grab the source code of the designer functions in the package. That's great and works fine locally because we can set the project to run R CMD INSTALL --no-multiarch --with-keep.source DesignLibrary by default.

The problem arises in two instances:

1. I don't think that R CMD CHECK has keep.source as an option, so our checks will always break. Is there a way to set global variables when the check runs, before the package is installed?
2. When users don't specify that source should be kept when installing, the package breaks massively.
   This breaks bigly:

   detach("package:DesignLibrary", unload = TRUE)
   devtools::install_github("DeclareDesign/DesignLibrary")
   library(DesignLibrary)
   simple_two_arm_designer()

   This works:

   detach("package:DesignLibrary", unload = TRUE)
   devtools::install_github("DeclareDesign/DesignLibrary", keep_source = TRUE)
   library(DesignLibrary)
   simple_two_arm_designer()

@nfultz is there a way to ensure that keep.source is always true when the package is built? Can we set R_KEEP_PKG_SOURCE=yes as a side effect before installing the package on the user's computer, for example?

• shiny
• design / designer vignettes
• should link up with Nick about this

macartan proposed tags: estimation, measurement, experimental, observational, quasi-experimental, null design replication design etc

Please add designs by editing the comment below, using the following syntax:

- [ ] [Author(s), year](url_for_article.html)

• delete RDS files
• from existing vignettes: delete stuff that made RDS files before, and other boilerplate

It would be good to have a designer for CLPM - should be able to specify # of items , # of Subjects, # of waves, response family, and a matrix of model coefficients.

For a while the travis build has been breaking when it tries to run the design declaration in the crossover_designer()

The code that's breaking looks like this, and runs fine in local checks, but not when travis tries to run checks:

    crossover_design <- declare_design(
      population,
      potential_outcomes_A,
      potential_outcomes_B,
      estimand,
      assignment,
      dplyr::mutate(A = as.numeric(Z %in% c("T2", "T4")),
                    B = as.numeric(Z %in% c("T3", "T4"))),
      declare_reveal(YA,Z),
      declare_reveal(YB,Z),
      estimator_sat,
      estimator_direct)

Travis error below and build log here.

Basically it doesn't seem to have access to dplyr. I've tried adding library(dplyr) to testthat.R, and it's also in the travis.yml.

What am I doing wrong here? Something obvious I am missing, clearly.

> test_check(package = "DesignLibrary")
── 1. Error: (unknown) (@test_designers.R#13)  ─────────────────────────────────
Could not evaluate step 6 as either a step or call.
1: the_designer() at testthat/test_designers.R:13
2: declare_design(population, potential_outcomes_A, potential_outcomes_B, estimand, 
       assignment, dplyr::mutate(A = as.numeric(Z %in% c("T2", "T4")), B = as.numeric(Z %in% 
           c("T3", "T4"))), declare_reveal(YA, Z), declare_reveal(YB, Z), estimator_sat, 
       estimator_direct) at /home/travis/build/DeclareDesign/DesignLibrary/DesignLibrary.Rcheck/00_pkg_src/DesignLibrary/R/crossover_designer.R:60
3: tryCatch(eval_tidy(qs[[i]]), error = function(e) tryCatch(callquos_to_step(qs[[i]], 
       qnames[[i]]), error = function(e) stop("Could not evaluate step ", i, " as either a step or call."))) at /tmp/RtmppZMCvC/devtools3fc36d3f8160/DeclareDesign/R/declare_design.R:151
4: tryCatchList(expr, classes, parentenv, handlers)
5: tryCatchOne(expr, names, parentenv, handlers[[1L]])
6: value[[3L]](cond)
7: tryCatch(callquos_to_step(qs[[i]], qnames[[i]]), error = function(e) stop("Could not evaluate step ", 
       i, " as either a step or call.")) at /tmp/RtmppZMCvC/devtools3fc36d3f8160/DeclareDesign/R/declare_design.R:153
8: tryCatchList(expr, classes, parentenv, handlers)
9: tryCatchOne(expr, names, parentenv, handlers[[1L]])
10: value[[3L]](cond)
11: stop("Could not evaluate step ", i, " as either a step or call.") at /tmp/RtmppZMCvC/devtools3fc36d3f8160/DeclareDesign/R/declare_design.R:154

• Observational designs for descriptive inference

  • Simple random sampling
  • Stratified clustered random sampling
  • Measuring latent variables (election forecasting)
  • Topic analysis
  • Respondent driven sampling - J
  • Weighted sampling

• Experimental designs for descriptive inference

  • Audit experiments
  • List experiments
  • Randomized response
  • Conjoint experiments
  • Experimental games - AJ

• Experimental designs for causal inference

  • Two-arm design
  • Multiarm designs
  • Experiments with blocks and clusters
  • Factorial designs
  • Encouragement designs (two-sided non-compliance)
  • Fixed allocation designs
  • Stepped wedge designs - J
  • Crossover designs
  • Partial population design for spillover analysis
  • Selective trials - J
  • Adaptive trials
  • Mediation analysis

• Observational designs for causal inference

  • Difference in differences - J
  • Matching
  • Regression discontinuity designs
  • Synthetic control
  • Process tracing
  • Cross national time series

Designers should all have code in attributes, our designers will do that recursively

• Go through and document all user-functions
• Ensure all functions are tested (check coverage)
• Documentation for DesignLibrary package itself
• Add internal checks to all functions
• Test that all the RDS stuff works, and has big sims
• Check all vignettes compiling nicely

Should be done on the fly when documenting package, if possible. If not we'll drop it.

systemfit package not currently loading with DesignLibrary installation

Add a simple stan model to the library ; priors enter as part of estimation declaration; compute frequentist properties of Bayesian design, eg Bayes risk given prior distribution but also given distribution provided in declare_population

Hi all -- trying to use simple_two_arm_designer(), but getting an error after a clean install of master branch. I see:

> library(DesignLibrary)
> simple_two_arm_designer(N = 100)
Error in construct_design_code(simple_two_arm_designer, match.call.defaults()) : 
  could not find opening tag in simple_two_arm_designer

What am I doing wrong?

• write guide on documentation for a designer (Lily)
  • how to link designers to designs, using seealso
• how to add code to attributes
• how to add shiny arguments to attributes, and restrictions

See https://github.com/nfultz/stackoverflow/blob/master/R/match.call.default.R

@acoppock

Current function is in design/R/helpers
It essentially puts the ses in parentheses under the diagnosands and does rounding and lets you add names if you use the defaults. Looks nice after kable.

• better to be part of diagnose_design; and produce default print out after diagnosis
• should be more reliable in figuring out whether bootsrapping was done or not
• default names need to be consistent with current names which is risky; better perhaps for each diagnosand to have a long form label that is then taken for this purpose

reshape_diagnosis <- function(diagnosis,
digits = 2,
col.names = NULL,
default.names = c("Estimator", "Coef Name", "Estimand", "Bias", "RMSE", "Power", "Coverage", "Mean(Estimate)", "sd(Estimate)", "Mean(se)", "Type S", "Mean(Estimand)")
) {

At the moment when make_vignette() doesn't have any front_text passed to it, it looks up the help file for the designer and takes it from there. That's nice functionality for automatically generating vignettes for any designers we have contributed with full documentation, but we need to think about the (hopefully common) use case in which a user is adding a designer and a vignette.

In such cases, we could just error and say, either you need to add a description to the documentation of your designer, or you need to specify front_text? We could also just generate a vignette with no front_text (also a nice alternative).

@medinali what do you think is the best approach? My temptation is for make_vignette() to look for front text firstly in the function's formal arguments (front_text), then in the help file, and if it fails to find any front text anywhere, then it just builds a vignette with no front text, and perhaps some boilerplate (e.g., "user has not provided a description of this desiger").

We have a method that is working well but can incorporate other methods (i.e., using the ugly duckling paste0 approach).

Need to write a separate vignette on these approaches. Should be clear about format of code object (vector of strings, each element is a verbatim line of code).

• biggest change is + operator
• this also requires changes so some of the tests

• eventually we will have a list of the designers here with checkboxes, each member of berlin team will take 3-4, fix up code, write blurb at front and back, and get in a plot if useful

Using new guide, with seealso, keywords, author attribution, etc.

• no more designs in the library package
• create download_design() function
• create test for website designs

i.e., we need to have a little guide / reminder for how to use roxygenize (document package in rstudio or use function)

Travis says:

Quitting from lines 34-35 (crossover_designer.Rmd) 
Error: processing vignette 'crossover_designer.Rmd' failed with diagnostics:
Please only send design objects to diagnose_design.
Execution halted

@medinali would you mind taking a look at some point this week?

• deprecate contribute_design
  • but use the example code in it for documentation
• how to write documentation
• where to put the design object as rdata file
• how to add metadata
• example of documentation

Related designers listed in the help for given designers (see also cluster_factorial_designer...)

see enhancing navigation, here for description of how to do this by creating "families" of designs, or by using seealso

Can't see how to issue on website so issuing this here.

A wishlist for the overview table (much of which I realize may be hard and will require plugins):

• Glyphicons to download Rdata for each design
• direct link to shiny app for each designer
• subset table by clicking on keyword
• sort table by column
• search tool

• ideally with test-run on WZB
• idea is to have it populate only with designers from DesignLibrary