acmsigsoft / open-science-policies Goto Github PK

Daniel, thanks for this. Overall it's excellent work. I just have the one concern.

The principle that "research output should be publicly accessible and reproducible" is rooted in a positivist epistemology that does not apply to all research. The SIGSOFT paper and peer review quality initiative were asked for clarification on this a while ago and here's what we came up with:

Recoverability, Replicability and Reproducibility. A study is "recoverable" when readers of a paper can understand how the work was done and why it was done that way. All research should be recoverable. A study is "replicable," when it can be repeated by an independent researcher. To replicate a study, it must be recoverable and its materials (e.g. tasks given to participants) must be available. Positivist research should be replicable; interpretivists and postmodernists reject the notion that social science is replicable. Case studies, grounded theory, qualitative surveys and action research are not replicable. A study is "reproducible" when the results can be precisely recreated using the original study's data and source code. While reproducibility is the gold standard for laboratory experiments, it is often impractical or unethical for other kinds of research. Not all datasets can be released safely; for example, qualitative interview transcripts typically should not be released due to de-identification risk; industry data often contains trade secrets. Qualitative analysis is typically not reproducible because interpretivist qualitative analysis (e.g. open coding), is inherently subjective. Reviewers should evaluate whether researchers have shared what is practical reasonable to share. Put simply: laboratory experiments that do not involve human participants should be reproducible. Positivist, predominately quantitative research should be replicable. Interpretivist or predominately qualitative research should be recoverable.

While the open science policy clearly states "All open science steps are optional for authors and reviewers," reviewers are going to look at policies like this one to evaluate papers, especially when the venue includes some reproducibility (or replicability or whatever) as a quality criterion. This further prejudices the review process against qualitative research, and research using sensitive quantitative datasets.

Is it possible to thread the needle a bit better on this, to avoid misapplication of open science standards to reject qualitative research?

Most links in the policies are raw URLs, e.g., https://en.wikipedia.org/wiki/Open_science instead of open science. I made it by design even if it is not elegant. This is because I suspect that many open science policies will be born out of copy pasting older policies. Formatted links get lost sometimes.

I left formatted links for those that I would be fine to lose.

There is something weird going on with Zenodo's automated archival and DOI. Once that is fixed (or clarified), I will archive the policies and add their DOI in text.

In case this is not possible, I will archive them manually.