Chapter 2 Introduction to reproducible research

The terms reproducible research was coined by Jon Claerbout in the 1980s when he writes an essay on reproducible computational research, and describes the hurdles he faces when making a text book that incorporates text, data, and results in a stand-alone document. According to Claerbout’s principle, ‘scholarship does not only consist of theorems and proofs but also and perhaps even more important of data, computer code and a runtime environment which provides readers with the possibility to reproduce all tables and figures in an article’.

This image by The Turing Way is a great representation of what Jon Claerbout may have had in mind:

The need for reproducibility is increasing dramatically as data analyses become more complex and involve larger datasets, analysts, and more sophisticated computations.

2.1 what is reproducible research?

According to a U.S. National Science Foundation (NSF) subcommittee on replicability in science, “reproducibility refers to the ability of a researcher to duplicate the results of a prior study using the same materials as were used by the original investigator. This entails that a result obtained by an experiment or observational study should be achieved again with a high degree of agreement when the study is replicated with the same methodology by different researchers

2.1.1 Terminology distinctions

Reproducible research is sometimes known as reproducibility, reproducible statistical analysis, reproducible data analysis, reproducible reporting, and literate programming.

2.1.2 Reproducible versus replicable

Replicability means obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data.

2.1.3 Reproducible versus repeatable

Repeatability measures the variation in measurements taken by a single instrument or person under the same conditions, while reproducibility measures whether an entire study or experiment can be reproduced in its entirety.

This is a way for researchers to verify that their own results are true and are not just chance artifacts.

2.1.4 Reproducibility crisis

The replication crisis (or replicability crisis or reproducibility crisis) is, as of 2020, an ongoing methodological crisis in which it has been found that many scientific studies are difficult or impossible to replicate or reproduce. The replication crisis affects the social sciences and medicine most severely.

2.1.5 The problem in the context of public service

  • Government is closed
  • The public doesn’t know how decisions are made
  • Duplication of work within the government
  • Types of reproducibility

2.1.6 The solution in the context of public service

  • Openness and transparency in government
  • Evidence based decision making
  • Efficiency
  • Enable others to use our methods or help improve ours

2.1.7 Barriers to open and reproducible workflows

  • Directives
  • Approval of management
  • Data privacy and security policies
  • Culture

2.1.8 What needs to be reproduced?

For evidence-informed decision making, results need to be reproducible.

The results include: - Stattistical and inferential tables - Visualizations/figures/graphs - Values reported in the text - The statistical evidence in support of the findings (e.g., p-values, confidence intervals, credible intervals).

2.1.9 Motivation

Some aspects to consider that may make your experiments, processes, and reports more reproducible: 1. Don’t Read Between the Lines. …
2. Be Strict. …
3. Keep Things Transparent. …
4. Collaborate. …
5. Automate Your Processes.

2.2 Benefits of reproducubility

  • Increased likelihood that the research will be correct
  • Reproducibility makes it easier to check the research
  • It is easier to reproduce the research independently
  • Easier to extend the research
  • Reusable code and instruction resulting in increased efficiencies

2.3 How to make research reproducible

  1. The first reason to repeat experiments is simply to verify results. Different science disciplines have different criteria for determining what good results are.
  2. The next reason to repeat experiments is to develop skills necessary to extend established methods and develop new experiments. For these, we need reproducible methods, tools, and platforms.

2.4 Requirements for reproducibility

  1. The “raw” data is made available, where “raw” refers to the data prior to any manipulation by the researcher (e.g., prior to any data cleaning and transformation).
  2. A complete set of instructions is provided explaining all steps used in the processing and analyzing the data.

2.4.1 Additional requirements

  1. A set of files is provided containing the data and code, and it is possible to create the tables and any data-derived charts/graphics/visualizations by running the code.

  2. Details about the system being used to run the analysis: operating system, patches, random number seeds, specific versions of all software/packages/libraries are listed.

  3. The code is written in a way that can be readily understood.

  4. Open/transparent. All the data and materials are available (as opposed to “available upon request”) – e.g., posted on GitHub, or in an international data repository.

  5. That is, either:

  • Another party (e.g., a reviewer) has successfully reproduced the results and certified them as such.
  • Logs demonstrate that key results were successfully created from the inputs.
  • The key results are linked to the data and code, so the relationship can be directly inspected.

A final requirement, which is sometimes known as literate programming, is that:

  1. The entire report is written using code. That is, a file or files are provided which, when run, import the data, produce all the results, insert the results into the text of the report, and format the report.