Next in Reproducibility: standards, policies, infrastructure, and human factors
Annual Tutte Lecture, May 27, 2019
Tutte Institute for Mathematics and Computing
The National Academies of Science, Engineering and Medicine released a consensus study report on “Reproducibility and Replicability in Science” on 7 May 2019. The report provides definitions of reproducibility and replicability accounting for the diversity of fields in science and engineering. It assesses the state of reproducibility and replicability across science, and it offers recommendations for researchers, agencies, policy makers, journals, professional societies, and more. In its definitions, the committee emphasized the ubiquity and importance of computing and the data-intensive processes in modern science. Reproducibility was defined as obtaining consistent computational results using the same input data, computational steps, methods, code, and conditions of analysis as an original study. Although this may sound straightforward, the report describes how a number of systematic efforts to reproduce computational results have failed in more than half of the attempts made, mainly due to insufficient detail on digital artifacts, such as data, code and computational workflow. In the wake of this comprehensive report, what is next for reproducibility? Research communities will need to develop standards of practice, institutions will adopt formal policies, and funding agencies may look to support more infrastructure and tools to enable reproducibility. At the same time, the training of researchers and data scientists should be strengthened and in some ways also standardized—imitating perhaps the training of pilots and aviation professionals. In this analogy, complex processes are managed with checklists and guides, and full accountability and auditability are ensured through automatic capture of records and mandated provenance. Last but not least, the aviation industry recognizes the importance of human factors, and places them at the center of design and policy. As data and computation dominate in all fields of human enquiry, pivoting our attention to the “human in the loop” is now critical.