Lorena A. Barba group

Reproducibility PI Manifesto

"Reproducibility PI Manifesto" is a pledge by Prof. Barba to lead her group with a consistent reproducibility policy.

"Reproducibility PI Manifesto" is a pledge by Prof. Barba to lead her group with a consistent reproducibility policy.

The ICERM workshop on "Reproducibility in Computational and Experimental Mathematics" was held December 2012 in Providence, RI. It brought together people from academia, funding agencies, national laboratories, professional societies and the publishing industry to discuss the technological and social challenges of ensuring reproducibility in this age of pervasive computation-based discovery.

Prof. Barba gave a lightning talk in the workshop in which she made a pledge to lead her research group with a consistent policy aiming to improve reproducibility practices. The slides of her presentation are available on figshare under a CC-BY license.

The pledge:

  1. I will teach my graduate students about reproducibility.
  2. All our research code (and writing) is under version control.
  3. We will always carry out verification and validation (V&V reports are posted to figshare)
  4. For main results in a paper, we will share data, plotting script & figure under CC-BY
  5. We will upload the preprint to arXiv at the time of submission of a paper.
  6. We will release code at the time of submission of a paper.
  7. We will add a "Reproducibility" declaration at the end of each paper.
  8. I will keep an up-to-date web presence.

Reference

  • swcarpentry

    I think these are wonderful goals, but as I said in http://software-carpentry.org/blog/2013/02/correctness-isnt-compelling.html, we need to do a much better job of explaining what the benefits are to the scientists themselves: on its own, in today's climate, reproducibility doesn't make economic sense. Which of these principles actually help the researcher herself/himself get more done in less time, and why?

    • Thanks for leaving this comment, and posing your challenge question. Reading your post, I think your main thesis is that the Reproducibility Movement is doomed because scientists really don't care about it, since they are not rewarded for it. When you say "reproducibility doesn't make economic sense," I think the assumptions are that (1) there is a sizable cost (time), and (2) the benefit, associated with insurance against retraction, is minimal.

      Let's first address cost. Although there is some set-up and maintenance cost in having a versioning system for code and writing, for me this cost is small and absolutely worth it. Other scientists may be less tech savvy and find this hard —some people still do their writing in Word, for example, and like to "track changes", instead of using LaTeX on version control. I'm sad for them. When I have been obligated to use that method for collaborative writing (e.g., NSF panel report), I was miserable: multiple versions of a Word document are sent over email and people read over changes and "accept changes" and it takes an incredible amount of time. It hinders collaboration, it is unmanageable, error-prone, unsafe and laborious. My workflow—update, diff or merge if needed, edit (in portable and fast raw text), latex, bibtex, commit—is a breeze in comparison.

      For code, version control is equally time-saving and a safeguard agains errors. I can't imagine how burdensome it would be to manually keep track of different versions of a code (appending 'v1' and 'v2' to the filename?). The chances of catastrophe with this method are endless.

      Besides setting up version control, which item in my pledge implies a time cost? Surely doing proper V&V on new code is a necessary step and cannot be skimped on. Uploading preprints to an archive takes minimal time, and sharing code openly has no added costs for me (it's already in a repository). Teaching my students how to do science is my job, and I like to do it well.

      Keeping an up-to-date web presence is indeed time-consuming, but it's not directly related to reproducibility. The reason I included it is that good science communication can enhance our goals by making our work understandable and available.

      This brings me to the benefits. I don't think they have anything to do with insuring against a retraction. In my opinion, the practices that we are talking about have an immediate benefit in the protection of the knowledge being created in my research group. Code, data, writing and graphics are available to all members in the group, are curated, backed up, and shared openly. This is not only efficient, but it fosters confidence and collaboration.

      But I agree with you that evangelizing about it can be pointless and annoying to others who just don't see the rewards.

      • swcarpentry

        Thanks for your reply, Lorena. I don't think Reproducibility is "doomed", but I do think it faces a challenging chicken-and-egg problem. If most people don't do it, then most people won't look for it when they review grants and papers; if it isn't being considered when G&P are reviewed, then most people who don't already have the required technical skills will look at it and say, "Not worth the effort." Software Carpentry is trying to break this cycle by giving them the technical skills so that doing things the right way won't seem as daunting; what's missing is any credible quantification of the benefits. How much time do you think you and your grad students invest in RR (both in learning and setup, and day-to-day in doing it)? How much time do you think that saves you on a day/month/year-long time scale? I think the RR movement would have much better luck persuading individual scientists, their deans, and funding agencies to emphasize this if they had a few of those numbers...

  • Geoff Oxberry

    Lorena asked me to comment here briefly, since I recently gave a presentation on how reproducible research makes sense from an economic standpoint (e.g., research productivity and research quality). The presentation can be found on FigShare, at http://figshare.com/articles/How_to_succeed_in_reproducible_research_without_really_trying/640512.

    So far, the reproducible research movement tends to motivate its work in the ways that Greg Wilson (swcarpentry) describes the blog post he links. To summarize that motivation, it's primarily an appeal to "doing science the right way", and sometimes also an appeal to prevent fraud through increased transparency. Greg and I have discussed why this appeal does not reach some audiences, and his counterargument is more or less the calculation that other researchers make. When I talk to skeptics, their response goes something like, "I've been doing research a long time, and it's been working out well for me. Why should I change what I'm doing, and why should I do more work than I need to?"

    The implicit premise here is not unlike the premise I see people formulate when they talk about not using best practices in software engineering: changing my habits is going to take more work, and I am not going to see the benefit. However, there are scientific studies that show that employing best practices in software engineering yields gains in productivity and improved software quality, and that the two are related: by making it easier to find bugs through habits like testing, you can cut down on debugging, which is the most expensive phase of programming. This data is what Greg uses in Software Carpentry to convince scientists to learn good software engineering practices.

    When it comes to computational science, the habits needed to do research reproducibly overlap a lot with good software engineering practices, such as using version control for everything, and writing tests for your software. The same studies that Greg quotes in Software Carpentry to argue that good software engineering practices make good economic sense (because they make researchers write code faster) also apply to reproducible research. There is also data (which I cite in my presentation) that reproducible research tends to be cited more, making the case for a research quality argument that complements the productivity argument Greg makes for software engineering practices.

    However, as Greg rightly points out in his Software Carpentry presentations and in his blog posts, data demonstrating efficacy is essential. More data must be gathered to demonstrate more conclusively whether reproducible research makes economic sense through increased productivity and better papers (through more citations, or acceptance in higher impact factor journals, etc.). The productivity argument is one that I have used in the past to convince my skeptical managers to let me do research reproducibly, and I think it is an argument particularly resonant with scientists because there is some data to support it, but without more data, it is hard to make the case that one type of workflow is better than another, and it is hard to make concrete recommendations beyond the ones I have made in my presentation.

    • Thanks so much for comment, Geoff. When I read Greg's argument about the need to provide evidence of the actual benefits of investing in reproducibility practices, it made me think of the move in manufacturing to total quality control. I don't know the historical developments, but at some point in time, quality control became a hot topic in industry, international standards were developed and companies invested heavily in their quality practices and pursued certification. Somehow, evidence must have shown that this was good for business!
      For now, I do think that one can build a reputation through healthy reproducibility habits. Eventually, I visualize a collective seal of approval of certain practices (e.g., publishing the code used to analyze data for a publication). But we have some work to do if we want to "make a business case" for those practices.

  • Krishna Bhogaonker

    Hey folks, I am a grad student in statistics at UCLA. I thought it might be useful to include the student perspective on this discussion. I genuinely agree with everyone that reproducibility is of great importance. There have been so many recent revelations where key findings in genetics (Begley and Ellis, Nature, 2012) were not reproducible--even by the labs that generated the original research. From the student side, I think that there is just not enough training in these programming and data management techniques. As a grad student--at least at my university--besides the standard curriculum we have to pick up so many skills on the fly: programming languages, mathematics, database management, merging data, etc. This learning approach is not conducive to producing quality code and outputs. Unfortunately, with the pressure to move quickly through graduate programs and with the demands of teaching, etc., the students are not learning these types of good research habits. I am lucky because I have a decent programming background, but most of the students I know--in statistics and in other disciplines don't use version control or literate programming techniques, etc. I think that students would be interested in learning these types of techniques--because the tools for reproducibility actually help to produce better quality code, increase the reuseability of code, and reduce the chances of embarrassing errors.

    I agree that until journals and Grant agencies push for these types of controls that the battle will be uphill. However, I think that there is a contingent of grad students out there who are interested in these tools but don't have access to anyone who can teach them. I just watched some of the webcasts from the recent SciPy 2013 conference and one big focus was reproducibility (https://conference.scipy.org/scipy2013/index.php). Perhaps a Coursera course on reproducibility would be an excellent medium by which to reach these interested students.