Reproducible Research

In our lab, we try to make our research reproducible. This means that all the results from a paper can be reproduced with the code and data available online. Please follow this link to the reproducible papers from our lab.

Motivation
After a colleague asked something about a paper you wrote, you spend a considerable amount of time finding back the right program files you used in that paper. Not to talk about the time to get back to the set of parameters used to produce that nice result.
Because this type of situations sounded all too familiar to many people of the lab, we are now trying to make our research reproducible. Most of the ideas about reproducible research originate from Jon Claerbout and his research group at Stanford University. We believe reproducible research can be helpful in many ways:

  • It will help us in the first place, to reproduce figures in the revisions of a paper, to create earlier results again in a later stage of our research, etc.
  • Other people who want to do research in the field can really start from the current state of the art, instead of spending months trying to figure out what was exactly done in a certain paper. It is much easier to take up someone else’s work if documented code is also available.
  • It highly simplifies the task of comparing a new method to existing methods. Results can be compared more easily, and one is also sure that the implementation is the correct one.
  • It increases the impact of our research. By making the entire research results available online, more colleagues will use these results in their work.

This may all sound very trivial, and in discussions with colleagues, there was a general agreement that this is how research should be performed. However, in practice, only few examples are available today. Making articles reproducible indeed requires a certain investment in time. However, we think that it is worth the investment. The interest is hard to quantify, but from download statistics and Google rankings, we can see that it really pays off!

How to make a paper reproducible?
Of course, it all starts with a good description of the theory, algorithm, or experiments in the paper. A block diagram or a pseudo-code description can do miracles! Once this is done, make a web page containing the following information:
  1. Title.
  2. Authors (with links to the authors’ websites).
  3. Abstract.
  4. Full reference of your paper, with current publication status, and a PDF of your paper. This allows others to cite it correctly!
  5. The code to reproduce all the results, images and tables. Make sure all the code is well documented, and that there is a readme file explaining how to execute it.
  6. The data (images, measurements, etc) to reproduce all the results, images and tables. Add a README file explaining what the data represent.
  7. A list of configurations on which you tested your code (software version, platform).
  8. An e-mail address that people can use for comments and remarks (and to report bugs).
Depending on the field in which you work, it can also be interesting to add the following (optional) information to the web page:
  1. Images (add their captions, so that people know what Figure xx is about).
  2. References (with abstracts).
For every link to a file, add its size between brackets. This allows people to skip large downloads if they are on a slow connection.
For examples, see the list of reproducible papers above.
More information
For more information, please see 
  • Our article on the topic in IEEE Signal Processing Magazine (link to the paper on Infoscience)
  • The EUSIPCO 2010 plenary talk by Martin Vetterli
  • LCAV has been pursuing the reproducible research initiative for a long time. You can check this website for our earlier contributions. LCAV’s current reproducible research repository can be accessed here.