Reproducible Articles (R-Articles), Short Replication Articles (r-articles), Reply Articles

Since September 2017 IEEE Robotics & Automation Magazine solicits R-articles (i.e., papers that report experiments aiming to be fully reproducible), short articles reporting on the replication of r-Article results, and author short article replies.

All papers are peer reviewed. The result will be a two stage very high-quality review process. The first stage will be the ordinary rigorous review process of a top-tier publishing venue. The second stage will be the replication of the experiments by the community (what is usually the core of the scientific method).

Cover letter

The authors will have to explicitly declare in the cover letter, if their article aims to be considered for publication as Reproducible Article (R-Article), as a short article illustrating the replication of other authors’ results (a ‘r-article’, or a short reply by the authors of a reproduced experiment.

It is the responsibility of the authors to publish all the necessary information to make their results reproducible.

What to put in a RAM R-Article

The authors will have to share whatever they believe is necessary and sufficient to reproduce their results and confirm that in the cover letter.

Minimum Requirements

  1. a description. i.e., an almost ordinary journal article with text, figures, and multimedia, complying at a minimum with the good experimental methodology (GEM) guidelines, [1], or similar.
  2. data sets stored in Code Ocean [2], and all the necessary information tests that have been run; only in well­ justified special cases will we accept different sharing platforms; videos of experiments should not be considered as data sets, but as multimedia attachments if they are not actually part of the data elaborated by some algorithm in the robot software. The statistical measures of confidence (such as ‘p-values’) of the results must be spelled out clearly, calculated and being retrieved on the datasets
  3. complete code identifiers and preferably downloadable code (executable files might in some cases be enough) on Code Ocean—the algorithms as they have been coded and all necessary library and middleware information; also in this case, only in well justified special cases will we accept different sharing platforms
  4. a hardware (HW) description or HW identifier (we need to know the physical details of the robots to be able to repeat robotics experiments); by HW description, we mean the full design (at the level of detail necessary for replication) of the robot or robots used in the experiment.
The information in (1) and (2) and any other the authors deem necessary for reproducing their results must be stored in a specific pdf file as ‘additional online materials.’

It is important that the code and HW are well identified to be able to replicate experiments.

  • Is it an experimental paper?
  • Are the system assumptions/hypotheses clear
  • Are the evaliuation criteria spelled out explicitly?
  • What is being measured and how?
  • Do the methods and measurements match the criteria?
  • Is there enough information to reproduce the work?
  • Do the results obtained give a fair and realistic picture of the system being studied?
  • Are the drawn conclusions precise and valid?
The Euron GEM guidelines in a nutshell [1]. The guidelines were meant as an adaptation of the basics of the scientific method to robotics and AI.

The R-Article Life Cycle

It will be possible to publish a short article about the result replication of an R-­article, what we call an r-article (non-capitalized r).

Similarly, the authors of the original R-­article will be able to submit, in the form of a short peer reviewed article, a Reply to the authors of the r­-article, we will call this a Reply article.

All these articles will have the same minimum requirements as the R-articles.

Peer Review, Data and Code Consistency Check

All articles above will be peer reviewed like any other RAM article and will undergo a quick data and code consistency check.

[1]        F. Bonsignorio, J. Hallam, A. P. del Pobil, Eds. (2008). GEM guidelines: Euron GEM Sig report. [On­ line]. Available: http://www.heronrobots.com/ EuronGEMSig/downloads/GemSigGuidelines Beta.pdf

[2]        CodeOcean. Discover and run scientific code. [Online]. Available: http://www.codeocean.com

Easy Links