Reproducible Articles (R-Articles), Short Replication Articles (r-articles), Reply Articles

Since September 2017 IEEE Robotics & Automation Magazine solicits R-articles (i.e., papers that report experiments aiming to be fully reproducible), short articles reporting on the replication of r-Article results, and author short article replies, see [5].

All papers are peer reviewed. The result will be a two stage very high-quality review process. The first stage will be the ordinary rigorous review process of a top-tier publishing venue. The second stage will be the replication of the experiments by the community (what is usually the core of the scientific method).

Cover letter

The authors will have to explicitly declare in the cover letter, if their article aims to be considered for publication as Reproducible Article (R-Article), as a short article illustrating the replication of other authors’ results (a ‘r-article’, or a short reply by the authors of a reproduced experiment.

It is the responsibility of the authors to publish all the necessary information to make their results reproducible.

What to put in a RAM R-Article

The authors will have to share whatever they believe is necessary and sufficient to reproduce their results and confirm that in the cover letter.

Minimum Requirements

1. A description. i.e., an almost ordinary journal article with text, figures, and multimedia, complying at a minimum with the good experimental methodology (GEM) guidelines, [1], or similar.
2. data sets stored in Code Ocean [2], IEEE Dataport  [3], GitHub [4]  or other suitable platform (to be agreed with RAM EIC and Reproducibility editor), and all the necessary information tests that have been run; only in well­ justified special cases will we accept different sharing platforms; videos of experiments should not be considered as data sets, but as multimedia attachments if they are not actually part of the data elaborated by some algorithm in the robot software. The statistical measures of confidence (such as ‘p-values’) of the results must be spelled out clearly, calculated and being retrieved on the datasets
3. complete code identifiers and preferably downloadable code (executable files might in some cases be enough) on Code Ocean —the algorithms as they have been coded and all necessary library and middleware information. The storage on other suitable platforms must be agreed with RAM EIC and Reproducibility editor.
4. a hardware (HW) description or HW identifier (we need to know the physical details of the robots to be able to repeat robotics experiments); by HW description, we mean the full design (at the level of detail necessary for replication) of the robot or robots used in the experiment.

The information in (1-2) and any other the authors deem necessary for reproducing their results must be stored in a specific pdf file as ‘additional online materials.’

It is important that the code and HW are well identified to be able to replicate experiments.

1. Is it an experimental paper?
2. Are the system assumptions/hypotheses clear
3. Are the evaliuation criteria spelled out explicitly?
4. What is being measured and how?
5. Do the methods and measurements match the criteria?
6. Is there enough information to reproduce the work?
7. Do the results obtained give a fair and realistic picture of the system being studied?
8. Are the drawn conclusions precise and valid?

The Euron GEM guidelines in a nutshell [1]. The guidelines were meant as an adaptation of the basics of the scientific method to robotics and AI.

Typical structure of an R-Article 

R-articles should be organized in three parts:

1. Magazine Article: It provides a ‘magazine style overview of the published research like any typical RAM article.  It should convey   the general ideas, motivation, relevance and obtained results. 
2. Supplemental Information: It provides an extended version of the magazine article. It describes, with sufficient details to allow to reproduce the work, Materials and Methods, Results and their analytical Discussion. The description of the research must be in accordance with the Euron GEM Guidelines [1]. It must provide the user guide for reproducing the results. It will be uploaded as a Multimedia Attachment (this may change in the future),
3. Data, Code and Design Repository: an online repository on  IEEE  DataPort , [3],or Github, [4],   includes  all  the  data,  code  and, if they are open sourced, the  hardware designs of the robots involved in the research, if they are not, the necessary information to clearly identify the hardware used for the experiments. All additional details  necessary  and  sufficient to reproduce the experiments must be given. If the published research includes simulations, authors are recommended to store code and data allowing the simulation experiments to be reproduced online on Codeocean, [2].

The R-Article Life Cycle

It will be possible to publish a short article about the result replication of an R-­article, what we call an r-article (non-capitalized r).

Similarly, the authors of the original R-­article will be able to submit, in the form of a short peer reviewed article, a Reply to the authors of the r­-article, we will call this a Reply article.

All these articles will have the same minimum requirements as the R-articles.

Peer Review, Data and Code Consistency Check

All articles above will be peer reviewed like any other RAM article and will undergo a quick data and code consistency check.

Difference with publishing your data and code for a standard peer-reviewed article

Publishing an R-article on RAM is not the same that publishing on another journal and sharing code and data. An R-article is specifically peer-reviewed also from the point of view of its reproducibility. Moreover, it enters into the research reproducibility process described above. This is a unique opportunity at the time of writing these guidelines 

[1]        F. Bonsignorio, J. Hallam, A. P. del Pobil, Eds. (2008). GEM guidelines: Euron GEM Sig report. [On­ line]. Available: http://www.heronrobots.com/ EuronGEMSig/downloads/GemSigGuidelines Beta.pdf

[2]        CodeOcean. Discover and run scientific code. [Online]. Available: http://www.codeocean.com

[3]        IEEE DataPort: Dataset Storage and Dataset Search Platform. [Online]. Available:

http://ieee-dataport.org 

[4]        Github. [Online]. Available: http://www.codeocean.com

[5]        F. Bonsignorio, "A New Kind of Article for Reproducible Research in Intelligent Robotics [From the Field]," in IEEE Robotics & Automation Magazine, vol. 24, no. 3, pp. 178-182, Sept. 2017, doi: 10.1109/MRA.2017.2722918.