TC on Model-Based Optimization for Robotics Best Paper Award

Policies and Procedures

Preliminaries:

The Best Paper Award of the TC on Model-Based Optimization for Robotics is a yearly award whose aim is to acknowledge and promote exceptional papers published during a calendar year and recognize the contribution of their authors to the TC topics.

This document presents the set of rules governing the process of this award. Some care has been taken to have comprehensive rules, while keeping the document short enough. Some corner cases might not be completely covered, in which case the handling of them is left to the discretion of the TC Co-Chairs.

The process consists of 4 steps:

  1. Nominations
  2. Short-list selection
  3. Evaluation by a jury of the short-list paper
  4. Announcement of the winner and finalists

In the following, we also detail procedures for handling conflict of interests (CoI) in the evaluation process. At the end of the document, we outline cases considered to provide a CoI between the co-authors of a nominated paper and those involved in the evaluation process. Beyond these cases, additional CoIs can be identified by self-declaration from the evaluators, and should be declared in circumstances with any appearance that the person would have bias in their evaluation.

Nomination:

Any peer-reviewed paper within the TC scope that is published during the year covered by the award is eligible for nomination. For journal papers, the electronic publication date is used, for conferences, the first day of the conference main session is used. There are no requirements that the authors of a nominated paper belong to the TC.

Special restriction: Papers co-authored by TC Co-Chairs are non-eligible for nomination unless the first author of the paper was a student at the date of publication. This restriction applies only to Co-Chairs, not Junior Co-Chairs.

Rationale: This restriction aims at avoiding any suspicion of bias in case a TC Co-Chair paper would receive the award or be recognized as a finalist. However, we do not want to unfairly penalize students that are advised by or collaborate with the Co-Chairs.

Anyone can nominate an eligible paper. In particular, self-nominations are welcome. 



Short-list selection

A first selection of the papers is carried out by the TC Co-Chairs to limit the number of papers for evaluation by the Jury. A maximum of 12 papers will be passed to the Jury, with a target range of 10-12 papers.

Procedure: 

  1. Each paper with a Co-Chair CoI is sent to a jury member of the previous year who needs to answer the question “Based on your experience from last year, do you think this paper has a good chance to be nominated as a winner or finalist?”. If the answer is positive, the paper is added to the list of selected papers.
  2. All papers without a Co-Chair CoI are graded from 0 to 5 by each of the TC Co-Chairs.
  3. The non-conflicted papers are ranked according to their average score, and the first n papers are taken, with n such that (i) there is a clear cut between the n-th and n+1-th paper, and (ii) n is in the target range (possible tradeoffs between these two criteria are solved at the discretion of the Co-Chairs).
  4. As an additional selection criterion, each first-author is only eligible to have one paper proceed to evaluation by the jury. Likewise, each author is only eligible to have two papers proceed to evaluation by the jury (no matter their authorship position). When an author has papers in excess of this limit, only their highest-ranked papers are eligible to proceed to jury evaluation.

Rationale: The goal of this selection is twofold:

  • Removing papers that are not within the scope of the TC or of clearly insufficient quality
  • Keeping the charge of the jury acceptable
  • Ensuring that there is diversity in the pool of papers considered by the jury

The underlying assumption is that the Co-Chairs are knowledgeable enough about the works done within the scope of the TC and that the target number of papers is enough to guarantee that no non-selected paper would have had a reasonable chance of being a winner or finalist for the award.

 

Evaluation by a jury of the short-list papers

The jury consists of 5-6 members, invited by the Co-Chairs. Those members should be recognized researchers in the topics covered by the TC, with experience, such as a past or current editorial position, that gives them a good overview of the field.

The selected papers are each sent to a minimum of three jury members for evaluation.

Procedure:

  1. Jury members are first asked to declare any CoI with the selected papers. 
  2. Papers are then split randomly among jury members so that each paper is evaluated by a minimum of three persons (that are not in conflict) and the load between jury members is evenly shared.
  3. Evaluation: the papers are graded on the following criterion:
    • relevance (0-1)
    • novelty (0-4)
    • impact (0-3)
    • quality (0-3)

The final grade is relevance x (novelty+impact+quality). 

For each paper, the 3 final grades are averaged.

 

 

Winner and finalists

The award winner is the paper with the highest average. If 2 papers are tied, two winners are announced.

The award finalists are the next two highest-ranked papers (regardless of the number of winners). In case there is a tie for the second finalist place, a third paper is accepted in this category.

If there are additional ties leading to more than 2 winners or 3 finalists, jury members will be asked for additional evaluations.

 

 

Conflicts of Interest

Evaluators are considered to have a CoI under any of the following circumstances:

  • a past advising relationship with an author of the paper (including post-doc advising)
  • employment at the same institution as an author of the paper
  • collaboration with an author of the paper in the last five years
  • any other circumstances that may create an appearance that the person might have a bias in the evaluation of the paper