The IROS Conference Review Board (ICPRB) consists of one Editor-in-Chief (EiC), 27 Senior Editors (SE’s), roughly 400 Associate Editors (AE’s), and several hundred Reviewers. Each paper to be reviewed will be assigned one Editor and to one of the AEs that that Editor supervises. The AE will be responsible for obtaining a minimum of two high quality reviews for each paper they handle, and for preparing an AE recommendation that explains the recommended decision regarding the paper acceptance and presentation type. The AEs will also help to identify papers to be considered for awards. The Editors will be responsible for reviewing and endorsing the work done by the AEs on the papers for which they are responsible.

Assignment of Papers to Editors and AEs. For IROS, the keywords are partitioned into sets, each of which is the responsibility of one Editor. When a paper is submitted to the conference (via PaperPlaza), the author’s choice of keywords will be used to determine to which Editor and AE the paper will initially be assigned. This initial assignment, which is done automatically by the system, is then reviewed by the Editor-in-Chief and the Editors to avoid conflicts of interest (COIs), to improve the matching of the expertise of the AE with the paper, and to provide load balancing across the AEs. Finally, the AEs must review the papers assigned to them and inform the Supervising Editor of any COI they may have with their assigned papers.

AE Tasks and Responsibilities. Once assigned papers to handle and notified that they should get started, the AE should follow the following steps:

1. As soon as possible, the AE should check the papers they have been assigned and make sure that they do not have a conflict of interest (COI) with any of their assigned papers. If a COI exists, they should notify the managing Editor and the EiC immediately so that the paper can be assigned to another AE.
2. By the posted deadline (typically one week after paper assignments are made), the AE should:
   1. identify papers to be recommended for summary rejection (rejected without review), or
   2. determine appropriate reviewers and request reviews from them in PaperPlaza.
3. The AE should monitor the review process as reviews come in. If reviews are not substantial enough or are not prepared in a appropriate way, then the AE should ask the reviewer to provide an improved review. In extreme cases, the AE may need to request an additional review if a reviewer does not provide a review or does not provide an appropriate review. If difficult situations arise, the AE should consult with the managing Editor and/or the EiC.
4. After a sufficient number of quality reviews (at least two) have been obtained, then the AE should prepare the AE report providing their recommendation on the paper. This should also include a recommendation on the presentation type (regular oral presentation or multimedia presentation), if the paper is to be accepted, and a recommendation on whether the paper should be considered for an award.
5. After the AE report has been submitted, the supervising Editor will review the AE recommendation and either provide the Editor Endorsement or suggest modifications.

Note about Organized Sessions: From the perspective of AEs and Editors, papers submitted to organized sessions will be handled in the same way as any other paper. The decision regarding whether an Organized Session Proposal (ISP) will be accepted, and if so, which papers will be included in it, will be handled by the Senior Program Committee (SPC).

The review process for IROS is managed using the PaperPlaza system. PaperPlaza provides a wide variety of tools to help AEs manage the review process. Reviewer assignments, review entry, AE reporting, and final decisions are all managed using PaperPlaza.

To access the system, go to the PaperPlaza page, click Start and then Log in. If you have forgotten your login information, you can retrieve it using the PIN management page.

It may be useful to spend a few minutes looking over the help pages, and in particular the Associate Editor’s FAQ at the PaperPlaza site: PaperPlaza Help Page.

In certain special cases, an AE may recommend that a paper be rejected without sending the paper for review. In such cases, the AE writes a brief summary review giving the rationale for this decision, and gives the paper an Unsatisfactory rating. This is done via the usual AE Report mechanism (described below). The Editor assigned to the paper will then make the final determination as to whether the paper should be rejected without further review, or should go through the formal review process.

A paper should be rejected without review in any of the following cases:

• It clearly makes no novel contribution to the state of the art.
• It contains significant technical errors.
• The paper has been published previously (i.e., the paper is identical to, or nearly identical to previously published work by the same authors).
• The paper plagiarizes previously published research by other authors.

A paper should not be rejected without review merely because it makes only an incremental contribution, because it fails to report real-world experiments or because of poor writing quality. Further, a paper should not be summarily rejected because the AE feels its subject lies outside the scope of IROS (this judgment is left for the Program Committee). If there is any doubt as to the decision, the paper should be sent for review.

No more that 10-15% of submitted papers will be rejected without review.

The ICPRB has a strict review quality policy: every submitted paper must receive at least two substantial reviews. An AE should reject any sub-standard review received – e.g., too short or shallow. It is also the AE’s responsibility to see that reviews are constructive and not diminishing the authors’ efforts, even when they have to be negative or very negative. Relevant information about the reviewing process is described in the page IROS Reviewer Information.

Selecting appropriate reviewers is critical in order to have a quality and timely review process. The most important criteria is for the reviewer to have the appropriate expertise. Candidate reviewers may include leading researchers in the topic area, colleagues who are familiar with the topic, authors of papers on the reference list, authors of relevant papers that have been previously presented at IROS or in other high-profile journals and conferences. Generally, a key idea to secure reviewers is to align the interest of reviewers with ours: the paper is on a topic that they would really like to see or to say something about. Having reviewers who are interested in going over the manuscript helps alleviate the need of repeatedly sending reminder messages, and avoids the need for AEs to write detailed comments themselves to supplement shallow and not-to-the-point reviews.

In addition to having appropriate expertise, the following provides some other requirements and guidelines in selecting reviewers.

• The reviewer cannot have a conflict of interest with the paper.
• An AE cannot provide a review for the paper.
• At most one of the reviewers for a paper can be at the same institution as the AE for the paper.
• A mix of reviewers from different geographical regions is also desirable.
• While it is not explicitly forbidden to ask other AEs to perform reviews, it is likely that they will decline, since they will be quite busy managing their own paper load.
• A good mix of senior and junior reviewers is desirable as they provide reviews from different perspectives and at different levels of detail.

IMPORTANT NOTICE: Paperplaza has a very nice feature to help you choose your reviewers, which however has a potential for misuse. The feature is that you can get a list of registered users whose keywords match the paper at hand, so that a reviewer can be recruited with a single click. The abuse of this feature is that the AE selects a reviewer whom the AE does not know, and whose expertise is not proven. It may thus happen that junior members of the community (e.g., undergraduate students), or simply people that are associated to a keyword by chance or mistake, are asked a review in a field where they are not competent. This is clearly unacceptable, and ultimately counterproductive in terms of time – as the reviewer will probably decline, or anyway the Editors will discard the review.

Also, please note that you should not add new users to the system unless you are certain that they are not already registered. Multiple PINs for a single user can cause significant difficulty and confusion.

Review requests are handled by the PaperPlaza system. After you have compiled your list of reviewers, a review can be requested by following the Reviews link for a submission, then clicking the Request a review link. You may edit the standard form letter as you please. You must click Send to actually generate the e-mail invitation to the reviewer.

You can find details about this process in the PaperPlaza help pages: Compiling my reviewer list.

Once you have collected all reviews, you should prepare and submit your AE report. You can find details about the report process in the PaperPlaza Help pages available after you have logged in. Please note that we will not be using the rebuttal process.

We would like to have the AE reports have a common structure that:

• First, summarizes the paper topic/contribution (could be brief, even just 1-2 sentences).
• Next, notes the main strengths and weaknesses of the paper, summarizing the main points from the reviews and noting any additional issues noted by the AE. Please avoid copying excerpts of the reviews in your report, as authors will see the reviews.
• Then, notes any special circumstances that may apply. For example, if two reviews are in serious conflict, the AE Report should resolve the conflict if possible. If neither review is substantive, the AE Report should give a solid rationale for the decision to be taken.

Here is guidance and reminders on what to put or not to put in your report and in determining your scores:

• Please do not mention charge/mention plagiarism in the comments to the authors. That should be in the confidential comments only.
• Please do not mention a decision (reject or accept) in the text part of the AE report that is meant for the authors, but only the motivations for your recommendation. The reason for this is that your recommendation is a recommendation to the program committee and that while they usually do agree with the AE, in some cases the Program Committee may change the recommended decision, and we want to avoid confusing the authors if that happens.
• You will give a recommended score, which is not visible to authors. If the reviews disagree, please do not merely give the average as your rating. As AE, your role is to settle the conflict. In recommending your score, please consider that papers with an “A” are “Definitely Accept”, “B+” are “accept”, “B” are “high borderline”, “B-” are “DO NOT USE” , “C” are “low borderline” and “C-” are “reject” and “D” are “reject”. Ratings of “B-” are discouraged and should not be used. Notice that “U” is not “F”, i.e., it is not the next lowest grade in the technical score scale. Rather, “U” stands for “Unsuitable”, and should be used as a flag to signal “problem” papers – e.g., out of scope, or suspect of partial plagiarism, or incomplete, etc.
• Note that once you have submitted your AE report, no further reviews can be obtained or entered into the system. When you submit your report, all pending reviews are canceled automatically. If you made a mistake or anyway need to change the report, you must ask the Profile Editor to reopen the paper.

Request revision of the review. After checking a review that has been submitted this link may be used to request the reviewer to prepare and submit a revised review. The form letter “Request to submit a revised review” is available for this purpose. This is a conference wide form letter that may be individualized on the Personalized form letters page. Revision of a review may be requested if the status of the submission is Under review or Decision pending. When this request is sent to the reviewer the status of the review is reset to ‘Saved’ so that the reviewer may re-access and resubmit the review. The usual option to save a review is not available to the reviewer for such reviews.

Restoring reviews to original after revision requested. To Restore a review for which revision was requested to the original review click on the link “Revision requested on…” for the review in question. This opens an information box with a link to view the original review and another link to restore the original review.

NOTE: Disposition of unreceived revised reviews. Reviews for which a revision was requested but not received are considered reviews that were requested but not submitted by the system. If they are not restored to the original review they are automatically canceled when the Review summary report is submitted for the submission.

IROS features a number of awards. The ICPRB is asked to help in the selection of these awards. In particular, the ICPRB will designate a number of outstanding papers from which the Senior Program Committee will draw finalists for Conference Awards. Accordingly, Associate Editors are asked to identify which papers in their assignments they would consider as potential candidates for an award. To do so, please use the “Confidential comments to the Program Committee” textbox in your AE Report form. It is not necessary to identify which award would a paper be a suitable candidate for. The actual selection of finalists for each award will be made by the Senior Program Committee, using your inputs.

The AE’s recommendation, expressed in the report, will be reviewed by the supervising Editor, who will be responsible for checking that the quality standards of the review process (including number and depth of reviews, significance of AE’s reports avoiding undecisiveness) have been met. The Editor will issue a brief statement for each recommendation, whereby the correctness and completeness of the revieweing procedure is endorsed.

Editors will also overview and endorse the processing of award candidates by the AEs.

An ICPRB Editor, Associate Editor and a ICPRB Reviewer is deemed to have a conflict of interest in a submitted paper if
A.  he or she is a (co-)author of the paper; or
B.  one (or more) of the authors of the paper:
B.i.  is, or has been, a student or advisor of that person, or
B.ii.  has co-authored a paper or has closely collaborated in a research project with that person
in the previous five years, or
B.iii. is employed at the same institution (at the Department or Division level)  as that person;
C. there are any other circumstances which may create an appearance that the person might have a bias in the evaluation of the paper.

Plagiarism cases involve serious accusations, which should be dealt with carefully. IEEE has clear policies to follow. IEEE defines plagiarism as the reuse of someone else’s prior ideas, processes, results, or words without explicitly acknowledging the original author and source. It is important for all IEEE authors to recognize that plagiarism in any form, at any level, is unacceptable and is considered a serious breach of professional conduct, with potentially severe ethical and legal consequences (source: Section “8.2 Publication Guidelines” of the IEEE PSPB Operations Manual, “Guidelines for Adjudicating Different Levels of Plagiarism.” )

CrossCheck database and iThenticate tool

IROS has access to the CrossCheck database, and initiative to prevent scholarly and professional plagiarism. Every submission will receive a plagiarism similarity score. The score and scan reports are generated by an external provider (iThenticate) and the scan reports are stored on the iThenticate servers and not downloaded to the conference submission system servers. Eventually the reports are deleted from the iThenticate servers at a time determined by conference and provider policy, after which they are no longer available.

IMPORTANT: It is very important to note that it is not possible to draw any conclusion from the iThenticate numerical score alone. Unfortunately, due to the output from iThenticate algorithms, there will be a number of false positives. One issue is that it represents a cumulative score so that, e.g., a 1% similarity with 40 papers is shown as 40% similarity. Another issue is that there may be large similarity, but still not plagiarism. For example, if an author has a version of their paper as an technical report or in a public dropbox someplace, it might get a very high (e.g., 99% or 100% similarity score). Hence, it is necessary that the detailed report be scanned to make sure that there is indeed a case of plagiarism. Also, some of the papers will not scan properly due to font problems.

The iThenticate reports are available to you as an Associate Editor and to your Editor, but they are NOT available to reviewers. To access the report for a particular paper:

• click on the “Workspace” link
• On your Workspace, there is a column labeled “Plagiarism scan” it shows the percentage overlap determined.
• Click on the “Go to the plagiarism scan page” link for the paper in the “Plagiarism scan” column
• Click on the “View” link in the “Report” column to see the plagiarism report. You should review the report for all of your papers, and need to pay particular attention if the score is 40 or higher.
• Also on that page you can set one of two plagiarism flags, “Possible case of plagiarism” if you think there is plagiarism, and “Plagiarism report needs to be followed up” if you are not sure but think it should be followed up. If you determine that you need to set either of these flags, please also email your Editor to let them know to take a look at this paper.

Self-plagiarism. The definition of self-plagiarism is that the paper includes substantial overlap with another of the authors published paper(s) and that the previously published paper is not cited in the references and/or the contribution of the current paper over that other papers is not described in the current submission, both of which are required by IEEE policy.

The AE should be able to determine if self-plagiarism is a concern by reviewing the paper and the plagiarism report. If this is considered to be the case, then the AE should set one of the plagiarism flags and inform their managing Editor. If the Editor agrees, then they will inform the Editor-in-Chief who will also review the paper and the report. If the EiC concurs, then the paper is a candidate for summary rejection and the AE will be asked to prepare a report describing the reason for the summary rejection.

The process

Notes:

• The similarity score and iThenticate report are available to AEs and to Editors, but not to reviewers.
• Discretion and confidentiality are extremely important. The reviewers, AEs, and Editors should not discuss the details or names of potential plagiarism case with anyone other than the persons above them in the chain, e.g., an AE could discuss with their supervising Editor or the Editor-in-Chief but not with another Editor or AE.
• Comments related to plagiarism should be in the confidential comments for the PC and should not be mentioned in the comments for the authors. It will be up to the plagiarism process to determine what actions to take and what to report to the authors.

AE Procedures:

1. For all papers whose similarity score is 40% or above, the AE should review the paper and the iThenticate report.
2. The AE should provide a report regarding their findings in the confidential comments of the AE report.
3. The AE should indicate their determination regarding potential plagiarism in the plagiarism report section of the AE report.
4. If the AE indicated “plagiarism report needs to be followed up” or “possible case of plagiarism”, then they should alert their Editor.
5. Unless instructed otherwise by their Editor or the Editor-in-Chief, the AE should follow the review process normally for the paper by obtaining reviews and making a recommendation for acceptance based on the reviews and their own technical evaluation of the paper. It is important that this is done so that the paper can be treated fairly if the plagiarism alert is determined to be unfounded.

Plagiarism may also be spotted/reported by reviewers (recall, they don’t have access to the iThenticate report). If a reviewer detects a potential case of plagiarism, then they should document their concerns in the confidential comments portion of their review and should alert their AE. It is important that the Reviewer is factual in their remarks, and that as much and detailed evidence is provided as possible. For instance, this could be a copy of the supposedly plagiarized paper with the copied parts highlighted. It should also be noted that there exist freely available software that can detect plagiarism automatically: if this was used, details on the query and its outcomes would also be useful. Based on this information, the AE should use the information provided by the reviewer in the same fashion in which they would have if the alert was prompted by the iThenticate report.

The reviews will help to maintain the quality of IROS. Each review is required to meet the 1,200 non-white character threshold, which has the purpose to provide useful and constructive feedback to the authors. Please refrain from filling the space with meaningless or repeated text. The reviewers should respect the time invested by the authors and should not accept to review unless they are willing to at least provide a meaningful review.