3.1.1 Timeline

The Senior Editorial Board (consisting of the Editors, who are also called Senior Editors, and the Editors-in-Chief) conducts a preliminary evaluation of each submission. Some submissions are rejected or returned at this stage, usually because the submission violates a T-RL policy, clearly does not advance the state of the art, or is outside the scope of T-RL. This evaluation is typically concluded within two weeks of submission.

After that, the handling Editor assigns the paper to an AE. Within two weeks, the AE should have either recommended Editorial Reject or recruited Reviewers for a full review process (more details are given below). If the paper goes to full review, the target time to a decision is three months from the original submission date. This corresponds to approximately two weeks for the Senior Editorial review (Editors-in-Chief and Editors), two weeks for the AE to confirm Reviewers, five weeks to receive the reviews, one week for the AE to make a recommendation, and one week for the Editor to issue a decision. This leaves one to two weeks of buffer to accommodate late reviews or solicit emergency reviews when necessary.

Editorial rejects are often issued by the Editor-in-Chief, but all other publication decisions are made by the Editors.

The highest priority is a fair and thorough review of each submission. Nearly as important for the reputation of the journal and the satisfaction of authors, however, is a timely decision.

3.1.2 Double-Anonymous Reviewing Process

The Transactions on Robot Learning offers a double-blind reviewing process. This means that the authors do not know the names of the reviewers and the reviewers do not know the names of the authors. The guidelines for this process are specified here: https://www.ieee-ras.org/publications/rules-for-the-double-anonymous-review-process.

3.1.3 “New” and “Evolved” Submissions

Papers evolved from previously published manuscripts are dealt with in the same way as regular papers. Every paper needs to specify in which way it goes beyond the state of the art and the potential previous work of the authors.

If the Editor or AE determines that a manuscript does not contain a “substantive” new contribution, the paper should receive an editorial rejection decision. If you are in doubt, please consult with the handling Editor, who may in turn wish to consult with the Editors-in-Chief.

If the submission is found to potentially contain substantive differences or additions compared to the state of the art, the submission should be reviewed

Authors are responsible for ensuring that the copyright for the material being considered for T-RL publication either belongs to IEEE or can be transferred to IEEE.

The Transactions on Robot Learning will NOT consider submissions that are currently under review for a conference or any other publication. This is considered an ethical violation and such submissions will be returned immediately without review.

PhD theses, preprints posted on paper archives, papers that were presented at small workshops (as opposed to more formal conferences with published proceedings), and other documents that have not been formally published do not count as previous publications. Therefore, a submission may completely copy these papers (if they are by the same authors, of course) and still be considered “new.”

When a paper is assigned to an AE by an Editor, the AE should give the paper a quick read to roughly understand what it is about, its potential contributions, who the leading experts are in the topic area, and so on. If the submission is “Evolved” (i.e., it has a “PCV” notation), the AE should check to see that the submission has a statement of the difference from the conference paper and confirm that the journal submission has substantive changes from the conference version (see the discussion above). If the authors disclose that the submission has evolved from a closely related conference paper (or if it later becomes clear that this is the case), but the submission either does not give a clear statement of the changes to the paper or does not include the previous conference paper among the submitted files, the paper should be returned without further review. Similarly, if the difference between the submission and the conference paper(s) is not substantive (see the discussion above), the submission should be returned without further review.

The AE should also check the ithenticate report for possible cases of plagiarism. Potential plagiarism cases should be investigated and reported to the handling Editor.

After this initial screening, the paper will usually fall into one of three categories:

• The paper has the potential to provide a contribution. The AE should initiate the review process and start securing Reviewers (see more details below).
• The paper clearly fails to provide a contribution to the state-of-the-art, or it has some other major flaws (e.g., no novelty, out of T-RL scope, etc.). The AE should submit an Editorial Rejection recommendation to the supervising Editor within two weeks of receiving the paper to allow a quick response to the authors. See the section below on Preparing a Summary Report.
• The AE is unable to handle the paper for some reason, such as a conflict of interest. The AE should quickly inform the supervising Editor (within 2-3 working days) so that the Editor can assign the paper to another AE.

The AE should invite experts in the field to serve as Reviewers for the paper, with the goal of ultimately obtaining three high-quality reviews. Selecting appropriate Reviewers is critical to a high-quality, timely review process. Candidate Reviewers may include leading researchers in the topic area, colleagues who are familiar with the topic, authors of papers cited in the submission’s bibliography, authors of relevant papers that have been previously accepted, and authors who have submitted a paper to T-RL on a related subject. A good mix of senior and junior Reviewers is desirable as they provide reviews from different perspectives and at different levels of detail. For example, a junior Reviewer may carefully check the various technical details of the paper, but may also lack the depth to assess the significance of the contribution, while a senior Reviewer may be in a position to judge whether the paper brings a real contribution to the community. A mix of Reviewers from different geographical regions is also desirable.

During the process of submitting a paper to T-RL, authors confirm that they are willing to perform up to three quality, timely reviews for T-RL. Therefore, in general, T-RL authors should be willing to provide reviews. (Note, however, that some authors perform other significant services for the community, such as serving as an AE for T-RL!) This commitment is also stated on the Information for Authors page of the T-RL website. If a candidate Reviewer declines to provide a review and fails to provide a recommendation for an alternate Reviewer, the AE should consider following up to ask if she/he has a postdoc or senior PhD student who could perform the review.

It is very important that AEs do their best to identify the “right” Reviewers from the beginning, even if this means spending extra time before sending out review requests. This initial effort will almost certainly be repaid later by high-quality reviews that will ease the whole decision process. Often it is easier to recruit Reviewers whose self-interest is served by providing the review, e.g., the paper is on a topic that they would really like to see or say something about. Personalizing the review invitation letter, for example by commenting on why the selected Reviewer is the right choice for the paper, may increase the likelihood that the candidate Reviewer accepts the invitation. Having Reviewers who are interested in going over the manuscript alleviates the need to repeatedly send reminders and minimizes the likelihood that the AE must write detailed comments to supplement shallow and not-to-the-point reviews.

Generally, it is not recommended to ask another AE to serve as a Reviewer, unless that AE is especially well-positioned to review the paper.

Finally, it is the responsibility of the AE to oversee the review process and make sure that invited Reviewers quickly confirm (or decline) to provide a review, and that reviews are provided in due time. The AE should periodically check the status of each Reviewer (confirmed or not, review sent or not) and send personalized reminder emails via PaperPlaza as needed. Auto-generated reminder emails are typically sent from PaperPlaza, but such emails are less effective at generating a response.

Three high-quality reviews is the target. After receiving a review, it is the AE’s responsibility to check its quality. In case of a substandard review (e.g., too short, shallow, offensive, etc.), the AE should ask the Reviewer to update or correct the review. If this is not possible (e.g., the Reviewer refuses to improve the review), and three quality reviews have not been obtained, the AE should attempt to find an emergency Reviewer with a shorter deadline to prevent significant delays in generating a recommendation.

It is generally recommended that the AE not also serve as a Reviewer, since the AE can provide her/his opinions in the recommendation letter. In cases where the AE has expertise in the area of the paper, and obtaining a third quality review would unduly delay the recommendation, it is acceptable for the AE to serve as a third reviewer.

Reviewers should provide information to the authors that demonstrates errors in the paper or that will be useful in revision. Anything that would be useful to the authors should be in the portion of the review that is visible to the authors, not in the confidential comments to the Editorial Board.

Reviews should be specific. If the results are already known, references to earlier papers that contain these or similar results should be provided. If the reasoning is incorrect or vague, where and why should be indicated. If the paper should be rewritten, suggestions should be given as to which parts to delete, amplify, or modify.

Reviews should be constructive. The ultimate goal of the review process is to provide constructive feedback to the authors on how their work could be improved, not just by noting deficiencies but also by indicating how they might be mended, when possible. The AE is responsible for ensuring that reviews avoid any diminishing or disrespectful remarks, even if the review is negative.

One review structure that works well consists of:

• a summary of the paper’s contribution, to demonstrate the Reviewer’s understanding of the paper;
• high-level comments on topics such as the paper’s originality, relevance, importance, technical correctness, and organization; and
• low-level comments, such as typo corrections, comments on figures and grammar, etc.

To avoid confusion, the Reviewer’s acceptance or rejection recommendation should not be included in the comments to the authors.

Sometimes papers are passed on to senior graduate students for review. Some excellent reviews may result from this. It is important, though, that the quality and professionalism of the review be maintained, and the originally assigned Reviewer should check the review personally before submission. If needed, Reviewers may include mathematical formulas, figures, tables, etc., in a PDF file. In this case, the Reviewer should ensure that the PDF file does not include any metadata or other indications of the Reviewer’s identity. The PaperPlaza software scrubs metadata from PDF files, which may cause annotations (such as comment bubbles, highlights, etc.) to also be scrubbed, so comments should not be transmitted by annotations to the original paper.

Once a sufficient set of reviews has been obtained for a paper, the AE prepares a summary report that synthesizes these reviews and makes a specific publication recommendation to the Editor. In general, at least three thorough reviews are required to make a recommendation; however, if two thorough reviews are available, if these reviews have been provided by trusted experts, and if the two reviews are in agreement, then two reviews can be considered a sufficient basis for submitting a summary report and recommendation.

The summary report to the authors should be written in a form that allows the Editor to directly cut and paste it into the decision letter to the authors. The report should be impartial, to the point, and constructive. It should contain a brief summary of the paper, an analysis of the reviews and the AE’s own opinions, and the issues that the authors must address in a revision if the paper may receive a “revise and resubmit” or “conditionally accept” decision.  The comments to the authors should not give the AE’s publication recommendation, since the Editor could issue a different decision.  (The field confidential comments to the Editor is the place to mention the publication recommendation, if needed.)  If the recommended revisions include shortening the paper, specific suggestions should be given as to how to shorten the paper. A request to shorten the paper should be realistic.

The AE’s summary report should not include the publication recommendations of the reviewers, and it should include neither their ratings nor their review text.  The ratings and reviewer recommendations are for internal use only, and the reviewers’ reports to the authors are made available separately by PaperPlaza.

Remember, the AE’s job is not simply to parrot the Reviewers’ comments and average their recommendations. AEs have been chosen for their technical judgment, and this must be exercised when writing a summary report. Reviewers’ opinions may be in conflict, some comments are more important than others, and some comments may be incorrect or safely ignored. The AE’s job is to generate a focused synthesis of the most important points and to provide clear feedback to the authors. AEs are also encouraged to do a quick search in IEEE Xplore to see if relevant T-RL papers have been cited and to suggest the authors examine or cite these papers as appropriate.

In addition to the summary report, the AE makes a publication recommendation to the Editor. PaperPlaza codes for the common recommendation types are AR for Accept as Regular; CR for Conditionally Accept as Regular; RR for Revise and Resubmit as Regular; R for Reject; and RE for Editorial Reject.

3.5.1 Accept as Regular (AR)

When a paper is accepted, small modifications to the paper can be suggested but none is required prior to publication. The next step is for the authors to submit their final manuscript for proofing and publication. In practice, an Accept decision is rarely issued to the first version of a paper, but it is certainly allowed.

3.5.2 Conditional Accept as Regular (CR)

This decision indicates that the paper is not yet ready for publication, but that it requires only minor modifications. If a paper is conditionally accepted, a revised manuscript must be submitted within 60 days of the date of decision. The revision has the same paper ID and the additional review cycle should take no more than one month, requiring only the intervention of the previously assigned AE and, possibly, a subset of the previous reviewers (typically the ones asking for modifications).

The submission of a revision to a Conditionally Accepted paper must be accompanied by a separate file with a clear response to the AE’s summary and the reviews, including indications of how the paper has been modified in response.

3.5.3 Revise and Resubmit as Regular (RR)

This decision provides a way to reject a paper while indicating to the authors that it may contain publishable results after a major revision. A revised manuscript is assigned a new paper ID, it is treated as a new submission, and it must go through a full review process. A reference within the system to the original paper and its reviews is preserved to facilitate the new review process. The Editor and AE who handled the previous version are often assigned to the revision, but this is not guaranteed.

Repeated Revise and Resubmit decisions should be avoided; after a first Revise and Resubmit, the second decision should be either (conditional) acceptance or rejection. A second Revise and Resubmit iteration requires a strong justification.

The submission of a revision to a paper with a Revise and Resubmit decision must be accompanied by a separate file with a clear response to the AE’s summary and the reviews, including indications of how the paper has been modified in response.

Do not use the Revise and Resubmit recommendation as a “soft” rejection. The Revise and Resubmit recommendation should be reserved for papers with real promise to be accepted after an appropriate revision. If the paper lacks this promise, the recommendation should be Reject. If an AE issues a Revise and Resubmit recommendation, there is a good chance the AE will handle a future version of the paper.

Do not be biased to issue an Accept or Conditionally Accept recommendation to the revision of a paper that received a Revise and Resubmit decision. There is a natural tendency to “reward” authors with an Accept or Conditionally Accept decision if they have invested effort to revise their paper. There should be no such bias in reviewing the paper, which is effectively a new paper.

3.5.4 Reject (R)

T-RL is a comprehensive robotics journal with a limited page budget, and we can only publish novel papers that report on contributions of high research significance and impact, on topics that are of interest to the robot learning research community. Reject is the appropriate decision for papers that do not meet these criteria.

Nothing prevents authors from resubmitting a revised version of a rejected manuscript, but the Reject decision signals to authors that a mere revision is also likely to be rejected, so a revision is discouraged. If authors do submit a revision, the submission must be accompanied by a separate file with a clear response to the AE’s summary and the reviews, including indications of how the paper has been modified in response.

3.5.5 Editorial Reject (RE)

Editorial rejection decisions are to be used mainly for incomplete or otherwise clearly unsuitable papers (see the official IEEE policy for more details). All editorial rejections must be approved by the EiC. If the reason for editorial rejection is a lack of technical contribution, IEEE rules require that at least three editors concur in such a decision. For these cases, the EiC and at least two members of the Editorial board (usually the SE and AE) have to consent. If an AE comes to the conclusion that an assigned manuscript should be summarily rejected, the AE informs the corresponding SE with a sufficiently detailed justification. As always, the report should avoid any disrespectful remarks and should be authoritative in its reasoning and conclusion. If an SE thinks that a manuscript should be summarily rejected, the SE assigns it to an AE asking for the AE’s opinion. If both the AE and the SE agree that the paper should be summarily rejected, they generate a sufficiently detailed decision letter (typically drafted by the AE and revised by the SE). They then inform the EiC about their proposed decision, provide the EiC with a decision letter, and ask for approval. If the EiC, the SE, and the AE agree, the SE issues the editorial reject decision. The decision should not be executed without approval by the EiC. An editorial reject decision should be issued no later than three weeks after the submission. If the EiC, the SE, and the AE disagree in their assessment of the manuscript, it should be sent out for full review.