Reviewer Information

Thank you for agreeing to serve as a reviewer for CoRL 2023!

This page provides an overview of reviewer responsibilities and key dates.

General Information

The CoRL review process is about three months long, submission-to-notification, including the discussion period.

As the first step, please update your OpenReview profile before the submission deadline.

We count on you, the Reviewers, to make the process smooth and ensure high quality of the accepted submission, and provide constructive feedback to the authors. We want to ensure an active and constructive engagement with the authors to improve the papers and update the reviews and scores during the discussion period.

The Area Chair (AC) assigned to a paper should be your first point of contact for that paper. You can contact the AC by leaving a comment in OpenReview with the AC as a reader. (PCs will also be listed as readers, but will not be notified.)

Important: The papers under review are strictly confidential. Do not share or discuss the papers except for your Area Chair and the Program Chairs.

If you encounter a situation that you are unable to resolve with your AC, please contact the program chairs at pc@corl2023.org.

Timeline:

Below is the overall timeline of your responsibilities in the next three months. All times are 23:59 Pacific Time, unless otherwise stated. You will receive information for each step and technical instructions by Email via the OpenReview system.

Phase 1: Review period

  • June 9-11: Reviewer bidding period

  • June 19: Paper assignments are available

  • July 19: Review deadline

Phase 2: Discussion period

  • August 3-15: Author / reviewer discussion period 

Phase 3: Decision period

  • August 16 - August 21: AC / reviewer discussion

Review writing guidelines

Summary of Paper: The first section asks you to re-express the paper’s position clearly and fairly. Avoid copy-and-paste from the paper’s abstract or introduction and instead try to re-state the paper’s contributions and ideas in your own words and view. This is most valuable to the ACs to assess how others understand the contributions of the paper.

Main Review: The main review part of the review form asks for an evaluation of the originality, quality, clarity, significance, relevance and limitations of this work:

  • Originality: Are the problems or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited?

  • Quality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?

  • Clarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a well written paper provides enough information for an expert reader to reproduce its results.)

  • Significance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult problem in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide a unique theoretical or experimental approach?

  • Relevance: Does the paper address a problem in robot learning? Does it apply or improve a learning-based method, or a learned model? Is the method evaluated in simulation, on real robots, or both? Will the CoRL audience be interested in reading this paper? 

  • Limitations: CoRL papers are required to include a limitations section. Please treat the honestly reported limitations kindly and with high appreciation. On the other hand, obvious shortcomings or flaws not reported by the authors should be pointed out extensively and be strongly reflected in your scores. 

Generally, be constructive and concrete; be courteous and respectful. Start with the positive aspects, suggest improvements, and accept diverse views in a discussion (e.g. agree to disagree). Further, make your review as informative and substantiated as possible. The following gives examples for reviewer mistakes to avoid:

  • Reviewer states “This is not novel.” or “This has been done before.” without evidence. → Provide citations of the prior work.

  • Reviewer states “I believe the problem could better be solved with another approach.” → Provide evidence, e.g. citations of prior work that shows that this problem can better be solved with another approach.

  • Reviewer states “You must publish your dataset and code!” or “You must have a theorem!” or “You must demonstrate on a real robot!” → None of these are strict policies and sufficient reasons for rejection.

  • “The paper does not beat SOTA, so it must be rejected!” → Not necessarily, if it presents novel perspectives, insights, and sufficient knowledge advancement. 

  • “The paper beats SOTA, so it must be accepted!” → Not necessarily, if it does NOT provide novel perspectives, insights, or sufficient knowledge advancement.

  • “There is this error, hence it should be rejected!” → Is the error making the main knowledge advancement invalid?


ICML (tutorial, guidelines) and NeurIPS provide extensive guidance to reviewers on how to write constructive reviews, which create value to both the authors and our community to advance our field. Our guidelines build on these and we highly recommend these resources. 


Instructions for Area Chairs

The CoRL review process is about three months long, submission-to-notification, including the discussion period.

We count on you, the Area Chairs, to make the process smooth and ensure high quality of the accepted submission, and constructive feedback to the authors. Most notable change, we want to ensure three high-quality reviews by the time the reviews are released to the authors on Aug 30. 

Timeline:

All times are 11:59 PT, unless otherwise stated. Please adhere to the schedule, and if there are problems, do let us know as soon as possible.

Phase 1: Prepare for the review period

  • June 9-11: AC and Reviewer bidding period

  • June 12-16: ACs review the assignments for the expertise and conflict of interest, recommend desk rejects, and assign the reviewers

  • June 16-18: ACs finalize the reviewer assignment. PC and AC discuss desk reject candidates. 

Phase 2: Prepare for the discussion period

  • July 20-23: AC’s check the quality of the reviews, ask for updates, and recruit emergency reviewers.

  • July 24-31: Emergency review period

  • July 24-August 2: Write initial meta-reviews

Phase 3: Finalize the decisions

  • August 3-15: Monitor author / reviewer discussions

  • August 16-25: AC / reviewer discussion; finalizing meta-reviews

Contact:

pc@corl2023.org

Thank you so much for your service and helping make CoRL a high-quality and successful conference.