Reviewers express concerns over inconsistent scoring and potential biases in the peer review process.
Category: Science
A post on r/MachineLearning that gained traction with over 90 upvotes and 50 comments highlights the stark variance in review scores for papers submitted to the International Conference on Machine Learning (ICML) 2026. Users are raising alarms about the reliability of the peer review process, particularly after observing discrepancies in scores before and after rebuttals.
Reviewers have noted a troubling trend where pre-rebuttal scores often hover below 3.5, but many scores jump significantly after authors submit their rebuttals. One commenter, u/tariban, reported that in their batch, pre-rebuttal scores were mostly 3.5 or lower, but post-rebuttal scores surged, with almost half reaching above 4. They expressed concern that reviewers appeared to disregard the significance of the papers, accepting dubious claims made during rebuttals.
Another user, u/Derpirium, echoed these sentiments, stating that most papers they reviewed received scores below 3.5, with only two getting accepted. They felt that a poorly rated paper was favored by other reviewers, which left them puzzled about the scoring system's fairness. This sentiment was shared by u/Aromatic-Low-5032, who mentioned that out of six papers reviewed, only one exceeded a score of 3.5, indicating a harsh review environment.
The inconsistency in scoring has been attributed to various factors, including the expertise mismatch among reviewers. User u/CriticalCup6207 pointed out that this issue has been recognized for years and is exacerbated by the increasing volume of submissions. They noted that score variance is systematically linked to area chair assignments and the quality of the reviewer pool, which can vary significantly across different areas of research.
Concerns about the peer review process aren't new. User u/claudiollm commented that variance in scores at prominent conferences has long been an issue, often depending on luck related to the reviewer pool and whether the area chair calibrates the scores effectively. They cited an example from their lab where three reviewers initially scored a paper between 3.25 and 3.5, but two later increased their scores to 4 without engaging deeply with the rebuttal content.
As the ICML conference approaches, the community is urging for more transparency and consistency in the review process. The discussion suggests that the current model may not adequately serve the growing number of submissions, leading to frustrations among authors and reviewers alike. With the upcoming conference, these issues have sparked a broader conversation about the structural integrity of the peer review system in machine learning.
One commenter noted that the high variance in scores could discourage researchers from submitting their work, fearing rejection based on arbitrary reviewer preferences. Another user argued that the selection of reviewers should be more strategic, ensuring they have relevant expertise to evaluate submissions accurately.
A top-voted reply pointed out that the peer review system might need a complete overhaul to address these systemic issues, advocating for a more standardized scoring approach. Some users suggested implementing a double-blind review process to minimize biases and improve fairness.
Others highlighted the importance of accountability in the review process, calling for measures that would require reviewers to justify their scores more thoroughly. This could potentially lead to more consistent evaluations and a fairer system for all authors.
The ICML is one of the leading conferences in machine learning, attracting thousands of submissions each year. As the field continues to grow, the demand for rigorous, fair peer review processes becomes increasingly important. Discrepancies in scoring can have consequences for researchers' careers, funding opportunities, and the advancement of knowledge in the field.
Experts have noted that the current peer review model may not be sustainable with the increasing number of submissions. A study published in the journal *Nature* indicated that as the volume of research increases, maintaining high standards in peer review becomes more challenging, often leading to inconsistencies like those observed in the ICML reviews.
The discussions surrounding the ICML 2026 review process reveal a growing concern within the machine learning community about the integrity and reliability of peer reviews. If these issues are not addressed, they could undermine the credibility of the conference and the research it promotes. As the community pushes for improvements, it how these changes will impact future submissions and the peer review process as a whole.