Mastodawn

AAAI, one of the leading AI research conferences, tested AI peer reviews for over 22k submissions. The authors received human reviews as well and then answered a survey on their preferences.

The findings are quite interesting:
"The large-scale survey of AAAI-26 authors, reviewers, senior program committee members, and area chairs found that participants broadly found AI reviews useful and preferred them to human reviews on key dimensions such as technical accuracy and research suggestions, but also identified some limitations and areas for improvement including technical errors in reading some equations and tables, difficulty in prioritizing the significance of issues, and producing reviews that were longer than readers preferred"

Critique by authors is also mentioned in the paper:
"Respondents also emphasized that AI reviews had the potential to mislead reviewers and other decision-makers in the review process. There were also concerns that authors might optimize papers for AI preferences rather than scientific quality, and that reliance on these tools could lead to a long-term decline in reviewing skill. Adding to this, many respondents voiced principled objections, arguing that the use of AI undermines the trust, human effort, and essential value of the peer review process."

Link: https://arxiv.org/abs/2604.13940
#PeerReview #aaai2026

Show thread

UKP Lab Feb 13

📜 Paper: https://arxiv.org/abs/2506.15498
🤖 Models: https://huggingface.co/collections/UKPLab/spare-prm
💻 Code: https://github.com/UKPLab/aaai2026-spare-prm

Follow the authors Imbesat Hassan Rizvi and Iryna Gurevych from the Ubiquitous Knowledge Processing Lab (UKP Lab), Technische Universität Darmstadt and Xiaodan Zhu from the Department of Electrical and Computer Engineering, Smith Engineering and Ingenuity Labs Research Institute at Queen's University.

#AAAI2026 #ProcessSupervision #Reasoning #RewardModelling #ReferenceGuidedEvaluation

SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modelling

Process or step-wise supervision has played a crucial role in advancing complex multi-step reasoning capabilities of Large Language Models (LLMs). However, efficient, high-quality automated process annotation remains a significant challenge. To address this, we introduce Single-Pass Annotation with Reference-Guided Evaluation (SPARE), a novel structured framework that enables efficient per-step annotation by jointly aligning solution steps to reference solutions and determine its accuracy with explicit reasoning in single generation. We demonstrate SPARE's effectiveness across four diverse datasets spanning mathematical reasoning (GSM8K, MATH), multi-hop question answering (MuSiQue-Ans), and spatial reasoning (SpaRP), showing consistent improvements in two applications: (1) training Process Reward Models (PRMs) for ranking and aggregating multiple generations, and (2) fine-tuning models via offline reinforcement learning for greedy decoding. On ProcessBench, SPARE demonstrates data-efficient out-of-distribution generalization, using only $\sim$16% of training samples compared to human-labeled and other synthetically trained baselines. Additionally, it achieves competitive performance with MCTS-based methods while offering 2.3$\times$ speedup in terms of total token count. Manual analysis reveals complementary precision-recall characteristics with MCTS approaches, suggesting potential for ensemble methods. These results establish SPARE as a practical and scalable solution for automatic process supervision in LLM reasoning.

arXiv.org

Ania Jacyszyn Jan 26

🎊 🇸🇬 Very successful second edition of the AI4SC bridge @ 40th AAAI! During the event we managed to bring together a community of researchers interested.

🥇 In between the sessions we also played a short game 🪨📜✂️.

Many thanks to:
💐 Our very active participants
💐 The keynote speaker, Mark Gahegan
💐 Authors and speakers
💐 Organising committee
💐 Program committee

🔜 AI4SC on Zenodo: https://zenodo.org/communities/ai4sc2026

@DiTraRe @soeren_auer

#ai4sc #ai4sc26 #ai4sc2026 #aaai26 #aaai2026 #research #conference

DICE Jan 22

We’re excited that our colleague Yasir Mahmood is presenting two papers at the AAAI Conference in Singapore!🚀

👉 "Structure-Aware Encodings of Argumentation Properties for Clique-width" by Yasir Mahmood, Markus Hecher, Johanna Groven & Johannes K. Fichte
👉 "Can You Tell the Difference? Contrastive Explanations for ABox Entailments" by Patrick Koopmann, Yasir Mahmood, Axel Ngona & Balram Tiwari

Wishing you a great time and many inspiring exchanges with the AI community!

#DICEontour #AAAI2026

FIZ KDAI Research Group Jan 21

Our colleague @AnnaJacyszyn is co-organising the AI4SC (AI for Scholarly Communication) Bridge event at #aaai2026 in Singapore.

https://sites.google.com/view/ai4sc/edition/ai4sc-2026-40th-aaai

#AI #digitalisation #scholarlydata #ditrare @DiTraRe @fiz_karlsruhe

Ania Jacyszyn Jan 21

💥 What an insightful keynote by Mark Gahegan at our AI4SC bridge in Singapore! Mark gave an outlook on a very broad topic on the influence of AI on the future of research: generative AI is probably the biggest disruption for researchers in their life time.

🪧 I highly recommend checking out Mark's presentation, it's already on Zenodo: https://zenodo.org/records/18309519

💐 Thank you Mark for the amazing talk!

#ai4sc #ai4sc26 #ai4sc2026 #aaai26 #aaai2026 #research #future #keynote

@soeren_auer @DiTraRe

Gautam Kamath Jan 19

The final invited talk at #AAAI2026 is by Derek Haoyang Li (Squirrel Ai Learning), on "Small Data: A New Paradigm for the Next Generation of AI." It will be on Sunday, January 25 at 2 PM. CC @[email protected] @[email protected]

Gautam Kamath Jan 19

The Patrick Henry Winston Outstanding Educator Award went to Alan Mackworth (@[email protected] UBC) and David Poole (@[email protected] UBC). At #AAAI2026, they'll be giving a talk "The Essence of Intelligence is Appropriate Action..." Sunday Jan 25 8:30AM @[email protected] @[email protected]

Gautam Kamath Jan 18

There will be a celebration of Edward Feigenbaum's 90th birthday at #AAAI2026, on Saturday, January 24 at ~5:30 PM. It will include a lecture “1956 to 2026: Highlights (and Advice) from 70 Years of Navigating the AI Spectrum.” CC @[email protected] @[email protected]

Gautam Kamath Jan 18

Ashok Goel (Georgia Tech) will be giving an invited talk at #AAAI2026 on Saturday, January 24 at 4:30 PM. The title will be "AI for Reskilling, Upskilling, and Workforce Development." Check it out! CC @[email protected] @[email protected]