Mastodawn

📣🧪 𝗡𝗲𝘄 𝗿𝗲𝘀𝘂𝗹𝘁𝘀: 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗻𝗴 𝗦𝗰𝗶𝗲𝗻𝘁𝗶𝗳𝗶𝗰 𝗪𝗿𝗶𝘁𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝗘𝘅𝗽𝗲𝗿𝘁 𝗣𝗿𝗲𝗳𝗲𝗿𝗲𝗻𝗰𝗲𝘀

👉 Today we are presenting our first two papers that build on HAI-Co2, our recent position paper on human-ai text co-construction in expert domains:

Scientific writing is an open-end expert task utilizing user preferences. Evaluating it means capturing both general and expert-specific quality criteria. Our new works push forward preference-aligned evaluation.

(1/🧵)

Show thread

UKP Lab Feb 26

🧾 𝗥𝗲𝘄𝗮𝗿𝗱 𝗠𝗼𝗱𝗲𝗹𝗶𝗻𝗴 𝗳𝗼𝗿 𝗦𝗰𝗶𝗲𝗻𝘁𝗶𝗳𝗶𝗰 𝗪𝗿𝗶𝘁𝗶𝗻𝗴 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻
Furkan Şahinuç, Subhabrata Dutta, Iryna Gurevych

https://arxiv.org/abs/2601.11374

(2/🧵 )

Reward Modeling for Scientific Writing Evaluation

Scientific writing is an expert-domain task that demands deep domain knowledge, task-specific requirements and reasoning capabilities that leverage the domain knowledge to satisfy the task specifications. While scientific text generation has been widely studied, its evaluation remains a challenging and open problem. It is critical to develop models that can be reliably deployed for evaluating diverse open-ended scientific writing tasks while adhering to their distinct requirements. However, existing LLM-based judges and reward models are primarily optimized for general-purpose benchmarks with fixed scoring rubrics and evaluation criteria. Consequently, they often fail to reason over sparse knowledge of scientific domains when interpreting task-dependent and multi-faceted criteria. Moreover, fine-tuning for each individual task is costly and impractical for low-resource settings. To bridge these gaps, we propose cost-efficient, open-source reward models tailored for scientific writing evaluation. We introduce a two-stage training framework that initially optimizes scientific evaluation preferences and then refines reasoning capabilities. Our multi-aspect evaluation design and joint training across diverse tasks enable fine-grained assessment and robustness to dynamic criteria and scoring rubrics. Experimental analysis shows that our training regime strongly improves LLM-based scientific writing evaluation. Our models generalize effectively across tasks and to previously unseen scientific writing evaluation settings, allowing a single trained evaluator to be reused without task-specific retraining.

arXiv.org

Show thread

UKP Lab Feb 26

📑 𝗘𝘅𝗽𝗲𝗿𝘁 𝗣𝗿𝗲𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗯𝗮𝘀𝗲𝗱 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗼𝗳 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗲𝗱 𝗥𝗲𝗹𝗮𝘁𝗲𝗱 𝗪𝗼𝗿𝗸 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻
Furkan Şahinuç, Subhabrata Dutta, Iryna Gurevych

https://arxiv.org/abs/2508.07955

(3/🧵)

Expert Preference-based Evaluation of Automated Related Work Generation

Expert domain writing, such as scientific writing, typically demands extensive domain knowledge. Although large language models (LLMs) show promising potential in this task, evaluating the quality of automatically generated scientific writing is a crucial open issue, as it requires knowledge of domain-specific criteria and the ability to discern expert preferences. Conventional task-agnostic automatic evaluation metrics and LLM-as-a-judge systems, primarily designed for mainstream NLP tasks, are insufficient to grasp expert preferences and domain-specific quality standards. To address this gap and support realistic human-AI collaborative writing, we focus on related work generation, one of the most challenging scientific tasks, as an exemplar. We propose GREP, a multi-turn evaluation framework that integrates classical related work evaluation criteria with expert-specific preferences. Our framework decomposes the evaluation into smaller fine-grained dimensions. This localized evaluation is further augmented with contrastive examples to provide detailed contextual guidance for the evaluation dimensions. Empirical investigation reveals that our framework is able to assess the quality of related work sections in a much more robust manner compared to standard LLM judges, reflects natural scenarios of scientific writing, and bears a strong correlation with the assessment of human experts. We also observe that generations from state-of-the-art (SoTA) LLMs struggle to satisfy validation constraints of a suitable related work section.

arXiv.org

Show thread

UKP Lab

💻 Project page:
https://ukplab.github.io/arxiv2025-expert-eval-rw/

🔁 Both papers build on HAI-Co2 described in this position paper:
https://direct.mit.edu/coli/article/51/4/1337/132031/Problem-Solving-Through-Human-AI-Preference-based

(4/4)

#NLP #NLProc #LLMs #Evaluation #PreferenceLearning #ScientificWriting #HumanAI #HaiCo2

SOCIAL MEDIA TITLE TAG

SOCIAL MEDIA DESCRIPTION TAG TAG