Jindřich LibovickýMy master student Lukáš Eigler just defended his thesis (co-supervised with David Hurych from Valeo.ai) 🎉 Congrats!
#NLP metric validation needs 🐌💰 human judgment data. Our fix: generate synthetic data for metric validation instead. ✅ Tested on MT, QA, summarization.
To appear at #ACL2026 Student Research Workshop:
https://arxiv.org/abs/2603.09403
LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation
Validating evaluation metrics for NLG typically relies on expensive and time-consuming human annotations, which predominantly exist only for English datasets. We propose LLM as a Meta-Judge, a scalable framework that utilizes LLMs to generate synthetic evaluation datasets via controlled semantic degradation of real data, replacing human judgment. We validate our approach using meta-correlation, measuring the alignment between metric rankings derived from synthetic data and those from standard human benchmarks. Experiments across Machine Translation, Question Answering, and Summarization demonstrate that synthetic validation serves as a reliable proxy for human judgment, achieving meta-correlations exceeding 0.9 in multilingual QA and proves to be a viable alternative where human judgments are unavailable or too expensive to obtain. Our code and data are publicly available at https://github.com/eiglerl/meta-judge.
UKP Lab
Sophie Schneider
kristallpirat
Tatjana Scheffler
Long Live Moonjoy
SIGGEN
Michal Ptaszynski
Benjamin Han
