Axel Pichler📄 Our new paper – with Martin Endres & @nilsreiter – is now published in the Journal of Computational Literary Studies!
We present a workflow to evaluate LLM-generated poem interpretations using standards and argumentative structures from literary studies, showing strengths in descriptive analysis and limitations in producing acceptable rules of inferences.
👉 https://jcls.io/article/id/4312/
#DigitalHumanities #ComputationalLiteraryStudies #LLMs
We present a workflow to evaluate LLM-generated poem interpretations using standards and argumentative structures from literary studies, showing strengths in descriptive analysis and limitations in producing acceptable rules of inferences.
👉 https://jcls.io/article/id/4312/
#DigitalHumanities #ComputationalLiteraryStudies #LLMs
Interpretation, Argument, Evaluation. A Workflow for Assessing LLM-Generated Interpretations of Poetry
This paper examines how interpretations of poems generated by LLMs can be evaluated in a way that meets standards from literary studies. To this end, we develop and evaluate a workflow that draws on reference data from literary studies and their argumentative structures when generating interpretations. This enables the generation of interpretations that themselves exhibit such structures and can be evaluated with respect to both their argumentative coherence and literary scholarship standards. Our experiments demonstrate that this workflow can be applied successfully, and that the model under investigation generate reasonable descriptions of the poems, but fail at more abstract interpretative tasks.