Evaluating machine translation of the Shan Hai Jing: an MQM-based analysis of google translate vs. ChatGPT with prompting effects

Culturally dense classical texts pose persistent challenges for machine translation, particularly in reconstructing compressed semantic hierarchies and culture-specific references. Although Neural Machine Translation (NMT) and Large Language Models (LLMs) have substantially improved fluency and contextual coherence, previous studies have given limited attention to the evaluation of their performance on culturally embedded classical texts using MQM-based human evaluation alongside automatic metrics. Focusing on the English translation of the Shan Hai Jing, a culturally dense and semantically complex classical Chinese text, this study investigates whether different translation systems produce distinct error patterns in culturally compressed contexts and whether prompting strategies influence translation performance. Selected textual segments were translated using Google Translate and ChatGPT under minimal and enriched prompting strategies. Translation quality was assessed through MQM-based human evaluation alongside several automatic metrics (BLEU, chrF, BERTScore, and COMET-Kiwi). MQM analysis reveals clear differences in error patterns across systems: NMT outputs show a higher incidence of high-severity mistranslations, whereas LLM outputs tend to exhibit semantic generalisation and shifts in cultural references. By contrast, automatic metrics show limited differentiation in system rankings, with no significant main effect of system observed. Prompt enrichment does not produce consistent quality improvements and occasionally increases semantic drift. These findings suggest that translation quality in culturally compressed texts may be better interpreted through structural error patterns across MQM dimensions rather than metric-based rankings alone. Evaluation sensitivity appears to be shaped by text type, and increased prompt complexity does not necessarily enhance semantic precision in classical translation tasks.

Text 126330.pdf - Published Version Download (1MB) Official URL or Download Paper: https://ejournal.ukm.my/gema/article/view/99954

Actions (login required)

View Item