NMT/LLM PE and Video game Localisation: Can Visual Novels Benefit from Machine Translation?

This exploratory paper presents one of the first comparative studies of PE (post-editing) and HT (human translation) quality in the field of video game localisation. We used an excerpt from the KillerTrait free visual novel, selected for its particular features in terms of gender, puns and oral language. This excerpt was submitted to the DeepL and ChatGPT MT engines, and these versions were respectively post-edited from English into French by nine and ten last-year translation students in the framework of their EN > FR post-editing class at the University of Mons, Belgium. Nine other students from the group translated the text from scratch. In these productions, we analysed fidelity and fluency. We compared the results from human quality evaluation to the HTER and BLEU automatic metrics. Human evaluation shows that the texts post-edited using ChatGPT display better PE quality scores than those post-edited using DeepL in terms of fidelity. Human translation was evaluated differently by both evaluators, thus leading to an impossibility to identify any trend. In terms of fluency, no clear trend can be found. Automatic metrics show that 1) no clear trend emerges in terms of raw MT quality, 2) PE effort is slightly lower (and quality supposedly higher) in the case of the raw ChatGPT output, and 3) no trend can be found in terms of quality when comparing the PE texts produced using each MT engine, but HT displays a higher internal variability, which could indicate that HT provides for less homogeneous productions than PE.