Distinguishing lexical deviation from semantic resilience in LLM-based round-trip translation
DOI: https://doi.org/10.3846/ntcs.2026.26227Abstract
Round-trip translation (RTT) is widely used for data augmentation and paraphrasing, yet its impact on meaning preservation remains poorly understood. We evaluate RTT systematically across two phases totaling 80,000 translation operations: 200 synthetic English passages are routed through single- and multi-hop pipelines spanning Bulgarian, Russian, and Chinese, translated by four systems ranging from a classical web translator (googletrans) to recent large language models (4o-mini, Claude Haiku 4.5, GPT-5-nano), with every passage undergoing ten RTT passes scored by BLEU, COMET-Kiwi, and xCOMET, joined by SBERT in the second phase. The results show that RTT outputs remain fluent while rapidly losing meaning: even the best configuration reaches only 0.646 xCOMET, and the four systems follow three distinct degradation regimes (front-loaded damage, continuous drift, and deterministic fixedpoint convergence). High surface overlap masks this loss, creating a surface fidelity trap, and genre-level analysis suggests that creative and news texts are most vulnerable. Sample-level inspection further reveals that xCOMET over-penalizes legitimate paraphrastic variation while under-penalizing domain-critical substitutions in legal text, exposing a mismatch between translation-oriented metrics and paraphrase evaluation. These findings argue against RTT in meaning-sensitive workflows and call for paraphrase-specific evaluation metrics.
Keywords:
round-trip translation (RTT), machine translation evaluation, semantic preservation, paraphrasing, data augmentation, synthetic data, large language models (LLM), xCOMETHow to Cite
Share
License
Copyright (c) 2026 The Author(s). Published by Vilnius Gediminas Technical University.
This work is licensed under a Creative Commons Attribution 4.0 International License.
References
Amara, K., Sevastjanova, R., & El-Assady, M. (2024). Challenges and opportunities in text generation explainability. In L. Longo, S. Lapuschkin, & C. Seifert (Eds.), Communications in computer and information science: Vol. 2153. Explainable artificial intelligence (xAI 2024) (pp. 244–264). Springer. https://doi.org/10.1007/978-3-031-63787-2_13
Bannard, C., & Callison-Burch, C. (2005). Paraphrasing with bilingual parallel corpora. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (pp. 597–604). Association for Computational Linguistics. https://doi.org/10.3115/1219840.1219914
Barzilay, R., & McKeown, K. R. (2001). Extracting paraphrases from a parallel corpus. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics (pp. 50–57). Association for Computational Linguistics. https://doi.org/10.3115/1073012.1073020
Cao, J., Li, M., Li, Y., Wen, M., Cheung, S.-C., & Chen, H. (2022). SemMT: A semantic-based testing approach for machine translation systems. ACM Transactions on Software Engineering and Methodology, 31(2), 1–36. https://doi.org/10.1145/3490488
Corbeil, J.-P., & Abdi Ghavidel, H. (2021). Assessing the eligibility of backtranslated samples based on semantic similarity for the paraphrase identification task. In R. Mitkov & G. Angelova (Eds.), Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021) (pp. 301–308). INCOMA Ltd. https://doi.org/10.26615/978-954-452-072-4_035
Crone, N., Power, A., & Weldon, J. (2021). Quality estimation using round-trip translation with sentence embeddings. arXiv. https://doi.org/10.48550/arXiv.2111.00554
Denkowski, M., & Lavie, A. (2014). Meteor universal: Language specific translation evaluation for any target language. In O. Bojar, C. Buck, C. Federmann, B. Haddow, P. Koehn, C. Monz, M. Post, & L. Specia (Eds.), Proceedings of the Ninth Workshop on Statistical Machine Translation (pp. 376–380). Association for Computational Linguistics. https://doi.org/10.3115/v1/W14-3348
Edunov, S., Ott, M., Auli, M., & Grangier, D. (2018). Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 489–500). Association for Computational Linguistics. https://doi.org/10.18653/v1/D18-1045
Gašpar, A., & Knežević, M. (2026). Cross-lingual and cross-domain evaluation of ChatGPT’s translation performance. In D. Vasić & E. Brajković (Eds.), Communications in computer and information science: Vol. 2609. Digital transformation in education and artificial intelligence application: MoStart 2025 (pp. 196–211). Springer. https://doi.org/10.1007/978-3-032-02801-3_13
Guerreiro, N. M., Rei, R., van Stigt, D., Coheur, L., Colombo, P., & Martins, A. F. T. (2024a). XCOMET-XXL [Computer software]. Hugging Face. https://huggingface.co/Unbabel/XCOMET-XXL
Guerreiro, N. M., Rei, R., van Stigt, D., Coheur, L., Colombo, P., & Martins, A. F. T. (2024b). xCOMET: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12, 979–995. https://doi.org/10.1162/tacl_a_00683
Han, S. (2025). googletrans (Version 4.0.2) [Computer software]. GitHub. https://github.com/ssut/py-googletrans
Kopanov, K., & Atanasova, T. (2025). A comparative pattern analysis of Qwen 2.5 and Gemma 3 text generation. WSEAS Transactions on Information Science and Applications, 22, 604–615. https://doi.org/10.37394/23209.2025.22.50
Kopanov, K., & Atanasova, T. (2026). Degradation of stylometric attribution accuracy for AI-generated text. Engineering Proceedings, 150(1), Article 98. https://doi.org/10.3390/engproc2026150098
Mallinson, J., Sennrich, R., & Lapata, M. (2017). Paraphrasing revisited with neural machine translation. In M. Lapata, P. Blunsom, & A. Koller (Eds.), Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (vol. 1, pp. 881–893). Association for Computational Linguistics. https://doi.org/10.18653/v1/E17-1083
Mondal, S. K., Zhang, H., Kabir, H. M. D., Ni, K., & Dai, H.-N. (2023). Machine translation and its evaluation: A study. Artificial Intelligence Review, 56, 10137–10226. https://doi.org/10.1007/s10462-023-10423-5
Papineni, K., Roukos, S., Ward, T., & Zhu, W. J. (2002). BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (pp. 311–318). Association for Computational Linguistics. https://doi.org/10.3115/1073083.1073135
Quelle, D., Cheng, C. Y., Bovet, A., & Hale, S. A. (2025). Lost in translation: Using global fact-checks to measure multilingual misinformation prevalence, spread, and evolution. EPJ Data Science, 14, Article 22. https://doi.org/10.1140/epjds/s13688-025-00520-6
Quirk, C., Brockett, C., & Dolan, W. (2004). Monolingual machine translation for paraphrase generation. In D. Lin & D. Wu (Eds.), Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing (pp. 142–149). Association for Computational Linguistics.
Rei, R., Guerreiro, N. M., Pombal, J., van Stigt, D., Treviso, M., Coheur, L., de Souza, J. G. C., & Martins, A. F. T. (2023). Scaling up CometKiwi: Unbabel-IST 2023 submission for the quality estimation shared task. In Proceedings of the Eighth Conference on Machine Translation (WMT 2023), (pp. 841–848). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.wmt-1.73
Rei, R., Treviso, M., Guerreiro, N. M., Zerva, C., Farinha, A. C., Maroti, C., de Souza, J. G. C., Glushkova, T., Alves, D., Coheur, L., Lavie, A., & Martins, A. F. T. (2022). wmt22-cometkiwi-da [Computer software]. Hugging Face. https://huggingface.co/Unbabel/wmt22-cometkiwi-da
Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-Networks. In K. Inui, J. Jiang, V. Ng, & X. Wan (Eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 3982–3992). Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-1410
Sennrich, R., Haddow, B., & Birch, A. (2016). Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (vol. 1, pp. 86–96). Association for Computational Linguistics. https://doi.org/10.18653/v1/P16-1009
Thompson, B., & Post, M. (2020). Paraphrase generation as zero-shot multilingual translation: Disentangling semantic similarity from lexical and syntactic diversity. In Proceedings of the Fifth Conference on Machine Translation (pp. 561–570). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.wmt-1.67
Wieting, J., & Gimpel, K. (2018). ParaNMT-50M: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (vol. 1, pp. 451–462). Association for Computational Linguistics. https://doi.org/10.18653/v1/P18-1042
Zan, C., Ding, L., Shen, L., Zhan, Y., Yang, X., & Liu, W. (2025). Building accurate translation-tailored large language models with language-aware instruction tuning. Frontiers of Information Technology & Electronic Engineering, 26(8), 1341–1355. https://doi.org/10.1631/FITEE.2400458
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2020). BERTScore: Evaluating text generation with BERT. In Proceedings of the 8th International Conference on Learning Representations (ICLR 2020).
Zhuo, T. Y., Xu, Q., He, X., & Cohn, T. (2023). Rethinking round-trip translation for machine translation evaluation. In Findings of the Association for Computational Linguistics: ACL 2023 (pp. 319–337). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.findings-acl.22
Zou, W., Yang, S., Bao, Y., Huang, S., Chen, J., & Cheng, S. (2025). Trans-zero: Self-play incentivizes large language models for multilingual translation without parallel data. Findings of the Association for Computational Linguistics: ACL 2025 (pp. 12337–12347). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.findings-acl.637
View article in other formats
CrossMark check
Published
Issue
Section
Copyright
Copyright (c) 2026 The Author(s). Published by Vilnius Gediminas Technical University.
License
This work is licensed under a Creative Commons Attribution 4.0 International License.