A comparative quality assessment of ChatGPT-4 and human translation of scientific texts

Abstract

You may not believe but we all have already surrounded by different technologies and have close connection with AI (Artificial Intelligence). For instance, we follow the needed destination using the voice-guided navigation of car’s GPS (Global Positioning System) or ask Alice or Siri voice assistants to dial number from the friends’ list. In various fields, we try to create automated advanced systems and applications to reduce our work. We have created robotic vacuum cleaners, wash machines, “smart house” systems, which distantly could be controlled by phone or PC, and we go forward, working on 3D printed Animal and Human Prosthetics to make happier disabled people. Undoubtedly, the Translation Science also undergoes significant changes and we are all witnesses to the inventions that people have waited for and tried to achieve for several decades. In 2022, the world appreciated for the large language model ChatGPT-3.5, one year later for ChatGPT-4, introduced and intensively developed by OpenAI company. These Generative Pretrained Transformer (GPT) language models trained on a huge network dataset, can generate texts, the latest language models translate myriad languages, solve math problems, answer questions, create imagines and just can be a good interlocutor, carrying on a dialogue. This paper aims to examine and compare translations of scientific texts from English to Russian performed by ChatGPT-3.5, ChatGPT-4, and human translators. The study conducts a comparative quality analysis of these translations and utilizes the neural-based machine translation evaluation metrics COMET-22 and BLEURT-20. A key goal of this paper is to identify errors, and shed the light on the strengths and limitations primarily of ChatGPT-4 model. Additionally, the paper finds out improvements of ChatGPT-4 capabilities and difference with its predecessor ChatGPT-3.5, and estimates their performance in comparison to human translation. Using mixed-method empirical research, this study analyzed 30 scientific articles, assessing the efficiency and capability of GPT models.