Our team was tasked with training a neural machine translation engine using Microsoft Custom Translator. Our language pair was English to Russian. Based on a case study we read, we decided that a suitable candidate for this kind of pilot project was the press releases featured on the website of the U.S. Embassy in Moscow, due to the fact that they are informative texts intended for a general audience. We outlined these goals in our pilot project proposal.
Our hypothesis was correct, as we were able to train the engine to translate texts, with the end result requiring a minimum amount of post-editing. We still encountered issues along the way, which are explained in our updated proposal and presentation on lessons learned. While our results were excellent, we ended up having to add parallel corpus data from the United Nations in order to have enough data. We could have had even better results with data pulled entirely from the State Department. In addition, we found that the BLEU score was not an entirely accurate representation of translation quality. A major issue in our test translation is that proper nouns were being translated when they should not be, i.e., “Huntsman,” as in “Jon Huntsman,” was being rendered as the Russian word for “hunter.” On the advice of an industry expert, we added a glossary, and while the BLEU score for this round was slightly lower, having the names of important people and organizations greatly reduced post-editing time.
Overall, this project provided valuable experience in training a neural machine translation engine. Our results show the important of domain when deciding whether a project is suitable for this kind of machine translation, and also the limitations of automated evaluation methods like a BLEU score in comparison to human evaluation.