Machine translation is free; but is it up to snuff, or do you get what you pay for?

SAECULII Y.K. (Tokyo, Japan) every four years runs an experiment to determine if machines translation (MT) is capable of producing quality, professional Japanese translation. Read more about that here

The free online translator -AKA machine translation- we tested in our first experiment was Babel Fish Online Translator. That was 2011.

Google Translate – Moving on to Bigger n’ Better Things

Feedback on the 2011 experiment was varied, ranging from interesting demonstration of the (poor) accuracy of machine translation to claims of unfairness for singling out one machine translation vendor, and the technology is still in its infancy so it’s to be expected, etc.

Overwhelmingly, though, most wanted to know where machine translation accuracy stands today. Many folk reason that since the demand for machine translation -the global machine translation market is expected to grow 23.19% to USD6.9 billion by 2019- is increasing, accuracy must also be increasing (otherwise there would be no demand, right?). These are all valid points.

So, this year -2015- we decided to re-run this experiment using Google Translate

Machine Translation Experiment Methodology

We took the following passage from the business article As Kindle Fire Faces Critics, Remedies Are Promised (Yahoo!, 2011-12-14):

The Kindle Fire, Amazon’s heavily promoted tablet, is less than a blazing success with many of its early users.

We ran this passage through machine translation for:

1. English-to-Japanese Translation (i.e., translate to get Japanese text), then
2. Japanese-to-English Translation (i.e., reverse translate back to English text).

Why do we reverse the translation?

The thinking is that if machine translation is any good -efficacy- the reverse translation should be the same, or very similar, to the original English passage. This approach will help folks that do not speak Japanese determine for themselves just how good machine translation really is.

Does Machine Translation Produce Professional Japanese Translation?

Read on.

Step 1: English-to-Japanese Translation
First, we translate the English passage to get Japanese text.

This is what your Japanese target audience see.

Step 2: Japanese-to-English Translation
Next, we reverse translate the Japanese text back into English text.

If machine translation produces professional quality translation, we should get the original English text, or something pretty close, right?

The verdict of our original experiment is gibberish.

In the intervening four years, not much has changed in terms of machine translation accuracy…Indeed, even the utility of “gisting” is lacking. Let’s call it what is folks:

Still gibberish, pure and simple!

Machine translation vendors regularly make claims of 70, 80 and even 85% accuracy. I’m sure you can see from the result above that that is simply not the case. And, therefore, by extension the accuracy of translation methodologies underpinned by machine translation, such as post-editing machine translation (PEMT), is also called into question.

Do you think there will be a significant improvement in accuracy by the time the end of the decade rolls around? Let us know what you think in the comments section below!

(Oh, and we’ll keep you posted when we re-run our experiment…about 2019.)

---

About the Author
Ivan Vandermerwe is the CEO of SAECULII YK, the owner of Tokyo based Translation Company Tokyo Visit SAECULII for the latest professional case studies, articles and news on Japanese Translation Services

Copyright (C) SAECULII YK. All rights reserved. Reproduction of this article is permitted with inclusion of the “About the Author” reference as is (including text links, japan-translators.saeculii.com/english/info/contact.cfm), and this copyright information. Articles may not be altered without written permission from SAECULII YK.