IFRAME: //www.googletagmanager.com/ns.html?id=GTM-TRBQMN
MIT Technology Review
Hello,
We noticed you're browsing in private or incognito mode.
To continue reading this article, please exit incognito mode or log in.
Not an Insider? Subscribe now for unlimited access to online articles.
Subscribe today
Why we made this change
Visitors are allowed 3 free articles per month (without a
subscription), and private browsing prevents us from counting how many
stories you've read. We hope you understand, and consider subscribing
for unlimited online access.
Back to MIT Technology Review home
Contact customer service if you are seeing this message in error.
MIT Technology Review (BUTTON) Menu
* Topics
+
o Business Impact
o Connectivity
o Intelligent Machines
o Rewriting Life
o Sustainable Energy
+
o 10 Breakthrough Technologies
o 35 Innovators Under 35
o 50 Smartest Companies
+ Views
+ Views from the Marketplace
* The Download
* Magazine
* Events
* More
+ Video
+ Special Publications
+ MIT News magazine
+ Newsletters
+ Help/Support
+ Advertise with Us
* Log in / Create and account
* Subscribe
* Log in / Create an account
* Search
* ____________________ Submit
Click search or press enter
Business Impact
Human translators are still on top—for now
Machine translation works well for sentences but turns out to falter at the
document level, computational linguists have found.
* by Emerging Technology from the arXiv
* September 5, 2018
*
*
*
*
*
*
*
*
*
*
*
*
You may have missed the popping of champagne corks and the shower of
ticker tape, but in recent months computational linguists have begun to
claim that neural machine translation now matches the performance of
human translators.
Recommended for You
1. A neural network can learn to organize the world it sees into
concepts—just like we do
2. One day your voice will control all your gadgets, and they will
control you
3. Never mind killer robots—here are six real AI dangers to watch out
for in 2019
4. Google Assistant now comes with a real-time translator for 27
languages
5. When Chinese hackers declared war on the rest of us
The technique of using a neural network to translate text from one
language into another has improved by leaps and bounds in recent years,
thanks to the ongoing breakthroughs in machine learning and artificial
intelligence. So it is not really a surprise that machines
have approached the performance of humans. Indeed, computational
linguists have good evidence to back up this claim.
But today, Samuel Laubli at the University of Zurich and a couple of
colleagues say the champagne should go back on ice. They do not dispute
their colleagues’ results but say the testing protocol fails to take
account of the way humans read entire documents. When this is assessed,
machines lag significantly behind humans, they say.
[machine-translation.png?sw=600&cx=0&cy=0&cw=1420&ch=53
5]
At issue is how machine translation should be evaluated. This is
currently done on two measures: adequacy and fluency. The adequacy of a
translation is determined by professional human translators who read
both the original text and the translation to see how well it expresses
the meaning of the source. Fluency is judged by monolingual readers who
see only the translation and determine how well it is expressed in
English.
Computational linguists agree that this system gives useful ratings.
But according to Laubli and co, the current protocol only compares
translations at the sentence level, whereas humans also evaluate text
at the document level.
So they have developed a new protocol to compare the performance of
machine and human translators at the document level. They asked
professional translators to assess how well machines and humans
translated over 100 news articles written in Chinese into English. The
examiners rated each translation for adequacy and fluency at the
sentence level but, crucially also at the level of the entire document.
The results make for interesting reading. To start with, Laubli and co
found no significance difference in the way professional translators
rated the adequacy of machine- and human-translated sentences. By this
measure, humans and machines are equally good translators, which is in
line with previous findings.
However, when it comes to evaluating the entire document, human
translations are rated as more adequate and more fluent than machine
translations. “Human raters assessing adequacy and fluency show a
stronger preference for human over machine translation when evaluating
documents as compared to isolated sentences,” they say.
The researchers think they know why. “We hypothesise that
document-level evaluation unveils errors such as mistranslation of an
ambiguous word, or errors related to textual cohesion and coherence,
which remain hard or impossible to spot in a sentence-level
evaluation,” they say.
For example, the team gives the example of a new app called “微信挪 车,”
which humans consistently translate as “WeChat Move the Car” but which
machines often translate in several different ways in the same article.
Machines translate this phrase as “Twitter Move Car,” “WeChat mobile,”
and “WeChat Move.” This kind of inconsistency, say Laubli and co, makes
documents harder to follow.
This suggests that the way machine translation is evaluated needs to
evolve away from a system where machines consider each sentence in
isolation.
“As machine translation quality improves, translations will become
harder to discriminate in terms of quality, and it may be time to shift
towards document-level evaluation, which gives raters more context to
understand the original text and its translation, and also exposes
translation errors related to discourse phenomena which remain
invisible in a sentence-level evaluation,” say Laubli and co.
That change should help machine translation improve. Which means it is
still set to surpass human translation—just not yet.
Ref: arxiv.org/abs/1808.07048 : Has Machine Translation Achieved Human
Parity? A Case for Document-level Evaluation
Learn from the humans leading the way in emerging technology at EmTech
Next. Register Today!
June 11-12, 2019
Cambridge, MA
Register now
(BUTTON)
(BUTTON)
Share
*
*
*
*
*
*
Tagged
emerging technology, arXiv
Emerging Technology from the arXiv
Emerging Technology from the arXiv
Emerging Technology from the arXiv covers the latest ideas and
technologies that appear on the Physics arXiv preprint server. It is
part of the Physics arXiv Blog. Email:… More KentuckyFC@arxivblog.com
Subscribe to the Physics arXiv Blog RSS Feed.
Related Video
More videos
[Xb-logo-circle.png?sw=75]
Business Impact
Finding the balance of human intelligence and artificial intelligence
00:53
[Xb-logo-circle.png?sw=75]
Business Impact
How does the customer experience change when you're in a world of
conversation? 00:39
[Xb-logo-circle.png?sw=75]
Business Impact
Trump's Deputy CTO on immigrant workers 02:27
[Xb-logo-circle.png?sw=75]
Business Impact
A View from the White House 23:50
Recommended for You
1. A neural network can learn to organize the world it sees into
concepts—just like we do
2. One day your voice will control all your gadgets, and they will
control you
3. Never mind killer robots—here are six real AI dangers to watch out
for in 2019
4. Google Assistant now comes with a real-time translator for 27
languages
5. When Chinese hackers declared war on the rest of us
More from Business Impact
How technology advances are changing the economy and providing new
opportunities in many industries.
* The Best of the Physics arXiv (week ending January 12, 2019)
This week’s most thought-provoking papers from the Physics arXiv.
by Emerging Technology from the arXiv
* This algorithm browses Wikipedia to auto-generate textbooks
Wikipedia is a valuable resource. But it’s not always obvious how
to collate the content on any given topic into a coherent whole.
by Emerging Technology from the arXiv
* The state of artificial intelligence
AI technologies are coming into mainstream business usage—but a
host of challenges remains. An interactive infographic illustrates
the opportunities and the hurdles.
by MIT Technology Review Insights
More from Business Impact
From Our Advertisers
* In association with VMware
Digital transformation sparks innovation in networking
* In association with Google
For data-savvy marketers, there’s a new keyword: Intent
* In association with Google
Machine learning teaches marketers to cultivate a growth mindset
* In association with Oracle and Intel
Machine learning - driven analytics: Key to digital transformation
Want more award-winning journalism? Subscribe to Insider Basic.
* Insider Basic {! insider.prices.basic !}*
{! insider.display.menuOptionsLabel !}
Six issues of our award winning print magazine, unlimited online
access plus The Download with the top tech stories delivered daily
to your inbox.
{! insider.buttons.basic.buttonText !}
See details+
Print Magazine (6 bi-monthly issues)
Unlimited online access including all articles, multimedia, and
more
The Download newsletter with top tech stories delivered daily to
your inbox
* {! insider.display.footerLabel !}
See international prices
See U.S. prices
Revert to MIT Enterprise Forum pricing
Revert to standard pricing
Clocking In A look into how technology is shaping the workplace of the
future
By signing up you agree to receive email newsletters and notifications
from MIT Technology Review. You can change your preferences at any
time. View our Privacy Policy for more detail.
Follow us
Twitter Facebook RSS
MIT Technology Review
The mission of MIT Technology Review is to bring about better-informed
and more conscious decisions about technology through authoritative,
influential, and trustworthy journalism.
Browse
International
Editions
* Company
+ About Us
+ Careers
+ Advertise with Us
+ Insights
+ Reuse, Permissions, and Licensing
+ Press Room
* Your Account
+ Log In / Create Account
+ Activate Account
+ Newsletters
+ Manage Account
+ Manage Subscription
* Customer Support
+ Help/FAQs
+ Contact Us
+ Feedback
+ Sitemap
* More
+ Events
+ MIT Enterprise Forum
+ MIT News
* Policies
+ Ethics Statement
+ Terms of Service
+ Privacy
+ Cookie Statement
+ Commenting Guidelines
MIT Technology Review © 2019 v.|e^iπ|
/3
You've read of three free articles this month. Subscribe now for
unlimited online access. You've read of three free articles this month.
Subscribe now for unlimited online access. This is your last free
article this month. Subscribe now for unlimited online access. You've
read all your free articles this month. Subscribe now for unlimited
online access. You've read of three free articles this month. Log in
for more, or subscribe now for unlimited online access. Log in for two
more free articles, or subscribe now for unlimited online access.