#270 AI Translation State of the Art with Tom Kocmi and Alon Lavie

0:00With the rise of the LLMs, we needed new ways how to evaluate them because they kind of catch up with the evaluating metrics and then be new problems.

0:11Florian:Hey everyone, and welcome to a great episode here at SlaterPod. Today on the podcast, we welcome Tom Kocmi and Alon Lavie. So Tom is a researcher at AI Lab Cohere and Alon is a distinguished career professor at the Language Technologies Institute at Carnegie Mellon University, one of the leading universities in language AI has been for decades. So we also have with us Slater's own research analyst, Maria Stassimiotti. So Maria covers language AI here at Slater and I'm very happy to have her on. And we're here today to talk about the findings of the world's most important AI translation competition, for lack of a better term, and generally the state of the art in AI translation more broadly.

0:54Florian:So hi everyone and thanks so much for joining. Hello. Nice to meet you. Cool. So look, I'm trying to give a little bit of background to today's conversation. So it's going to revolve around the WMT competition that concluded recently, and we've been covering it on slater.com quite extensively. So just super briefly, and I will ask you to tell us more about it later on, Tom and Alainz, but super briefly, I think the WMT, it stands for Workshop on Machine Translation and it was initially first held like in the mid 20, like 2006 or something as part of a, it was a chapter at the North American chapter of the Association of Computational Linguistics, but then I looked it up.

1:36Florian:It said in 2016 with the rise of NMT, it kind of became, the WMT became a conference of its own. And yeah, so this concluded recently and you both were instrumental in doing this. And yeah, and generally, so we want to talk about that, but we also want to talk with you about the kind of general state of the art in translation AI. So now let me hand it over to you, Tom and Alon. Can you just tell us a bit more about your background, how you got into translation slash language AI and your current role, maybe starting with Tom? Currently, I'm a researcher at Cohere where I'm part of a multilingual team focusing mostly on machine translation and evaluation.

2:15And before I joined like a year ago and before I was working for four years for Microsoft Translator where I've been leading the evaluation part of the whole pipeline. And so I'm leading the WMT conference or mostly the general empty share task where Alon is taking care of the metrics share task or automatic evaluation. So maybe a little bit about the WMT. it's as you said it's a conference nowadays and it contains multiple share tasks that focus on different parts of the machine translation either the how to improve the performance of the models or how to evaluate them or how to evaluate the metrics and this year we hit the 20 years of wmt conference i've had a pretty long career in in this space and i've been here in pittsburgh all around sort of Carnegie Mellon for just about 30 years.

3:18But I was here a professor already at CMU for about 20 years, the last starting from 1996. My career, my research areas have really focused on machine translation. And over the last 10, almost 20 years, actually, one of my areas has been automated evaluation. of translation quality, primarily machine translation, but increasingly any translation quality that involves automatic translation or human translation or a combination of those. So earlier on, soon after BLUR came out in 2002, I actually initiated a research project that developed one of the earliest metrics that was trainable for machine translation, That was the Meteor metric that was actually used by a lot of people in the field up until for over 10, 15 years, I guess, until the neural era came out.

4:29Meteor is actually used beyond machine translation, actually. I keep being surprised. It's the most cited of all of my research on Google Scholar and on Semantic Scholar. I have immense numbers of citations for Meteor, and it's used in various NLP applications quite successfully. But then in 2009, I had a startup. I spun out from my lab here at CMU, a startup company in machine translation adapted for enterprises, a company by the name of Safaba. and long story short, Safaba was acquired in 2015 by Amazon, at which point I took a hiatus from my academic career. I remained as an adjunct professor and continued to do some advising, but I went for four years into Amazon as a senior manager for machine translation and ran the machine translation research group here in Pittsburgh for Amazon.

5:32And after that, I left Amazon in 2019 and I did two senior AI management, R &D management roles, two years at Unbabel. And then for the last two years from 2023 until earlier this summer, I was the VP of AI research at Frays. And then I concluded my executive kind of management full-time role at Frays early this summer and I decided to come back late in my career to academia. But along the way, at Unbabel, we launched this initiative developing a new neural age machine translation evaluation method called Comet, which was widely adopted and is still widely used until today. I think We'll talk about this maybe a little bit later, but I think in the age of LLMs, new technology is already taking over.

6:34So I don't expect that Comet will be the dominant or the primary metric to be used for much, much longer, but it's done very well and has been very useful for progressing the research for machine translation over the last few years.

6:50Florian:Great. Thanks. That was super helpful. So maybe before we go into the WMT, I just want to take a quick detour into, if you were, or before the kind of LLM revolution were in the field of AI translation, how would you recap the past three, four years since, you know, since these models came online in a big way, maybe starting with you, Tom, and then moving to Allon? I would say, like, maybe over the last five years is the best timeframe. So the main topic that's been changing is like the huge progression in the evaluation, where we change a lot of stuff that we've been doing differently. And with the rise of the LLMs, we needed new ways how to evaluate them, because they kind of catch up with the metrics and then be new problems.

7:35I think another change is mostly about how the translation looks like from the LLMs. So the style is slightly different. They are maybe a little bit more fluent. And this also affected all of the other parts of how to train them, how to evaluate them and everything. So I think the main progress in MT was mostly into evaluation or from my perspective. And how to actually find out that they are better than previous models, that they are improving. Or it's not just a mirage that humans are biased by the fluency, for example. I think it's well known kind of like the generational kind of milestones of the technology, you know, and I was deeply involved in both the statistical machine translation age, where we, you know, we built dedicated models for enterprise customers.

8:32And then the shift, of course, to the neural age around 2015, and then the advent of transformers. So from the early neural to late neural and then to LLM. But I think the change from neural to LLMs, maybe the most important aspect of that is, first of all, that grammaticality and fluency in most of the major target languages is largely solved. Basically, the problem with machine translation today is not, you know, before, even in the neural age, but certainly, you know, around 10 years ago, if you were using machine translation, you know, you were getting, you know, rough, rough language out there.

9:18there were grammatical mistakes, you know, in actresses related to actual semantics, but also just the generated language was completely rough. And so editing of machine translation 10 years ago was a very different type of endeavor than what it is. Now, late in neural, you know, things got already much, much more fluent, but fundamentally we're still translating sentence by sentence. So still, you know, when you're with translators that are working or any kind of enterprise project that's working on translating a document or a piece of content or a web page, right, had to really worry about how those sentences fit together.

9:59And there was still a lot of editing work. But now with LLM and the move to LLM doing the translation tasks, they do it in full context. They generate highly grammatical language, right? It's very rare to find actually grammatical error. So they do make mistakes, you know, semantic mistakes here and there. But humans translators make mistakes all the time. The characteristics of the mistakes that LLMs make are not necessarily exactly the same. But the fact that they're actually so fluent and can generate so much in context, right? And either you can do that nowadays just off the bat and use a model that is going to generate that type of very fluent contextual language.

10:45Or maybe for whatever reason, there are domains, certainly on a commercial space where people are still using neural models, but then you can use an LLM to basically post-check itself, right, and automatically smooth out and correct and, you know, install things that only humans were able to do before. So in my mind, those are the main really radical differences. And it has deep impact on the workflows that are used across the industry. And we're in a transition period of moving from well-established, well-understood, both in terms of process, but also in terms of economics models of MTPE and post-editing and automated workflows, to LLMs and agents doing various parts of the puzzle.

11:39And that transition is quite difficult and challenging to a lot of users and enterprises that use this technology at scale. I'd like to ask Tom and Alan to tell us a bit more about WMT, what WMT actually is, and why does it matter for the field? Let me start about the general WMT and what it is. So why is it important is that over the last 20 years, every year, we are evaluating what's the state of the art and all the academics and industry can compete with their best machine translation systems and see how far we can actually push the frontier of the research to see how good the systems are. And as well, we are also pushing how to evaluate them.

12:29And every year, we are improving on the pipeline, because if you would be evaluating them the same way like 10 years ago, we could kind of claim that machine translation is solved and there is no need to focus on it. So the main important is that every year, it's completely new set up, a fresher data set for evolution, evaluated by humans. And it's kind of like a competition with among the academics and industry in this realm. And as for what is WMT? So WMT is a conference which also has multiple shared tasks that focus on different subparts of the whole machine translation field. So from a low-resource language, terminology-focused translation, post-editing, etc.

13:12And then there are two main tasks, I would say. The one which is the oldest, it used to be news translation. Nowadays, it's a general machine translation. But the main focus of the task is to train a general purpose machine translation that can translate any domain, any text into a subset of the languages, which is selected every year differently. And then the other most important task is the metrics share task or automatic evaluation share task, which is focusing how can we design automatic metrics that will correctly evaluate how systems are evaluated. Maybe with this, I would give a floor to Alan.

13:58So automatic evaluation of translation has become very, very important because human evaluations are just so slow, expensive and difficult to run. And there's been many evolving use cases, but two main ones is just for machine translation developers is to be able to drive their research, to know when they are actually training a new model or new variants of a model, whether that new model is better than the previous one. You need an automated evaluation system that would actually assign scores and give you that type of information at a granular level as much as possible. And then there's what's become known as QE, which is basically the translation time quality assessment of machine translation or a translation, where the goal is actually to know at translation time what the quality of any generated translation is so that you can actually be actionable, you know, do things, make decisions on that, whether they're workload decisions, whether it's post-editing decisions where editors should focus more attention, what can be pushed out maybe in raw machine translation form and that nature.

15:12So those two scenarios, the main scenarios of kind of offline contrastive evaluation for machine translation development, which is known as reference-based evaluation, also typically called as an empty metrics, that started first around 2002. And then in WMT, starting around 2008, there was a shared task on that. And then QE metrics, which kind of came later, or QE systems, was a separate shared task at WMT, even though there's a lot of similarities. They're both automatic quality evaluation of machine translation. But the two scenarios were different enough and the technologies were different enough that there were two kind of parallel shared tasks for quite a few years.

15:59And this is the first year, actually, where we unified and merged both of those things together because the underlying technology, particularly with LLMs, for solving, for building systems that solve this task have come together very closely together. So this year we did a unified share task on both reference-based and reference without references, so both metric use cases and QE use cases together. And we adopted fundamentally all of the language pairs and the data set up from the translation test that Tom just described to you a few minutes ago. So 16 language pairs, including low resource language pairs, much more difficult data for the machine translation system, generating human.

16:46So here in these tasks, we do generate human assessments because that's the gold standard against which we evaluate all these automated translation quality systems. How well they're doing is evaluated against human annotations of translation quality. And predominantly, this has been over the last several years done using MQM annotation. The listeners probably know about MQM, but if not, we can talk a lot, clarify that. And this year, it was mostly done with a simplified MQM schema that Tom and several other researchers actually proposed a couple of years ago called ESA. It's basically a simplified version of MQM that allows for a little bit more faster and easier annotation by humans and hopefully a little bit more consistency between the annotators.

17:41That was one of the major changes this year, Alan, correct? Yes. So there were several significant changes this year, one of which was the unification of the reference metrics plus NACUE under one unified shared task. So references are optional and the systems that are submitted can use them when they're there or not use them when they're not there. One was the data setup that we adopted from the general shared task, which much larger, longer segments. So it's paragraph-long segments and harder data for the machine translation system. And the third was the human evaluation that's done primarily with ESA.

18:25So, Alan, you mentioned that the test sets used from the NDPs or the tasks were actually harder this year. And Tom, would you like to explain more on the more difficult tests that you use for these tasks? I think the one thing that we've been seeing over the last two years or three years is that there have been more and more claims that machine translation is soft and then maybe there is no need to push more in that realm and like let's focus on something else. But the actual reason that we as a researcher saw is that, well, we've been evaluating on relatively easy sentences and especially for the high resource languages.

19:01translating news article or Wikipedia pages is really easy. That's simple language. And you don't need a specialized system to do that. And with LLMs, they actually do it perfectly. So what we focus this year the most is how to increase the difficulty, how to focus on something that's actually challenging, not just taking random news articles, but instead we focus, for example, on news commentary, some reviews that are more creative language. We focus on the literary domain, which was one long story. We focus on speech translation. We took videos and transcribed them with automatic speech recognition.

19:40So the systems would either have to use the speech or the audio input or know how to fix the errors that are from the automatic speech recognition. And lastly, we focus also on the social domain, basically user-generated content that contains errors, contains different vocabulary and style. So those being like one of the ways how we try to improve the difficulty. Second is that we also developed a difficulty sampling algorithm, which when you have a large pile of data, let's say that you collect news articles, how to select those that will be more challenging for translation. And for that, we developed the algorithm that can sort them, and basically, we took the hardest ones.

20:23So once we had all of these more difficult sources, then we asked a human translator to translate them. And, for example, from some of their feedback, we found out that it's difficult even for humans. We've been getting feedback back that, oh, I don't need more time to translate it. This is more challenging. I don't understand this. Do you have some terminology to support it? So that was one of the first feedbacks that we got. And later, when we also got the automated systems, we saw that they do struggle more in contrast to last year. So they contain much more errors in the output. So it's the direction that we went, basically.

21:04Let's actually focus on something that's challenging and needs more effort to translate rather than simple news articles, for example. So I want to point out what the consequences of that are on actually on the evaluation system shared task. So the fact that the content is harder for the machine translation task means also that it's more challenging for human evaluators to evaluate the quality on that content. So our human annotations for gold standard that were developed also for the main shared tasks, but also for our shared tasks, were more challenging for the annotators to do. On the other hand, the fact that it's harder for machine translation doesn't necessarily mean that it's harder for automatic metrics to distinguish between those systems.

21:57Because, you know, when machine translation was really bad, it was easy to develop actually metrics that distinguish between them. because it was very easy to tell between good, medium, and bad machine translation. But nowadays, when most of the translation generated by machine translation systems are actually very high quality, and indeed, it's not the case for every of the language pair that we saw this year, especially with the techniques that Tom mentioned for making the data harder. It's not that everything was still very easy, but for certain language pairs, we saw it was still pretty easy for some of the content types.

22:37So now it's actually much harder for automated systems to make, to identify, you know, the small differences between those systems, especially on a segment by segment basis, right? So when we run our task, actually, we have kind of two meta evaluation processes there. I don't want to go into too much technical detail, but one of them looks at what we call the system level, which is fundamentally indeed still the ability to distinguish between the quality of two systems on a collection of data that we have, on a collection of documents. And there, because it's aggregated over a collection of data, overall kind of the strengths and weaknesses kind of pop out and automated metrics can do a fairly good job of distinguishing between the systems in the same way that humans do.

23:34But on a segment by segment basis, even with these long segments, it turns out to be quite challenging, both for humans, but even more so for automated systems. And we see that particularly this year. And this year, one of the things is that we tried out, we didn't just collect submissions from participants from the outside, but we, the organizers, not only ran some legacy metrics, basically, to have as baselines and comparisons, but we ran all of the major LLMs with simple prompting as an LLM as a judge for the first time. This is the first time that we actually, in this year, TAS did this. We ran all the major LLMs with promptings to do machine translation quality scoring, and we evaluated how well they do.

24:26And we got some interesting results. We can talk about those in a minute in more detail. So, Mylon, would you like to tell us a bit more about the findings this year? I think that the most important thing or the most important finding this year is how strong the reasoning based large language models were as evaluators, correct? Yes. So, for our shared task for the automated translation quality evaluation, I think a few things kind of popped out and there were also some things that we didn't quite expect. But the main kind of headline, I would say, is indeed we had two very strong LLMs. Both of them were kind of the latest generation with reasoning that did very well in the evaluation, both at the system level, but also at the segment level.

25:17I can mention them by name I don't know if that's important the findings paper has all the details and you can actually see exactly who those are one of those was one of the LLMs that we ran and you know it was fairly basic prompting the other one was a system that was submitted by an outside participant where they not only use a strong LLM with reasoning but they did extensive prompting and setup of the task for the LLM to including very specific examples about how to actually score translations, including detailed instructions about what to generate, etc. And that seemed to be that seemed to work very, very well.

26:12But probably one, we only had very few submissions that actually, to my surprise, actually, I was hoping we would get more submissions of this nature, but we didn't get too many submissions that had detailed prompting with an LLM to do this task. And I think that's clearly in my mind where the future is going. So I think, yes, that's probably the highlight of the shared task this year. One of the surprises was actually that we got a lot of submissions that were variants of trained neural metrics that have been very strong performers in recent years, particularly Comet, but not just Comet. as metric X in quite a few submissions did some, you know, basically took the baseline comment and did something, either extended it to multiple sentences or incorporated other types of knowledge or various things like that.

27:11And all of those variants of those trained neural metrics did not do very well this year. They were surprisingly weak. And we're still in the process of doing some in-depth analysis to try and fully understand really why those metrics underperformed this year. I have a feeling it's mostly, and we have some smoking guns. Tom was helpful also already on trying to find one highlight there. I'll mention it, actually. Maybe, Tom, you can articulate even more about this. But it turned out that many of the submitted machine translation systems that were submitted to Tom's task, to the general translation task, were actually tuned on Comet.

27:55Basically, they used Comet in the process of developing the variant of their model to hill climb or to improve results. And you get better Comet scores as you do that. But any metric has some weaknesses and biases. And if you just hill climb on a metric, then you're incorporating those biases in the output in ways that are subtle and you're not aware of. And then when you evaluate that model and compare it to independent human judgments again, then those biases pop out. And then therefore that model will actually underperform. You know, it will get high comet scores, but those comet scores will not agree very well with human judgments anymore.

28:40So we think that that was one of the things that happened this year, that because many of the systems hill climbed on these metrics, then it caused the metrics actually to underperform on those systems when data from the system was used for the evaluation. But the other thing is just the radically different data scenario. These really long segments, these low resource languages, the hard data, it might indicate that these trained neural metrics were over-trained for data scenarios that were basically no longer, did not match the data that we had this year. I remember you, Alan, mentioning in a call we fired a couple of days ago that a comet became a victim of its own success.

29:27Exactly. But indeed, that is the case. I think the fact that many people in the community adopted Comet as the way to do automated evaluation over the last few years and then used it to train their system, in fact, it is a victim of its own success because now it is underperforming on evaluating systems that went ahead and did that.

29:53Florian:Tom, do you have anything to add to the eval part? I wanted to just highlight why I think it's critical that evaluation is staying on the tiptoes and keep pushing. And it's more important than even the modeling. That basically, this is what we are currently seeing in the field. It's like the classical good heart's law problem. That basically, the moment that the metric became a target, it ceased to be a good metric. And since everyone starts using Comet as a reverse model, for example, or filtering, then the metric can't even catch up. And it's like just one of the puzzles and like why we need to, why it's super critical that WMT is happening every year with new setup, we are updating all the problems that we found in the past and this is just underlying the importance.

30:37Florian:All right, let's go to the performance, right? I mean, we would probably have to screen share because you have so many kind of rankings and tables and it's always challenging for our production team to reproduce them in our tool, but I guess one of the takeaways is that the general purpose LLMs like Gemini 2.0 Pro and GPT, I think you guys used 4.1, did really, really, really well. So what does that tell us, I mean, compared to some of the other systems like things like Google Translate, et cetera, like what does that tell us about LLM-based translation, where it is today? So I would say one comparison is basically like you mentioned a couple of production systems like Google Translate, et cetera, versus the Gemini and GPT.