{"id":8432,"date":"2018-01-03T14:22:50","date_gmt":"2018-01-03T13:22:50","guid":{"rendered":"http:\/\/www.cpsl.com\/2018\/01\/03\/lsp-perspective-applying-human-touch-mt-qualitative-feedback-mt-evaluation-part-2\/"},"modified":"2025-04-02T14:25:19","modified_gmt":"2025-04-02T14:25:19","slug":"lsp-perspective-applying-human-touch-mt-qualitative-feedback-mt-evaluation-part-2","status":"publish","type":"post","link":"https:\/\/www.cpsl.com\/en-GB\/lsp-perspective-applying-human-touch-mt-qualitative-feedback-mt-evaluation-part-2\/","title":{"rendered":"LSP Perspective: Applying the Human Touch to MT (Part 2)"},"content":{"rendered":"

In all the discussions about Machine Translation<\/a>, we do not often hear much about post-editors and what could be done to enhance and improve the task of PEMT, which is often viewed negatively.\u00a0 Our expert\u00a0<\/strong>provides useful insights into her direct experience of improving the experience for post-editors doing this kind of work.<\/h2>\n

(PART 2)<\/em>… More than asking for severity and repetitiveness, what I really want to know is what I call \u2018annoyance level,\u2019 i.e. what made the post-editing job too boring, tedious or time-consuming \u2013 in short, a task that could lead the post-editor to decline a similar job in the future. These are variables that quantitative metrics cannot provide.<\/strong> Automated metrics cannot provide any insight on how to prioritise error fixing, either by error severity level or by \u2018annoyance level.\u2019 Important errors can go unnoticed in a long list of issues, and thus never be fixed. I have managed several MT-based projects where the edit distance was acceptable (< 30%) and the post-editors\u2019 overall experience, to my surprise was still unpleasant. In such cases, the post-editor came back to me saying that certain types of errors were so unacceptable for them that they didn\u2019t want to post-edit again. Sometimes this opinion was related to severity and other times to perception, i.e. errors a human would never make. In these cases, the feedback form helped detect the errors and turned a previously bad experience into an acceptable job.<\/p>\n

It is worth noting that one cannot rely on one single post- editor’s feedback. The acceptance threshold can vary quite a lot from one person to another, and post-editing skills are also different. Thus, the most reasonable approach is to collect feedback from several post-editors, compare their comments and use them as a complement to the automatic metrics. We must definitely make an effort to include the post-editors\u2019 comments as a variable when evaluating MT quality, to prioritise certain errors when optimising the engines. If we have a team of translators whom we trust, then we should also trust them when they comment on the raw MT results. Personally, I always try my best to send machine-translated files that are in good shape so that the post-editing experience is acceptable. In this way, I can keep my preferred translators (recycled as post-editors) happy and on board, willing to accept more jobs in the future. This can make a significant difference not only in their experience but also in the quality of the final project.<\/p>\n

<\/div>\n

5 Tips for Successfully Integrating Qualitative Feedback into your MT Evaluation Workflow<\/strong><\/h3>\n