Benchmarking Machine Translation Efficacy for Teaching EAP Reading Comprehension Skills

Although Machine Translation (MT) is extensively researched within the field of Artificial Intelligence (AI) and translation studies, few studies have attempted to implement MT output in foreign language teaching (FLT). One potential aspect of using MT in FLT refers to the implementation of MT output for reading comprehension. Considering the existing gap in the body of research on this issue, the present study aimed to investigate whether MT output is qualified enough to be used as an aid in EAP reading comprehension courses. More specifically, this study intended to benchmark the efficacy of MT output for EAP reading comprehension courses based on the data obtained from testing its comprehensibility and probing the students’ perceptions. To achieve the objectives of the study, MT was operationally defined as quality assessment in terms of output efficacy, a combination of usability and comprehensibility, which mirrors the ultimate goal of MT use in EAP reading comprehension courses, from the users' or target readers’ standpoint. Within this perspective, the current research was an attempt to assess the quality of MT output in terms of comprehensibility and the degree to which MT output might be comprehensible to the EAP students participating in this study.

The participants of the study, 140 Iranian undergraduate university students majoring in the field of education at Farhangian University, Iran, were selected based on simple random sampling. Oxford Quick Placement Test was used to homogenize them in terms of English proficiency. Two versions of a reliable reading comprehension test, human translation (HT) and Machine Translation (MT), were given to. This test included 25 multiple-choice items, assessing the participants' literal comprehension of information stated in the passage as well as higher-order comprehension that required making inferences and conclusions. In particular, the items measured textual coherence, inference, reference, scanning, skimming, and word-meaning inference. To test the reliability of the tests, the KR-21 formula was applied and the results showed that both HT test (.83) and MT test (.78) were reliable. To investigate the perceptions of the participants on the efficacy of the MT output they encountered on the test, semi-structured interviews were conducted with some of the participants in Persian. 

With reference to the results of non-parametric tests such as Spearman’s rho, and Mann-Whitney Tests, and considering the observed effect sizes (Cohen’s d), it was revealed that, generally, the efficacy of MT output is comparable to that of HT. Moreover, in terms of reading comprehension sub-skills, the qualities of the two translations were comparable with regard to scanning, and inference, but not skimming and reference. Furthermore, the findings from the interview indicated that the students perceive MT to be a seminal aid for their EAP reading comprehension activities despite the minor problems that exist in the output such as morpho-syntactic errors or inappropriate lexical equivalents.

The present study confirmed the fact that the efficacy of MT output is target-reader-dependent and text-dependent since it is determined both by the characteristics of the readers, such as their disciplines, and text features, as demonstrated by the significant differences in comprehension levels of the same readers measured by the same questions for HT and MT output. Accordingly, this study shed limelight on comprehensibility as a criterion of MT output efficacy; that is to say, it has to be reminded that MT quality needs to be defined as a context-bound and target-reader-specific concept.