Extracting Parallel English/Persian Sentences from Comparable Corpora using Syntactic Information

Parallel corpora have always been considered among the richest resources in the field of natural language processing. These corpora include translated texts from two or more languages that are usually aligned at the different levels of word, clause, or sentence. Notwithstanding the many uses of these corpora in different studies such as linguistic researches, statistical machine translation, and cross language information retrieval; unfortunately parallel corpora have always been rare and limited in number and quality.. Accordingly, in this paper an automatic method for extracting parallel sentences from comparable resources is introduced which exploits syntactic information. In this method, by using syntactic information of the sentences, an alignment model is trained. The highest practical accuracy of the alignment model on the test set (208 pairs of sentences) was measured to be 77% and the highest precision on the training set (830 pairs of sentences) was 97.7%. Considering the tiny size of the golden corpora, n-fold cross validation technique was used in all training algorithms. To attain higher precision, a new similarity search algorithm was implemented which increased the practical accuracy on the test set from77% to 85.15%. The final outcome of this research was an alignment toolkit and framework which was named "Isfahan University Parallel Corpus Framework" or IPCF, which can be used by the researchers in the field of computational processing of Persian language to construct standard parallel corpora.