Supervised approach for keyword extraction from Persian documents using lexical chains

Keywords are the main focal points of interest within a text, which intends to represent the principal concepts outlined in the document. Determining the keywords using traditional methods is a time consuming process and requires specialized knowledge of the subject. For the purposes of indexing the vast expanse of electronic documents, it is important to automate the keyword extraction task. Since keywords structure is coherent, we focus on the relation between words. Most of previous methods in Persian are based on statistical relation between words and didn’t consider the sense relations. However, by existing ambiguity in the meaning, using these statistic methods couldn’t help in determining relations between words. Our method for extracting keywords is a supervised method which by using lexical chain of words, new features are extracted for each word. Using these features beside of statistic features could be more effective in a supervised system. We have tried to map the relations amongst word senses by using lexical chains. Therefore, in the proposed model, “FarsNet” plays a key role in constructing the lexical chains. Lexical chain is created by using Galley and McKeown's algorithm that of course, some changes have been made to the algorithm. We used java version of hazm library to determine candidate words in the text. These words were identified by using POS tagging and Noun phrase chunking. Ten features are considered for each candidate word. Four features related to frequency and position of word in the text and the rest related to lexical chain of the word. After extracting the keywords by the classifier, post-processing performs for determining Two-word key phrases that were not obtained in the previous step. The dataset used in this research was chosen from among Persian scientific papers. We only used the title and abstract of these papers. The results depicted that using semantic relations, besides statistical features, would improve the overall performance of keyword extraction for papers. Also, the Naive Bayes classifier gives the best result among the investigated classifiers, of course, eliminating some of the features of the lexical chain improved its performance.