Correlation between Quantitative Citation Analysis and Opinion Mining of Citation Contexts

Quantitative citation analysis fails to take into account different citation motivations which may be neutral, confirmative, or negational. It necessitates devising new methods or techniques to evaluate cited documents based on the attitudes of their citing articles, to increase the accuracy of the results of the quantitative approach. The challenge is believed to be partially answered by content analysis of citations, including citation opinion mining. It is based on extracting and analyzing the sentiment words occurring in citations, or citation contexts, i.e., a word window surrounding any given citation within a citing paper. However, there exist few pieces of evidence on the degree of divergence or convergence of the results of the quantitative and content-based approaches. To provide further research evidence, the present study investigated the correlation between the results of these two citation-analysis approaches.

Using a citation analysis method with the quantitative and opinion-mining approaches, this communication explored a sample consisting of 524 medical papers. Their bibliographic information and citations were extracted from PubMed and CoLil, respectively. 3663 citations were identified, of which 3639 contexts were available through CoLil. The citations were processed using the KNIME data mining platform. The opinion scores of the words were extracted from SentiWords. The citations opinions were measured in terms of the polarity and strength of the average opinion scores of their words. The data were then analyzed by Spearman Correlation.

The citations were revealed to carry numerous sentiment words. They were mostly positive in their polarity, however, the number of citations with negative polarity was also considerable. The citation counts were found to be directly and strongly correlated to the absolute count of the opinionated citations. However, they were revealed to be indirectly and weakly correlated to the relative count of opinionated citations, i.e., the number of opinionated citations normalized by the total count of the citations. Furthermore, the citation counts had an insignificant correlation with the relative frequency of positive citations, while displaying significant direct relationships with negative and neutral ones. Moreover, they were indirectly associated with the average opinion scores.

In general, the findings of this research showed that as the number of citations increases, the relative number of opinionated contexts decreases. They were dominated by a positive polarity which is in line with previous studies revealing a confirmative motivation in citation behavior. The predominance of positive opinions implied explicit and implicit confirmation motives of researchers reflected in the citing works: the citer may explicitly express her opinion about the cited article and its features, or implicitly express her approval by citing it (i.e., using algorithms, methods, tools, findings and etc.).According to the findings, as citation counts increase, the cited papers witness a reduction in their opinionated contexts’ relative numbers, while experiencing an increase in their negative and neutral ones. Consequently, the content-based citation analysis with an opinion-mining approach may be able to adjust the results of the quantitative approach. However, this finding and its generalizability should be treated with caution. Because the sample of the current research was not selected randomly. Given the differences between disciplines and scientific communities in their citation behavior, it is necessary to replicate the research in various contexts to support the results. Moreover, in the dictionary-based opinion mining method applied in the present study, the big challenge is precisely detecting negative opinions. As in negative citations, the negative opinions of citers may be mingled with their reports of negative objective findings. In other words, the method cannot precisely distinguish these two types of opinion contexts, i.e., the negative attitudes of the citers and their narration of negative findings. Moreover, citations are of social nature. Therefore, negative citations are mostly hidden and indirect, and their lexical identification is hence quite difficult. Thus, advanced methods using, for example, machine learning algorithms, are required to detect and analyze any possible implicit and indirect negative opinions which the direct natural language method may fail to capture.