Article Ranking by Recommender Systems vs. Users’ Perspectives

To compare the rankings of articles by Google Scholar and Web of Science recommender systems against users’ perspectives.

120 PhD candidates of Shiraz University in the fields of humanities, sciences, engineering and agriculture, (30 from each field) voluntarily participated in the study. They were asked to introduce three articles had recently read for their thesis. One of the three which was indexed by both databases was chosen and named as the core article. For each core article 10 recommended articles recommended by each recommender system were retrieved (2,400 overall). Using imitating software exclusively designed for this study, participants were asked to rank the articles retrieved by the two recommender systems for their core articles. Normalized Discounted Cumulative Gain (NDCG) measure was employed for analysis.

There was a noticeable but weak relationship between the users’ assigned rankings and the rankings of Google Scholar and Web of Science databases. Correlation between the rankings of both databases with NDCG measure was also weak.

The algorithms used for ranking by both recommender systems hardly in matched that of the users. Therefore, ranking algorithms of both databases may need some revision.