Document Readability Level and Objective Complexity of Work Tasks: Any Impact on Relevance Judgment and Ranking?

Aims to identify the role of document readability level and the objective complexity of the work task on students' relevance judgment and document ranking.

Quantitative content analysis was utilized to determine the readability level of texts. The statistical population of the research included 2825 graduate students of Shiraz University of Humanities. To determine the sample size, due to time-consuming process of relevance judgment, students’ time constraint, and sample size of similar studies, a non-probability sampling method of voluntary type was used. Two work tasks were designed based on Borland's (2000) Repository of Assigned Search Tasks and experts' opinions. After confirming the validity of the work tasks by the experts, searching through the Persian article databases, including SID, Magiran, and Noormags, four relevant, four partially relevant, and two irrelevant articles were selected by five experts The articles were divided into difficult and very difficult levels by determining their readability levels using Flash-Diani readability formula. In the next step, ten articles related to two work tasks were scored by users in a six-level spectrum (from entirely irrelevant to entirely relevant) then ranked based on the relevance degree (from 1 to 10). Data was analyzed using social science statistical software version 23 (SPSS), and T-tests and Mann-Whitney U tests were conducted.

The results showed significant difference between students' relevance judgment with different readability levels. Students rated documents with difficult readability levels more relevant than documents with very difficult readability levels. In addition, considering the role of readability level on the ranking, there was a significant difference between the rankings of relevance in the documents with different readability levels according to the students. Documents with difficult readability levels were ranked lower in terms of relevance. Therefore, students performed finer on rating documents with difficult readability levels.  Findings about the role of complexity level of work showed that objective complexity of the work task affected the students' relevance judgment. The retrieved documents related to the simple work task were found to be more relevant than the complex work task. Regarding the role of complexity level of work tasks on ranking, findings  revealed that the level of objective complexity of the work task affected the ranking of documents, and the simple work task ranked lower in terms of relevance ranking. Therefore, students performed finer on the simple work task than on the complex work task regarding relevance rating.

The role of level of readability of documents and objective complexity of the task affected relevance, judgment, and ranking. Further research on the effect of these indicators on relevance judgment  can improve the design of information retrieval systems and increase the quality of the relevance and ranking algorithms.