Partial Knowledge in Multiple-Choice Testing

The intent of this study was to discover the nature of (partial) knowledge as estimated by the multiple-choice (MC) test method. An MC test of vocabulary, including 20 items, was given to 10 participants. Each examinee was required to think aloud while focusing on each item before and while making a response. After each test taker was done with each item, s/he was required to provide answers to retrospective questions. The specific purpose of the questions was to elicit the examinees’ ‘systemic knowledge’ of each item (i.e., how much they knew about each component of the item as well as their knowledge as to the relationship between the components). Based on the nature of the test takers’ protocols, task analysis, and objective of thestudy, a coding scheme was developed for analyzing the protocols. Then, the protocols were closely examined to find out the coding categories that contributed to the basic identity of the two polar classes of knowledge (i.e., Absence of Knowledge and Full Knowledge). The same approach was used in the rest of the protocols to find out the possible subcategories of partial knowledge. Similar codes were categorized into natural classes to develop a model of knowledge in MC testing which resulted in a model of knowledge comprising five categories.