Compilation Reckase's Method and Mathematical Programming Method in Designing Optimal Item Pool for Computerized Adaptive Tests

Computerized adaptive testing (CAT) is a testing procedure that can result in improved precision for a specified test length or reduced test length with no loss of precision. But, for computerized adaptive tests (CATs) to work well, they must have an item pool with sufficient numbers of good quality items. Many researchers have pointed out that, in developing item pool for CATs, not only is the item pool size is important, but also the distribution of item parameters. Yet, there is little research on how to identify those desirable features. This paper applied and extended the basic idea of the “bin-and-union” method proposed by Reckase (2003),- which is a Monte Carlo method to determine the properties of an optimal item pool-, and mathematical programming method to develop the optimal item pool for a mathematic operational CAT. This study extended the method for designing item pools calibrated with the three-parameter logistic model and applied it to situations where the Sympson-Hetter procedure is used to control the item exposure rate. The designs include estimates of desired item pool size and item parameter distribution. The design process includes identifying a series of candidate item pool features by taking into consideration multiple factors that may affect the desired features of the item pool. The performance of the simulated item pools has been compared with operational item pool by considering some evaluation criteria. The result of evaluation indicated that the mechanism used to identify the desirable item pool features has functioned well and appropriate for identifying a desirable item pool features of mathematic operational CAT.