Current operational CATs mainly use dichotomous items. However, including polytomous and set-based items into CAT is attracting growing attention. Few studies have been conducted to investigate how to assemble a mixed-item-format CAT efficiently. The requirements for assembling a CAT are often in conflict with each other; the test assembly approach should advance progress toward all objectives. The shadow test approach (STA) is one of the most appealing CAT assembly methods as it can handle complex constraints. It is very flexible and can deal with many constraints simultaneously. However, STA solves the optimization problem uniquely for each examinee, which may result in some problems in operational CATs, such as context effects and difficulty in item replacement. These problems can be partially solved by the bin-structured method, which aims to find a single standardized solution to divide the item pool and solve the constrained combination optimization problem. However, though the bin-structured method is promising in future applications, as a relatively new method, research in bin-structured method is still rare, and none uses mixed-item-format based CAT. And no study investigates what factors may influence the quality of results from the bin-structured method.This study compared the mixed-item-based CAT and dichotomous-item-based CAT to see whether the mixed CAT had advantages over the dichotomous-item-based CAT and what challenges it brought. Furthermore, it compared three CAT test assembly approaches, including STA, combination of STA and bin-structured method, and bin-structured method in context of CAT containing mixed item formats. The psychometric models used in item pool, item parameter distribution, test length and imposed test constraints were manipulated to simulate various real test situations.The results supported incorporating polytomous items and set-based items into CAT, as mixed CAT had higher test accuracy and stability than the binary CAT. However, the mixed CAT had a fairly skewed exposure rate distribution, and further analysis showed that the highly exposed items were all polytomous-scoring items. Another relevant problem for mixed CAT was its low item usage efficiency, as a lot of items (mainly dichotomous items) were unused. This study also supported the application of bin-structured method in mixed CAT as it can produce equal or even better outcomes than the traditional STA. Meanwhile it can also simplify the computation involved in CAT, standardize the look of the test, provide good control over the content sequences in advance, and facilitate item replacement and exposure control.
Read
