Impact of Scoring Instructions, Timing, and Feedback on Measurement : An Experimental Study (Journal Article)

***______{For Hard Copy, Please visit Library.}________***

Abstract: We investigated whether and to what extent different scoring instructions, timing conditions, and direct feedback affect performance and speed. An experimental study manipulating these factors was designed to address these research questions. According to the factorial design, participants were randomly assigned to one of twelve study conditions. We collected data from 2,484 participants on 20 quantitative reasoning items obtained from an admissions test for graduate and professional schools. The results showed that there were significant differences in both performance and speed between the conditions. Both item time limits and feedback led to faster but less accurate responses. The results for scoring instructions with an emphasis on speed and test time limits were mixed with respect to accuracy, but the responses in these conditions were generally faster. Notwithstanding these experimental effects, measurement invariance held for models fitted to response accuracy and response time, which means that the manipulations could reasonably be summarized through impact on structural parameters (latent means and variances) of the studied models. This finding is supported by the lack of differences between conditions in the correlations with an external measure of quantitative reasoning.