Exploring difficult-to-score essays with a hyperbolic cosine accuracy model and Coh-Metrix indices (Journal Article)
Jue Wang, George Engelhard Jr. & Trenton Combs
Exploring difficult-to-score essays with a hyperbolic cosine accuracy model and Coh-Metrix indices (Journal Article) - USA :Taylor and Francis Group and Routledge ,March 2023 - 125-144 p. - The Journal of Experimental Education, Volume 91,2023 number 1 .
***______________***
Abstract-
Unfolding models are frequently used to develop scales for measuring attitudes. Recently, unfolding models have been applied to examine rater severity and accuracy within the context of rater-mediated assessments. One of the problems in applying unfolding models to rater-mediated assessments is that the substantive interpretations of the latent variable are challenging. The purpose of this study is to evaluate the use of Coh-Metrix indices as an approach for defining the characteristics of essays related to their scoring difficulty. Essays from a rater-mediated writing assessment were selected (N = 100), and rater accuracy was defined using an unfolding model. The scoring difficulty of essays along the unfolding scale (i.e., difficult-to-score below, easy-to-score, and difficult-to-score above) was related to essay feature indices defined with the Coh-Metrix text analyzer. Six areas of essay features were considered: descriptive, text readability, lexical diversity, latent semantic analysis, syntactic complexity, and word information. The results suggest that the substantive interpretation of the underlying unfolding scale can be enhanced by examining essay characteristics with Coh-Metrix indices. This methodology offers a promising approach for understanding the unfolding continuum and gaining information about the characteristics of difficult-to-score essays. Suggestions and implications for evaluating rater-mediated assessment systems using unfolding models are discussed.
0022-0973
RIEBPL Library
Coh-Metrix indices
essay features
rating accuracy
writing assessments
Unfolding models
370.5
Exploring difficult-to-score essays with a hyperbolic cosine accuracy model and Coh-Metrix indices (Journal Article) - USA :Taylor and Francis Group and Routledge ,March 2023 - 125-144 p. - The Journal of Experimental Education, Volume 91,2023 number 1 .
***______________***
Abstract-
Unfolding models are frequently used to develop scales for measuring attitudes. Recently, unfolding models have been applied to examine rater severity and accuracy within the context of rater-mediated assessments. One of the problems in applying unfolding models to rater-mediated assessments is that the substantive interpretations of the latent variable are challenging. The purpose of this study is to evaluate the use of Coh-Metrix indices as an approach for defining the characteristics of essays related to their scoring difficulty. Essays from a rater-mediated writing assessment were selected (N = 100), and rater accuracy was defined using an unfolding model. The scoring difficulty of essays along the unfolding scale (i.e., difficult-to-score below, easy-to-score, and difficult-to-score above) was related to essay feature indices defined with the Coh-Metrix text analyzer. Six areas of essay features were considered: descriptive, text readability, lexical diversity, latent semantic analysis, syntactic complexity, and word information. The results suggest that the substantive interpretation of the underlying unfolding scale can be enhanced by examining essay characteristics with Coh-Metrix indices. This methodology offers a promising approach for understanding the unfolding continuum and gaining information about the characteristics of difficult-to-score essays. Suggestions and implications for evaluating rater-mediated assessment systems using unfolding models are discussed.
0022-0973
RIEBPL Library
Coh-Metrix indices
essay features
rating accuracy
writing assessments
Unfolding models
370.5