This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Patz, R. J. Articles by Mariano, L. T. Search for Related Content Social Bookmarking                         What's this?

The Hierarchical Rater Model for Rated Test Items and its Application to Large-Scale Educational Assessment Data

CTB/McGraw-Hill

Brian W. Junker

Carnegie Mellon University

Matthew S. Johnson

Baruch College

Louis T. Mariano

RAND

Open-ended or "constructed" student responses to test items have become a stock component of standardized educational assessments. Digital imaging of examinee work now enables a distributed rating process to be flexibly managed, and allocation designs that involve as many as six or more ratings for a subset of responses are now feasible. In this article we develop Patz’s (1996) hierarchical rater model (HRM) for polytomous item response data scored by multiple raters, and show how it can be used to scale examinees and items, to model aspects of consensus among raters, and to model individual rater severity and consistency effects. The HRM treats examinee responses to open-ended items as unobsered discrete varibles, and it explicitly models the "proficiency" of raters in assigning accurate scores as well as the proficiency of examinees in providing correct responses. We show how the HRM "fits in" to the generalizability theory framework that has been the traditional tool of analysis for rated item response data, and give some relationships between the HRM, the design effects correction of Bock, Brennan and Muraki (1999), and the rater bundle model of Wilson and Hoskens (2002). Using simulated and real data, we compare the HRM to the conventional IRT Facets model for rating data (e.g., Linacre, 1989; Engelhard, 1994, 1996), and we explore ways that information from HRM analyses may improved the quality of the rating process.

Key Words: generalizability • hierarchical Bayes modeling • item response theory • latent response model • Markov chain Monte Carlo • Multiple ratings • rater consensus • rater consistency • rater severity

Journal of Educational and Behavioral Statistics, Vol. 27, No. 4, 341-384 (2002)
DOI: 10.3102/10769986027004341


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?

This article has been cited by other articles:

				L. T. Mariano and B. W. Junker Covariates of the Rating Process in Hierarchical Models for Multiple Ratings of Test Items Journal of Educational and Behavioral Statistics, September 1, 2007; 32(3): 287 - 314. [Abstract] [Full Text] [PDF]

				R. J. Mislevy Can There Be Reliability without "Reliability?" Journal of Educational and Behavioral Statistics, January 1, 2004; 29(2): 241 - 244. [PDF]

				M. S. Johnson and B. W. Junker Using Data Augmentation and Markov Chain Monte Carlo for the Estimation of Unfolding Response Models Journal of Educational and Behavioral Statistics, January 1, 2003; 28(3): 195 - 230. [Abstract] [PDF]